Add LoRA support for the tensor-parallel models supplied by the installed Transformers integration.
Add LoRA support for the tensor-parallel models supplied by the installed Transformers integration.: a task in HF ML Tasksmith (Harbor dataset). A model loaded with tp plan="auto" has linear layers marked with hf tp plan and hf device mesh . LoRA should work when those layers use the supported…
The task
A model loaded with `tp_plan="auto"` has linear layers marked with `_hf_tp_plan` and `_hf_device_mesh`. LoRA should work when those layers use the supported colwise and rowwise plans across two GPUs. Keep the ordinary non-parallel LoRA behavior working.
Part of FineEnvs/HF_ML_Tasksmith.