Add support for periodic reference model synchronization (TR-DPO style) to KTOConfig and KTOTrainer …
Add support for periodic reference model synchronization (TR-DPO style) to KTOConfig and KTOTrainer …: a task in HF ML Tasksmith (Harbor dataset). When sync ref model=True , the reference model is periodically updated: π ref = α · π θ + (1−α) · π ref prev , where α is ref model mixup alpha and the…
The task
When `sync_ref_model=True`, the reference model is periodically updated: `π_ref = α · π_θ + (1−α) · π_ref_prev`, where α is `ref_model_mixup_alpha` and the update happens every `ref_model_sync_steps` training steps.
Part of FineEnvs/HF_ML_Tasksmith.