Scheduler is spending most of its time in clear not launched queued tasks function
Scheduler is spending most of its time in clear not launched queued tasks function: a task in MiMo-V2.6-RL-harbor-code: MiMo-V2.6-RL Code (Harbor) (Harbor dataset). Airflow running the clear not launched queued tasks function on a certain frequency (default 30 seconds). When we run the airflow on…
The task
Airflow running the clear_not_launched_queued_tasks function on a certain frequency (default 30 seconds). When we run the airflow on a large Kube cluster (pods more than > 5K). Internally the clear_not_launched_queued_tasks function loops through each queued task and checks the corresponding worker pod existence in…
Part of FineEnvs/MiMo-V2.6-RL-harbor-code.