Add a corpus shuffling-and-splitting utility for streaming data loading
Add a corpus shuffling-and-splitting utility for streaming data loading: a task in MiMo-V2.6-RL-harbor-code: MiMo-V2.6-RL Code (Harbor) (Harbor dataset). When we train a language model from scratch we feed the trainer with a StreamingDataSilo , which reads training data lazily from disk instead of…
The task
When we train a language model from scratch we feed the trainer with a `StreamingDataSilo`, which reads training data lazily from disk instead of loading everything into memory. To get good randomization across epochs with that streaming setup it helps to pre-process the raw corpus once: break one huge text file into…
Part of FineEnvs/MiMo-V2.6-RL-harbor-code.