I want a public process context with edu rerank(processor: EduRerankProcessor, context: str, query: str…
I want a public process context with edu rerank(processor: EduRerankProcessor, context: str, query: str…: a task in MiMo-V2.6-RL-harbor-code: MiMo-V2.6-RL Code (Harbor) (Harbor dataset). The stats dict should always use the keys orig tokens , new tokens , and ratio . Token counts should use…
The task
The stats dict should always use the keys `orig_tokens`, `new_tokens`, and `ratio`. Token counts should use `tiktoken`'s `cl100k_base` encoding when available, with special tokens disallowed, and otherwise fall back to `len(text) // 4`; successful compression should report `ratio` as `new_tokens / orig_tokens`…
Part of FineEnvs/MiMo-V2.6-RL-harbor-code.