HF RL Explorer

Loading a SentencePiece tokenizer with byte fallback as a fast tokenizer hard-fails

Loading a SentencePiece tokenizer with byte fallback as a fast tokenizer hard-fails: a task in MiMo-V2.6-RL-harbor-code: MiMo-V2.6-RL Code (Harbor) (Harbor dataset). After updating to a recent transformers version, converting a slow SentencePiece tokenizer that was trained with the byte fallback…

The task

After updating to a recent `transformers` version, converting a slow SentencePiece tokenizer that was trained with the `byte_fallback` option into its fast counterpart no longer works at all — it crashes outright instead of just notifying me about the limitation.

Part of FineEnvs/MiMo-V2.6-RL-harbor-code.