Repair Dask's two Parquet metadata engines for all-or-none overlap between Hive partitions and physical…
Repair Dask's two Parquet metadata engines for all-or-none overlap between Hive partitions and physical…: a task in MiMo-V2.6-RL-harbor-terminal: MiMo-V2.6-RL Terminal (Harbor) (Harbor dataset). A local data pipeline reads Hive-style partitioned Parquet datasets through two Dask backends. Some…
The task
A local data pipeline reads Hive-style partitioned Parquet datasets through two Dask backends. Some producers retain every partition key as a physical file column, while others omit every key and rely on directory metadata. The current frozen readers do not handle these layouts consistently.