See: https://xethub.com/blog/improving-parquet-dedupe-on-hugging-face-hub https://joshleeb.com/posts/content-defined-chunking.html
