Feat: incremental glacier directory creation

[gampnico]

It's finally here!

For an easier-to-read demonstration of the new system, download and try the notebook:

Features

• Backwards compatible
• 30% less data
• Append your own files
• Manifests for each glacier directory, tracking OGGM version, user-added files, dataset tags, unique hashed IDs, timestamps and more...
• Manifest peeking so you can check what's in a gdir before loading it into memory
• Wrappers to batch convert existing glacier directories in utils/compat.py, with support for multiple base URLs and checks to make sure datasets are compatible. You can convert any region, number or combination of RGI IDs.
• Comes with zarr

Overview

This changes the default glacier directory system from cumulative levels to incremental ones. Most levels on the cluster are now deltas, which contain only the files that are different or added relative to the previous levels. Clients layer levels they need into a single glacier directory, which now contain manifests to track changes and identify datasets.

L3 and L5 datasets are standalone. L3 because there are differences in certain files between L1 and L3 e.g. inversion_flowlines, and L5 because it's a "minimal" preset.
I'm calling a combination of levels a "materialisation"
plus a per-level manifest, and has clients layer the levels they need into one glacier directory. Legacy (manifest-less) datasets keep working via a fallback that treats them as cumulative materialisations.

Points for Discussion

• This is a really big change that includes changes from feat: read and write zarr methods to replace read_pickle #1908.
• This is incremental in the sense that users pick from preset deltas available on the cluster, rather than "get this one file". I've decided to do this for multiple reasons:
  • This issue is solved with streaming, because we can use fsspec and xarray to pick and mix the data we want without downloading it first!
  • Zarr increases the number of files stored in each glacier directory. This means if a user downloads data for the globe, this is potentially millions of small files, which is going to strain the server and makes [Bug]: OGGM can't detect if cached downloads are truncated #1947 impossible to solve.
  • Compatibility checks between datasets become significantly harder when checking multiple single files.
  • Most OGGM workflows require a standard set of files anyway. If we want to give users access to extra files, we could introduce this with a higher preprocessing level (since these are now deltas the storage impact is minimal).

TODO:

• Incorporate @pat-schmitt's changes after dev is merged to master
• Replace the test base URL with an actual URL once data is available.
• Decide on terminology for documentation. So far I'm calling merged deltas or standalone prepros in a single gdir "materialisations" (e.g. L3, L5). but I think this is a bit awkward.
• Maybe add manifest schema version to dataset_id hash to prevent future manifests clashing?

Closes #1900
Refs: #1896

• Tests added/passed
• Fully documented
• Entry in whats-new.rst