Revise downscaling pipeline so outputs meet USGS GeoDataPortal specs

[Joshdpaul]

USGS GeoDataPortal is now an Open Storage Network (OSN) Pod - basically an object store full of zarr files.

For our downscaled data to be uploaded here as part of an official data release, it has to meet a bunch of specs (TBD). We should revise the pipeline so that outputs meet or are very close to the required spec, which will save effort in the data release process.

[Joshdpaul]

Some notes from 9/16/25 meeting with USGS data people
(Ellen Brown + ORISE fellows Peter + Nathan)

Data Release on GeoDataPortal:

The data holdings on the OSN Pod will be cataloged in the USGS Water Mission Area’s spatio-temporal asset catalog (STAC) , which is currently under development in this public repository . This combination of tools - data storage on the OSN pod, cataloged for access through the STAC - will replace the USGS WMA THREDDS Catalog around Summer 2024, further described in the USGS Geo Data Portal migration to labs.waterdata.usgs.gov blog.

The datasets on the OSN Pod have free access to their zarr stores via S3 API. This is a programmatic data access option that we want for the data release!

The LOCA2 datasets are an example of what the downscaled CMIP6 will eventually look like. Note the structure of the dataset, which includes a single "ensemble" dimension that flattens the model, variant, and scenario into one:

...
"ensemble": {
    "description": "Ensemble dimension combining the model name, ensemble member, and temporal range of interest.",
    ...
  }
...

We provide our individual output zarr stores (one for each model / scenario / variable combination), then Ellen and her team will aggregate them, flatten the dimensions, fix the CRS, etc. We will work with them to create the master metadata for the combined zarr store.

Data Release on ScienceBase:

Ellen + team will also take our individual output zarr stores and convert them to NetCDF format for direct download from ScienceBase. Even though this makes 2 separate datasets with different structures, we want to keep this option for anyone who is not using programmatic access via GDP!

We will create the metadata for each of the NetCDF files, and for the ScienceBase landing page itself.

To Do list for formatting our individual output zarr stores:

• Assess chunk shape + size in outputs - what data access pattern are we prioritizing for? We can probably decide in general terms and let Ellen + team do the actual chunking of the master dataset as they aggregate it.
• Decide on NetCDF structure (one per model/variable combination?) and restructure zarr stores accordingly
• Add variant (e.g., “r1i1p1...”) in zarr store metadata (or provide a lookup table like this)
• Attempt an aggregation to make sure dimensions are sane and able to be combined
• Follow CF conventions in metadata and data structure