Organize a common structure for raw data and observational data

[jmaerz]

Ideally, we would want to have raw data (e.g. used to prepare input/boundary condition files for BLOM/iHAMOCC) and observational data (e.g. for evaluation) stored on nird/betzy in a well-defined folder/file structure.

Partially related: NorESMhub/NorESM#27

Get in contact with: Lise (MET) and Tyge (NORCE) to potentially set this up in /cluster/shared/noresm

For ESMvalTool, data are stored in /nird/datalake/NS9560K/ESGF/obsdata/

[JorgSchwinger]

In my opinion, I think we should NOT duplicate existing and well maintained data repositories as e.g. input4MIPs, WOA, or GLODAP. For each script that generates BLOM/iHAMOCC input data it should be stated which data is used (by giving the DOI, this should also be written as an attribute into the produced netCDF file), so this should suffice to ensure reproducibility.

(of course this is different for ESMVAL, where the data needs to be stored locally to run the tool)

[TomasTorsvik]

I agree that we should not duplicate existing data repositories. If we have input data that is not part of existing repositories, it would be good to involve Lise and Tyge in the discussion. It may be valuable to have a NorESM repository for data that we need to generate input files. Even if there is a DOI, datasets are sometimes withdrawn or retired.

[jmaerz]

Hi @JorgSchwinger, hi @TomasTorsvik,
first, I feel that always downloading the raw data for input file generation didn't work out well in the past (and sometimes even that it wasn't clear at the first place, where input data comes from, how it was regridded, etc. - and took some time to trace this info). Hence, and since not much input data is needed (at least for iHAMOCC), I would still vote for making the raw data accessible in a documented manner on the local system (which will likely also come through the CDEPS system that tries to make use of original input files, when possible).
Second, there is a lot of data available for model evaluation (which I currently store in my own folder on both betzy, nird and even my local computer) which I feel should be better in a shared data/observation/evaluation folder structure - again decently -but not overly- documented (meaning that it should be easy for others to contribute/share evaluation data). I would expect this to be less than a Tb (I currently have about 130Gb input data including all the re-gridded input files, etc. - so quite some doubling that I will clean up the next days - and about 30Gb of observations needed for model evaluation tools).
In any case, I suspect having a shared place is better than likely doubling all these data in individual personal folders... - while I might have started a discussion instead of an issue here. I just feel it could be worth considering such common data structure.

[jmaerz]

Maybe for an interim stage, we can have such data folder in the betzy /cluster/projects/nn2980k project space - becoming thus more visible to those, who have access.

[jmaerz]

Converted to discussion.

[TomasTorsvik]

If we want to do something now, without involving the larger NorESM community, it makes sense to me that we put this under the nn2980k project. We are currently using 340 GiB / 1.0 TiB storage quota, but 800K / 1M files, so we have room for some datasets as long as the file count doesn't increase too much.