Sampling

[limash]

Now we train a model on the same (by coordinates) snippet of data. That means that a model wont work if we change a geographical location since it just remembers local distributions. Do we really want such a behavior? Maybe it would be better to cut small snippets (again by coordinates) on fly during sampling. With the current way 'static' variables can be redundant, but who knows.

[efvik]

Well, the convolutional network is translation-invariant, right? So as long as we work only with CNN, it should generalize well to any location (within the training distribution). It's worth keeping this in mind though, if we move on to other models.

[limash]

I mean that if we train a model on the area around Norway it wont work in Brazil. Or if we assume that it will we should test it. Type of network doesnt matter

[efvik]

Sure, I'm all for testing. Feel free to implement this as an option.

[limash]

sure, do you download only certain coordinates from ERA5? Could you please put the downloading scripts somewhere to ml-ds/app/?

[efvik]

Please see the scripts in ml-ds/src/ml_ds/data . I only download half a year at a time, because cds has limits on the size of each request. Then I used two scripts to merge and interpolate the files (with light compression).

If you have not done it - you will have to create an account at https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=download and add the api key to your home folder. File should be named .cdsapirc , and look like:

url: https://cds.climate.copernicus.eu/api
key: your key

[limash]

Also now all the data is in memory. And sampling is still a bottleneck btw. Usually stuff like input_tensor = torch.tensor(input_arr, dtype=torch.float32) is better to do in advance, if possible, and with bigger chunks of memory. Transfering data from memory to video memory is very slow. But we anyway need to reimplement datasets so the data comes from files just before training. pytorch has out of box prefetching, so it should not be too hard. We should just choose the proper format for the data. But maybe even netcdf is okay, but unfortunately I dont know, before I used tfrecords and parquet for fast parallel loading before training. And maybe we can keep the data in GCP bucket, so we will be able to use colab.

[efvik]

xarray supports lazy loading with xr.open_dataset - as it is currently written, each sample should only be fetched when needed, not all in memory. Did you run into memory issues with it?

NetCDF is probably not optimal, but it is quite convenient when adding new variables or using the result for further analysis. However, if there is significant issues with it then of course we should choose a more suitable format. I also have some experience with parquet, so we can go for this. However, I don't know the best way to format this type of data in parquet. But perhaps we can try with NetCDF first, unless we see it as a big limitation?

[limash]

ahh ok great :) lets check it with more files.

No I didnt have any problems with memory, the sampling was slow (in the original setup profile said it took like 99 % of time). But I should check up on prefetched number of samples and it may help.

[efvik]

99%!? For me, the profiler attributed less than 5% of the time to data loaders. It maybe depends on the setup - are you on google colab, gpu?

Or was it perhaps the training dataset computing the mean and stddev included in the profiler? Since that will require loading all the data.

But for sure, if parquet is significantly faster, or NetCDF proves too cumbersome then we should switch. But with 99% of the time it sounds like something else could be the issue, or?

[limash]

i use my laptop with gpu. yes with 8 filters and 2 blocks gpu load is essentially 0, even with 8 workers. Originally I thought because of xarray operations since I believed that the data was in memory already. But that neural net is super tiny. With a larger net gpu load increases, but sometimes it drops to 0. So still there are some troubles with the data supply.

[limash]

I checked the first results, after 5 epochs it is still worse than direct mapping. I mean x - y is smaller than y_pred - y. I will try to use y-x as y (predict difference)

[limash]

oups, it was wrong, actually the model works very well

[limash]

oh i made a mistake last time, it seems that x - y is indeed smaller than y_pred - y. I hope I didnt mix up anything again