Group datasets for TopoFlow in Ethiopia (GLDAS and GPM)

[dgarijo]

When the TopoFlow data was transformed, it was grouped by year. This is problematic if you want to run simulations e.g., from mid December to mid Jan.

Can you please run your transformation pipeline to consolidate the 10 years under 1 file? For GPM and GLDAS data. And for the 30/60sec resolution.

We would need these registered in the data catalog as individual datasets (with only 1 resource in this case)

[sumwmer]

Hi @dgarijo, I'm currently working on your issue. Can you provide me with more information on the climate.rts file (contained in here), such as the bounding box size so I can sanity check before I combine everything? My current understanding of the rts file is this is a long 1d numpy array and it's ordered by time stamp, so straight forward concatenation will suffice. Right now we have the scripts to do so but I just want to make sure I'm doing the right thing before I register on data catalog :)

[dgarijo]

@summer7xinting, unfortunately I didn't do the original conversion, so I don't know the details. It was @binh-vu and @minhptx the ones who did the conversion from GLDAS and GPM to the different months. They have all the details about it. Please ask them :)

[dgarijo]

Any updates on this?

[dgarijo]

When doing this, would it be possible to publish on the data catalog the 30 and 60 versions of the data separately? Thanks

[sumwmer]

Hi @dgarijo, sorry about the late update. I'm still working on your issue but the files are really big (two regions took ~250G on my local machine) so I can only do two regions at a time (instead of automating the whole thing). I will finish as soon as I can and let you know 😔

[dgarijo]

Thanks for the heads up. Just to be sure, each region has to be registered separately as well. If my estimate is correct, the final uncompressed climate.rts data should be around 10-12 gigs per region (the aggregated one per year is around 1.2)

[sumwmer]

Hi @dgarijo, here's the uploaded alwero and awash part:

alwero_gldas_2008_2018_30 - 12087202-7dae-4b80-be17-d2bab77d1720
alwero_gldas_2008_2018_60 - 94c77f12-d8a4-4423-accd-e31ecad2304a
alwero_gpm_2008_2018_30 - c83f82ab-e97f-4cfa-bf71-c56c62ffa7a5
alwero_gpm_2008_2018_60 - ec1510ad-ee2c-4e7e-b404-15717a38919f
awash_gldas_2008_2018_30 - ecf5c924-13f3-43a6-b667-e6534fab2b58
awash_gldas_2008_2018_60 - 32170c4e-b6ae-4358-9ad8-34dcbe9adf0c
awash_gpm_2008_2018_30 - 41d23e72-b9de-4bf6-8cf5-ef9c86c8314a
awash_gpm_2008_2018_60 - c1382cf2-cf7d-4af5-ad69-e03924553f16

Can you sanity check any one of them and let me know if this is what you want? I'll proceed with other regions in the mean time :)

[sumwmer]

baro_gldas_2008_2018_30 - f14e9d03-2205-41e9-b685-0d9e6d064e6c
baro_gldas_2008_2018_60 - d8bc4c9e-521a-4aea-823b-7405c1ca63a8
baro_gpm_2008_2018_30 - 5e5ec399-2fea-4df4-849c-5bf0ddf7f846
baro_gpm_2008_2018_60 - ae836492-d70f-4427-854f-4e9a2b731902
beko_gldas_2008_2018_30 - ed6cdf85-f952-443c-928a-4a6e5b9816ee
beko_gldas_2008_2018_60 - d4652be8-cb68-4591-bd56-93db4caa0f81
beko_gpm_2008_2018_30 - ddf51a9e-95d5-4a3b-85f9-b71b6dc2b89e
beko_gpm_2008_2018_30 - 11603147-124d-484b-9d28-21ec54ab904d