Homogeneous data types in datacubes

[m-mohr]

Regarding:

Ensure homogeneous data types for the values in the datacube, choosing the most precise data type of the data types present (e.g. float over int).

@alexgleith wrote:

Don't necessarily agree with this.

Can be useful to support different data types, and sometimes using Int is better, e.g., when working with very large volumes of data... halves the data volume.

@m-mohr wrote:

This one is indeed a bit debatable, indeed. But again, it needs to be predictable. If some libraries apply scale and offset for example and some don't that's not good. If you load some bands as int and some as float and then apply an operation over different data types and one implementation fails and one works, also not good. Proposals?

@clausmichele wrote (and @przell gve thumbs up):

Sentinel-2 L2A could be an example scenario, since it contains different data-types (uint16 and uint8) but also different resolutions. In my opinion it's fine to suggest to use the same data type, if we want to achieve a common guideline. It would still be possible to create different datacubes for specific scenarios like I'm used to do: cube1 (Sentinel-2 bands at 10m uint16) cube2 (Sentinel-2 SCL at 20m).

@przell wrote:

In the context of best practices it sounds right to have one resolution and one data type per data cube.
Since they are newly generated anyway from file lists, I think this makes sense.

The scale and offset is a tough one though. Doesn't seem practical to not "allow" for scale and offset at all. There is scale and offset fields in STAC, if they are filled it should be possible for implementations to pick them up reliably, no?

Origin: #59 (comment)

[alexgleith]

Scale and offset is definitely a consideration, but lots of EO data make that an exercise for the user whether or not they apply it.

And some people want to work in Kelvin, while some in Centigrade, so doing the transform may be unnecessary too...

Still, another example of multiple data types is if you had Red, Green and Blue in floats, you may still need to keep SCL/Pixel QA in uint8, as a bit flag.

[przell]

Still, another example of multiple data types is if you had Red, Green and Blue in floats, you may still need to keep SCL/Pixel QA in uint8, as a bit flag.

One could argue that in this case a seperate collection should be created for the SCL/Pixel QA since the user has to treat them differently in the workwflows anyway (filter them out for most applications). On the other hand, the user would have to load multiple collections.
There was comment from @m-mohr that predictability is better when a homogenous datatype is used. I generally agree. In theory, could that be somehow mitigated by the data_type field?

It seems a decision on this topic will be quite opinionated (?). There's pros and cons for scale and offset and also for homogenous datatype

[alexgleith]

a seperate collection

A separate STAC Collection?

[przell]

A separate STAC Collection?

Yes. That was my intention. On a second thought, it's also a lot of work on the data providers side.

[m-mohr]

I don't think separate STAC collections make sense. In openEO terms, you may want to load two different datacubes though.