dev log

Author: davidfrantz

docs/source/history/v3/v-03-09-00.rst

@@ -0,0 +1,163 @@
+.. _v3900:
+
+FORCE v. 3.9.00
+===============
+
+Release: 13.10.2025
+
+**Read through the changelog closely. In this version, there are some important changes that may affect your workflows!**
+
+- **FORCE L2PS**
+
+  - When merging chips of the same day/sensor to avoid data redundancy, the merging 
+    strategy for the QAI files has been revised. Before, it was just updating the images,
+    i.e., merely overlaying them on top of each other. Now, the QAI files are merged feature-based
+    on a custom logic such that the most restrictive QAI value is kept. This ensures that reproducibility
+    is guaranteed, independent of the order in which the chips are merged. 
+  - It is now possible to specify the resampling method for the DEM.
+    The parameter ``DEM_RESAMPLING`` can be set to ``NN`` (Nearest Neighbor), ``BL`` (Bilinear), or ``CC`` (Cubic Convolution).
+    The default is ``BL``. This allows users to choose the resampling method that best fits their needs.
+    Thanks to Oleg Zheleznyy for the suggestion.
+  - When co-registering L7 SLC-Off imagery, a big portion of the data failed. 
+    This issue was addressed by altering the no-data filtering logic in the pyramid building process of LSReg.
+    Thanks to Oleg Zheleznyy for reporting this issue.
+  - Updated COG default parameters. COG has been set as the default output format for L2PS 
+    (note: HLPS still uses GTiff as default)
+  - The default format for L2PS has been changed from ``GTiff`` to ``COG``.
+    This affects the output format of the L2PS products. COGs are more efficient for cloud storage and access,
+    making them a better choice for modern geospatial workflows. Users can still specify ``GTiff`` if needed.
+  - The default creation options for ``GTiff`` files  have been updated to include ``COMPRESS=ZSTD`` and a gridded tile layout.
+    Refer to https://github.com/davidfrantz/force/discussions/376 for more details.
+    These options improve the compression and efficiency of the GTiff files. 
+    Users can still use the previous parameterization through custom GDAL options if needed.
+  - This choice has resulted in a decoupling of internal file layout and higher level processing (more below and in the issue above).
+    As a result, the ``BLOCK_SIZE`` parameter has been deprecated and is no longer used in L2PS.
+    Also, the block size is no longer part of the data cube definition.
+    This comes with a new file format for the data cube definition in tag and value notification.
+    **IMPORTANT: FORCE has backward compatibility for existing data cubes, but the new format is used
+    when creating a new data cube. Existing data cubes do not need to be converted. 
+    However, older software versions will not be able to read data cubes created with the new format!**
+  - The ``TILE_SIZE`` has been changed into a two-dimensional parameter.
+    This allows users to specify different tile sizes in x and y direction, which can be useful for certain applications.
+    The sizes should be given in the unit of the data cube's coordinate reference system (CRS), usually meters.
+    The sizes should be a multiple of the processing resolution. 
+    Note that it is not generally recommended to make use of non-square tiles, but hey, you can now if you want to.
+
+- **FORCE HLPS**
+
+  - The internal file layout (blocks) has been decoupled from the sub-tile processing chunks. This means that the
+    penalty for using a non-optimal block size became less significant.
+    See the details described in https://github.com/davidfrantz/force/discussions/376.
+    This has been implemented throughout FORCE. To make this clearer, the term "block" is now only used
+    in the context of the internal file layout, whereas "chunk" is used for the processing units.
+    These chunks can now be specified by the user via the ``CHUNK_SIZE`` parameter. 
+    The old ``BLOCK_SIZE`` parameter has been deprecated and is no longer used in HLPS.
+    Note that ``CHUNK_SIZE`` requires two values, i.e., the chunk size in x and y direction.
+    The sizes should be given in the unit of the data cube's coordinate reference system (CRS), usually meters.
+    The sizes should be a multiple of the processing resolution and the tile size should be divisible by the chunk size.
+    This is to ensure that the chunks fit neatly into the tiles. This is the same behaviour as with the previous ``BLOCK_SIZE`` parameter.
+    The only usage difference is that the chunk size can now follow a two-dimensional approach, i.e., different sizes in x and y direction,
+    which e.g. can make processing way more efficient when using masks.
+    Further note, that the chunk size must be greater than 0 and less than or equal to the tile size (in the specific direction).
+
+    **IMPORTANT: When working with an old data cube, you should ensure that the chunk X-size is equal to the tile X-size to avoid a huge penalty on efficiency.**
+
+  - It became apparent that using COGs is not possible when using sub-tile processing chunks. 
+    So, just don't do that. If you want to use COGs, set the chunk size to the tile size!
+    You need enough memory to process a full tile at once, though.
+    https://github.com/davidfrantz/force/issues/374
+    Alternatively, use GTiff as output format, which works with sub-tile chunks, and the new configuration is actually
+    pretty comparable to COGs.
+
+  - in ``force-higher-level``, all ARD-based modules:
+    The ``SENSORS`` definitions were moved out of code and are now held in a the directoy ``force-misc\runtime-data``, 
+    which is installed with FORCE. This provides users with way more flexibility regarding sensors that shall be
+    used. The files are in ``JSON`` format and can be easily modified to change sensor behavior. An example application
+    might be the remapping of the various NIR bands if the default band mapping does not suit your analysis. 
+    Another application might be the addition of a new sensor that has not yet been supported by FORCE. 
+    When doing this, new band names can be introduced, too. As an example, this opens the door to support sensors
+    like EnMap. As the directory name suggests, these files are evaluated at runtime, hence no recompilation is necessary.
+    When you place a file ``EnMap.json`` in the directory, FORCE can make use of it by specifying ``SENSORS = EnMap`` in 
+    the configuration file. Note that when using containerization, individual files can be mounted into the container to 
+    make them available, and separating your user-defined sensors from the default ones.
+    While this gives users a lot of flexibility, it also requires some responsibility: some internal logic might
+    be affected by modifiying exising sensor definitions. Known examples are the spectral adjustment, which might
+    become dysfunctional if the involved sensor definitions are changed.
+    Also, the previous logic of automatically naming ouptut files with a matching sensor name is no longer possible.
+    The user must now specify the desired output sensor name via the ``TARGET_SENSOR`` parameter. This target sensor
+    must have a sensor definition file in the runtime-data directory, too. And it needs to match the band subset of the
+    ``SENSORS`` combination. E.g., if ``SENSORS = LND08 SEN2A``, the ``TARGET_SENSOR`` could be ``LNDLG``, wherein the 6-Band
+    band subset is defined. If ``SENSORS = SEN2A SEN2B SEN2C``, the ``TARGET_SENSOR`` could be ``SEN2L``, which has a 10-Band 
+    subset defined. Note that the ``SPECTRAL_ADJUST`` parameter influences the band subset as it adds additional bands to the processing,
+    which are synthesized. As an example, if ``SPECTRAL_ADJUST = TRUE`` and ``SENSORS = LND08 SEN2A``, the ``TARGET_SENSOR`` could be ``SEN2L``.
+
+  - in ``force-higher-level``, TSA-submodule:
+    The ``INDEX`` selection has been moved out of code and is now held in a the directoy ``force-misc\runtime-data``, 
+    which is installed with FORCE. This provides users with more flexibility regarding indices that shall be
+    used. The file ``indices.json`` is in ``JSON`` format and can be easily modified to change index behavior. 
+    As an example, NDVI can be remapped to use different bands to "create" a new, but not yet implemented index 
+    without touching the code as such. Do note however, that this only works with existing index templates implemented in the code.
+    That said, simply adding a new index definition in the JSON file will not work, as there is no corresponding implementation in the code.
+    However, together with the new ``SENSORS`` flexibility, this creates interesting possibilities.
+    Also: in addition to the listed indices, the user can specify any band name defined in the ``SENSORS`` definition files.
+    This allows users to directly use spectral bands as "indices", e.g., ``INDEX = NIR SWIR1`` 
+    (or any other "new" band name defined in the sensor definition files).
+    Additionally, the ``SMA`` index can be used to perform a Spectral Mixture Analysis, in which case the endmember file 
+    must have the same number of bands as the selected ``SENSORS`` combination. 
+
+  - in ``force-higher-level``, UDF sub-module:
+    a new feature was added to the UDF module, which allows users to add auxiliary products
+    to the data array that is passed to the UDF. 
+    The user can specify which auxiliary products to use in the configuration file via the new 
+    ``REQUIRE_AUX_PRODUCTS`` parameter. The auxiliary products are specified as a white-space separated list,
+    e.g. ``REQUIRE_AUX_PRODUCTS = DST VZN AOD``. Custom products may also be specified (*Int16!*), thus you can invent 
+    and use new tags. An auxiliary product is a product should always accompany the main product (usually ``BOA``).
+    In the UDF, the auxiliary products are appended to the data array, thus increasing the number of bands.
+    The bandnames of the auxiliary products are set to the product name, e.g. ``DST`` for the DST product.
+    If no auxiliary products are wanted, the user can set ``REQUIRE_AUX_PRODUCTS = NULL``.
+
+  - in ``force-higher-level``, UDF sub-module:
+    the ``REQUIRE_AUX_PRODUCTS`` mechanism has been implemented here as well. 
+    You may use the ``DST``, ``HOT``, and ``VZN`` products.
+    Before, the usage of a specific product was solely managed by using a corresponding score > 0. 
+    To prevent accidental use of a product, the user must now explicitly specify the product in addition to the score.
+
+  - in ``force-higher-level``, all ARD-based modules:
+    a new parameter ``DATE_IGNORE_LANDSAT_7`` was added. During the last years of Landsat 7's life,
+    the satellite was slowly de-orbited, which caused the acquisition times to slowly shift away from
+    the nominal time. Operational production continued. This parameter allows the user to ignore
+    the Landsat 7 data after a specific point in time. The default is ``2099-12-31``, which just means 
+    that all Landsat 7 data will be used.
+
+  - in ``force-higher-level``, all modules:
+    a new parameter ``FAIL_IF_EMPTY`` was added by Florian Katerndahl.
+    If set to ``TRUE``, the module will fail if no input data is found for the specified configuration, 
+    e.g. time range, or if no output was written. This is meant to be used in complex workflows, where
+    one want the program to signal an error condition. 
+    The default is ``FALSE``, which is better suited for production runs, 
+    as e.g. an NRT application may not have data for a specific day or region.
+    In this case, only a warning is displayed with some hints on how to resolve potential issues.
+
+  - endmember, sampling, feature and response tables can be a csv now, i.e. "," has become an allowed column separator. 
+    Tab and whitespace are still allowed, too.
+    @German users, take care to use English csv settings!
+
+--  **FORCE AUX**
+
+  - ``force-mosaic`` was overhauled:
+    The tool should now be much more efficient and faster as it splits the working load into smaller chunks
+    instead of relying a singular find command. As suggested by Max Freudenberg, the tool can now be used
+    if your data cube does not have files in *.tif or *.dat extentsion. The user can now specify the
+    file extension with the ``-e`` option. The default is ``tif``, whereas ``dat`` is no longer part of the default.
+    The tool can now also be used on write-protected data cubes by specifiying an output directory that no longer
+    needs to be a subdirectoy of the data cube. The default is still a `mosaic` folder within the datacube.
+    
+  - added a new tool ``force-virtual-datacube``:
+    This tool allows users to create a virtual datacube from a physical datacube. It is useful for creating
+    a virtual representation of the data without duplicating the actual files, thus saving disk space.
+    It can be used to combine multiple datacubes into a single virtual dataset, which can be useful for analysis.
+    The tool can be used with various options to customize the output, such as specifying the pattern of files
+    to include and whether to overwrite existing files.
+
+  - Minor adjustments have been made in a couple of AUX tools to reflect the changes in handling chunks.
+