Description
We've encountered multiple issues with the netcdf-to-ioda and bufr-to-ioda converters producing garbage or corrupted values, most recently within the MetaData group.
These invalid outputs propagate downstream and end up choking the JEDI data assimilation applications.
Rationale
The converters should ensure that only valid, self-consistent data are written to IODA files.
Instead of letting corrupted or uninitialized values pass through (and later break the DA), the converters should detect and safely discard invalid entries during conversion.
Tasks
Expected Outcome
- Converters output only valid, consistent IODA files
- JEDI applications no longer crash due to corrupted or invalid data
- Invalid observations are automatically filtered out at conversion time, without requiring additional QC steps downstream
Description
We've encountered multiple issues with the
netcdf-to-iodaandbufr-to-iodaconverters producing garbage or corrupted values, most recently within theMetaDatagroup.These invalid outputs propagate downstream and end up choking the JEDI data assimilation applications.
Rationale
The converters should ensure that only valid, self-consistent data are written to IODA files.
Instead of letting corrupted or uninitialized values pass through (and later break the DA), the converters should detect and safely discard invalid entries during conversion.
Tasks
MetaData,ObsValue,ObsError, etc.)Expected Outcome