Add missing ocean obs files to the 2020–present obsForge database

[guillaumevernieres]

Add missing ocean obs files to the 2020–present obsForge database

Description

A previously generated obsForge ocean observation database (satellite + in situ), produced for PSL appears to have missing in situ observation files. These gaps need to be identified and corrected by re-gathering the raw data and re-running obsForge as needed.

While issues have only been observed in the in situ data so far, a consistency check of the satellite observations is also requested.

Reference dataset location

/scratch3/NCEPDEV/da/common_obsForge/

Scope

• Ocean observations only (satellite and in situ)
• IODA-formatted output
• Coverage: 2020–present
  • 2020 observations appear to be available on WCOSS2

Questions / decisions needed

• Confirm inclusion of 2020 data from WCOSS2? YES
• Extend coverage through December 2025? YES, and 2026 as well, up to present

Notes

• Raw data collection is the time-limiting step; obsForge processing follows.
• Primary issue is in situ completeness; satellite data should be sanity-checked to ensure consistency.

[ilianagenkova]

This task will be a team effort.

@nicholasesposito

1. Please rsync the following directories
   /lfs/h2/emc/global/noscrub/emc.global/ocean/gfs.YMD
   /lfs/h2/emc/global/noscrub/emc.global/ocean/gdas.YMD
   to
   /scratch3/NCEPDEV/da/common_obsForge/
   starting from Nov29,2025 until "now".

2. Add a task to GDA to rsync the above mentioned /ocean directories, in real time, onto URSA, to: /scratch3/NCEPDEV/da/common_obsForge/

(if you don't have write permission, please reply here)

@givelberg ,
Inventory the directory /scratch3/NCEPDEV/da/common_obsForge/ for data gaps between 20210101 and 20251212
and share with me and Ashley.

@AshleyStanfield-NOAA,
You and I will generate any missing bufr dump files, after extracting the necessary tanks from HPSS
We will also stage any necessary marine data from $DCOMROOT for obsForge
@givelberg , once we stage all the marine in-situ bufr dumps, and marine satellite data, you will run obsForge

@apchoiCMD ,
Did we (you and I) not process 2020 already?

@guillaumevernieres , to be on the same page -

• "2020 observations appear to be available on WCOSS2" - the in-situ obs are very likely available on HPSS, but the satellite products may not have been generated that far back.
• I will assume that ocean satellite observations would be : sst, sss, icec (AMSR2), adt

[apchoiCMD]

@ilianagenkova I have already generated 2020 and synced them "TODAY" to Ursa- However, there are many missing in-situ/sat and I think this issue is for that-

@nicholasesposito You are not probably to be able to sync them to that location since myself is only the person who has a permission- I will tag up with you soon or I will do it from my side-

[ilianagenkova]

@nicholasesposito , do you have the right permissions for 1) and 2) ?

[apchoiCMD]

@ilianagenkova The permission for directory of /scratch3/NCEPDEV/da/common_obsForge/ is only for me- The first thing that I think is to look at/scan what are in there.

[ilianagenkova]

Update:

• established a benchmark (file count per obs type, per cycle) for the obsForge output
  (using /lfs/h2/emc/da/noscrub/emc.da/obsForge , Mar1-3,2026)

|-----------|-adt--|--sss--|--icec--|--sst--|--insitu-|
|gfs(gdas)-|-6----|--0(1)-|--8-9-- |--5----|--16----| (per cycle, 4 cycles per day)

• inventoried the 2020-2025 record and identified the periods that need to be reprocessed (/scratch3/NCEPDEV/da/common_obsForge)

Caveat: The obsForge retro-runs will be working with a static $DCOMROOT data archive , i.e. $DCOMROOT will not be populated in real time. As a result, obsForge might "see" and extract more observations (i.e. higher counts) compared to what is available in real-time mode. If needed we may adjust obsForge jobs and scripts.

[ilianagenkova]

@apchoiCMD and @HyundeokChoi-NOAA ,
What are the kick-off times for the obsForge jobs?

[nicholasesposito]

@ilianagenkova I do not have permission using my regular account on Ursa write to the directory in 1) and likely 2). I do not have any sys admin groups on Ursa either.

[apchoiCMD]

@apchoiCMD and @HyundeokChoi-NOAA , What are the kick-off times for the obsForge jobs?

GFS at 02:55 and GDAS at 05:55, then every 6 hours after that for marine obs.

[ilianagenkova]

Benchmark
|--------------|-adt--|--sss--|--icec--|--sst--|--insitu-|
|--gfs(gdas)--|-6----|--0(1)-|--8-9-- |--5----|--16----| (per cycle, 4 cycles per day)

Inventory
2020
|--gfs(gdas)--|-0-5--|--0---|---0--- |--0-5--|--1-7----|
2021
|--gfs(gdas)--|-3-5--|--0-1-|--0-2--|--5-6--|--7-9----|
2022
|--gfs(gdas)--|-5-6--|--0-1-|---2--- |--4-5--|--1-7----| (Note: August is half-misisng across most obs types)
2023
|--gfs(gdas)--|-6----|--0(1)-|---2--- |---5---|--1-3---|
2024
|--gfs(gdas)--|-6----|--0(1)-|---2--- |---5---|--1-3---|
2025
|--gfs(gdas)--|-6----|--0(1)-|--2-9---|---5---|--7-16--| (Note: "insitu" and "icec" increase on Nov1,2025, likely obsforge changed)

[ilianagenkova]

@guillaumevernieres,
Adding missing ocean obsersvation on a date/cycle basis (i.e. only where the data gaps are) is not very easy/efficient.
Given the inventory, I propose we reprocess:

• adt, sss, icec, sst for the entire 2020 and 2021, and August 2022
• insitu for Jan2020-Oct2025 - that will require running obsproc first, and then obsForge.

Do you agree with this plan? Is there a timeline for this task?