nightly darks: add ability to force a break even if timing, config, detector don't change

[sbailey]

Currently nightly darks use dark exposures within a time window as long as they have the same timing file (CCDTMING), CCD config (CCDCFG), and detector (DETECTOR), even if those span multiple configurations in the desi_spectro_calib yaml files. e.g. if the PSF file changes, we can still use darks spanning two different configs.

However, sometimes we need to force a boundary in what darks can be used, e.g. from desihub/desisurveyops#422, z0 was saturated while taking pixflats on 2026-01-28, leading to much higher dark current for the next several days. When we run Matterhorn, we need to make sure that nightly darks from the days leading up to that don't use darks from 2026-01-28 and beyond even though CCDTMING, CCDCFG, and DETECTOR didn't change.

Option 1

Add a new config parameter like "NEWDARKS", which if present and True/1, would cause desi_compute_dark to treat that as a boundary for selecting darks, even if CCDTMING, CCDCFG, and DETECTOR are all the same.

This would require additional bookkeeping in scripts/compute_dark.py and/or ccdcalib.py to also track configurations and whether there is an enforced boundary.

Option 2

A file in exposure_tables/YEARMM/ could serve as an override list of which exposures to use for the dark night, and if that file isn't present then it is algorithmically calculated.

Option 3

Something else. Ideas welcome.

I'm currently leaning towards Option 2 as providing more flexibility with less coding hassle.

@jbanovetz @julienguy @akremin

[akremin]

Can you elaborate a bit more on idea #2 @sbailey ? Would it be one file per night, but that file contain all of the night+expid+cam pairs for all surrounding nights that we want the pipeline to use for a given configuration used on a given night? Or was it just a list of the exposures from a specific night's expid+cam pairs? I could see the former being useful, albeit a bit redundant across nights. I'm not sure what the latter buys us. Or maybe you had something else entirely in mind?

[sbailey]

For option 2, I was thinking the first case you described: an optional file per THISNIGHT+CAMERA, that if it existed would list the full set of NIGHT,EXPID to use for making a nightly dark for CAMERA on THISNIGHT. Or maybe just a range of EXPIDs to consider, and then standard filters apply within that range. That could and likely would include NIGHTs before and after THISNIGHT.

This approach would give us a lot of flexibility, but we'd likely need to generate helper scripts to make them in the first place. e.g. in this case with z0, we'd need to go back 2 weeks worth of darkexps-z0-THISNIGHT.csv to list the exposures to use to make sure the default window didn't bring in data from 20260128 and beyond. And it is fragile — if we changed the desi_compute_dark default window to be 30 days in the future instead of 15 days, then we'd have to remember to go back even further in time to create more override files. I'm becoming less interested in this option.

[jbanovetz]

I have been working with Option 1 and have a (little rough) first draft of how to do this. The changes to the code are in the dark_reset branch and the DESI_SPECTRO_CALIB folder used is /global/cfs/cdirs/desi/users/banovetz/desi_spectro_calib_new_dark_keyword. Essentially what it does is loads a reference config from the yaml file and looks for a new keyword DARK_RESET and if true, applies it (right now I just have it set up that if the keyword is present to do something and doesn't actually look at the value). Then when running through the list of darks to use, it keeps a running track of when the last time a DARK_RESET was set (dark_reset_begin) and when it was set in the future (dark_reset_end) and just uses the files in between those dates.

I have checked by running three scenarios when there is a DARK_RESET set for z7 in a config for 20260202-03 and 20260204 onwards:
When used on a past date, does it not include the dark_resets:
desi_compute_dark --specprod daily --save-preproc --preproc-dark-dir $PSCRATCH --camera z7 -r 20260201

On the date of the first reset, that it doesn't include any past or future data
desi_compute_dark --specprod daily --save-preproc --preproc-dark-dir $PSCRATCH --camera z7 -r 20260202

On the date of the second reset so that it doesn't include any past data:
desi_compute_dark --specprod daily --save-preproc --preproc-dark-dir $PSCRATCH --camera z7 -r 20260204

These three tests have passed but the big if-elif tree has me worried about corner cases and if there isn't a more elegant solution (which might be better solved in a PR). I am happy to try ay more tests that might be applicable!