Processing from a fresh SPECPROD leads to problems with linkcals

[jbanovetz]

I have been trying to run desi_proc_night on a couple of nights (20230306, 20231218) that never make it to the ccdcalib step because of a DependencyNeverSatisfied problem in slum. This is in my personal SPECPROD which I have cleared out (besides exposure tables).
Looking into it, I am running into a problem on the biaspdark step on some of the earlier/later days that require linkcals. The link cal step needs badcolumns as well as the biases but those aren't created when running desi_proc_night. Here is the link cal log I am getting for trying to run desi_proc_night -n 20230306:

Running srun -N 1 -n 1 -c 2 --cpu-bind=cores desi_link_calibnight --refnight=20230308 --newnight=20230309 --cameras=a0123456789 --include=badcolumns ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b0-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b1-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b2-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b3-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b4-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b5-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b6-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b7-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b8-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-b9-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r0-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r1-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r2-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r3-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r4-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r5-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r6-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r7-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r8-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-r9-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z0-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z1-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z2-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z3-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z4-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z5-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z6-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z7-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z8-20230308.csv ERROR:link_calibnight.py:95:check_link: Missing /global/cfs/cdirs/desi/spectro/redux/banovetz/calibnight/20230308/badcolumns-z9-20230308.csv CRITICAL:link_calibnight.py:171:main: 30 errors; not proceeding with making links srun: error: nid006284: task 0: Exited with exit code 1 srun: Terminating StepId=47835781.0

Looking at the pipeline, the badcolumn files aren't created until the ccdcalib step, which isn't launched for one of these other days that only get the biaspdark job. I think this will be a problem when running Matterhorn and launching all jobs at once as it might cause a race condition. For now, I will just copy over the bad column files from what is in daily and try again.

[sbailey]

@akremin please evaluate whether this is a race condition problem for Matterhorn, or whether we are ok as long as we submit the nights in order.

[akremin]

I will look at this in more depth tomorrow, but I think we are okay here. The only thing I want to verify is that we have the correct dependency tracking when doing linkcal to make sure it waits for the job producing the data on the earlier night to complete. That should be in place from Kibo, but I will see if I broke anything in the darknight refactor. Otherwise running nights consecutively should generate the desired outcomes.

And in any case I think the worst-case outcome is that these specific nights refuse to run until that data shows up to link. Obvious that has knock on effects because of cumulative redshifts, so we want to mitigate such issue as much as possible. I'll look more tomorrow (Friday).