CTE Corr not working with new darknight cutoff date

[jbanovetz]

I am trying to implement a cutoff date of 20240509 in which before that day the pipeline uses a non-linear dark model and afterwards, it uses a linear dark model using darknight (see desispec-#2571). The lines of code I changed were in

desispec/py/desispec/workflow/submission.py

Line 264 in 0c5c094

dodarks = 'dark' in proc_obstypes and len(dark_expid_to_process) > 0 # 'darknight' not in files_to_link and

Which now reads:

` # Only submit pdark if it is after 30 days before 20240509 (see desispec issue #2571)

dark_date=night>20240408

dodarks = 'dark' in proc_obstypes and len(dark_expid_to_process) > 0 and dark_date # 'darknight' not in files_to_link and`

And in

desispec/py/desispec/scripts/proc_night.py

Line 806 in 0c5c094

do_darknight = not calibjobs['accounted_for']['darknight']

, which now reads:

`# Only do darknight if date is past 20240509 (see desispec issue #2571)

dark_date = np.median(cal_etable['NIGHT']) > 20240509

do_darknight = not calibjobs['accounted_for']['darknight'] and dark_date` 

in my local desispec. When doing dry runs of desi_proc_night, the changes have the intended effect, before the cutoff date no jobs for preprocessing the darks and darknight are made, and afterwards vice versa. However when trying to test it without a dry-run, there is an error in the ccdcalib job with correct_cte. The date used was 20231218 and the traceback is:

""" Traceback (most recent call last): File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/conda/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/conda/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "/global/cfs/cdirs/desi/users/banovetz/desispec/py/desispec/scripts/fit_cte_night.py", line 56, in _fit_cte_night_kwargs_wrapper out = desispec.correct_cte.fit_cte_night(night=opts["night"],camera=opts["camera"],expids=opts["expids"]) File "/global/cfs/cdirs/desi/users/banovetz/desispec/py/desispec/correct_cte.py", line 805, in fit_cte_night return fit_cte(images) File "/global/cfs/cdirs/desi/users/banovetz/desispec/py/desispec/correct_cte.py", line 598, in fit_cte assert image.meta['OBSTYPE'] == obstype AssertionError """

But I also see this which might be a hint:

`INFO:correct_cte.py:776:get_cte_images: Wrote /global/cfs/cdirs/desi/spectro/redux/banovetz/preproc/20231218/00210511/ctepreproc-r6-00210511.fits.gz

WARNING:correct_cte.py:593:fit_cte: Really should not use this function with a non-flat?!`

[sbailey]

The correct_cte crash appears unrelated to your changes. Please confirm whether the main branch also has this crash for that date or whether it only occurs as a side effect to your changes. Please also push your changes in a branch to GitHub and post the branch name so that others can try it too without reapplying your changes.

[jbanovetz]

I just submitted desi_proc_night --night 20231218 on main and will report back (probably ~1 day). The changes I made are under the branch darkinght_date_cutoff for reproducibility.

[jbanovetz]

I was able to find a date and run it with main before the cutoff (20230516). This also produced the cte_corr error:

INFO:correct_cte.py:776:get_cte_images: Wrote /global/cfs/cdirs/desi/spectro/redux/banovetz/preproc/20230516/00181011/ctepreproc-z3-00181011.fits.gz WARNING:correct_cte.py:593:fit_cte: Really should not use this function with a non-flat?! INFO:image.py:85:write_image: iotime 11.697 sec to write ctepreproc-z1-00181011_tmp690000.fits.gz at 2026-01-23T21:12:06.010790 INFO:correct_cte.py:776:get_cte_images: Wrote /global/cfs/cdirs/desi/spectro/redux/banovetz/preproc/20230516/00181011/ctepreproc-z1-00181011.fits.gz WARNING:correct_cte.py:593:fit_cte: Really should not use this function with a non-flat?! multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/conda/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/conda/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/main/py/desispec/scripts/fit_cte_night.py", line 56, in _fit_cte_night_kwargs_wrapper out = desispec.correct_cte.fit_cte_night(night=opts["night"],camera=opts["camera"],expids=opts["expids"]) File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/main/py/desispec/correct_cte.py", line 805, in fit_cte_night return fit_cte(images) File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/main/py/desispec/correct_cte.py", line 598, in fit_cte assert image.meta['OBSTYPE'] == obstype AssertionError """

I also noticed we didn't run this in the past with daily. Here is the daily slurm job from 20230516:

#!/bin/bash -l

#SBATCH -N 2
#SBATCH --qos realtime
#SBATCH --constraint=cpu
#SBATCH --account desi
#SBATCH --job-name ccdcalib-20230516-00180972-a0123456789
#SBATCH --output /global/cfs/cdirs/desi/spectro/redux/daily/run/scripts/night/20230516/ccdcalib-20230516-00180972-a0123456789-%j.log
#SBATCH --time=00:10:00
#SBATCH --exclusive

# ccdcalib exposure with 30 cameras
# using 30 cores on 2 nodes

echo Starting at $(date)
export OMP_NUM_THREADS=1

# Run nightlybias first
echo Running srun -N 2 -n 15 -c 32 desi_proc --nightlybias --cameras a0123456789 -n 20230516 --mpi --starttime $(date +%s) --timingfile /global/cfs/cdirs/desi/spectro/redux/daily/run/scripts/night/20230516/ccdcalib-20230516-00180972-a0123456789-timing-$SLURM_JOBID.json
srun -N 2 -n 15 -c 32 desi_proc --nightlybias --cameras a0123456789 -n 20230516 --mpi --starttime $(date +%s) --timingfile /global/cfs/cdirs/desi/spectro/redux/daily/run/scripts/night/20230516/ccdcalib-20230516-00180972-a0123456789-timing-$SLURM_JOBID.json

if [ $? -eq 0 ]; then
  echo nightlybias succeeded at $(date)
else
  echo FAILED: nightlybias failed; stopping at $(date)
  exit 1
fi

# Process exposure
echo Running srun -N 2 -n 30 -c 2 --cpu-bind=cores  desi_proc    --cameras a0123456789 -n 20230516 -e 180972 --mpi --starttime $(date +%s) --timingfile /global/cfs/cdirs/desi/spectro/redux/daily/run/scripts/night/20230516/ccdcalib-20230516-00180972-a0123456789-timing-$SLURM_JOBID.json
srun -N 2 -n 30 -c 2 --cpu-bind=cores  desi_proc    --cameras a0123456789 -n 20230516 -e 180972 --mpi --starttime $(date +%s) --timingfile /global/cfs/cdirs/desi/spectro/redux/daily/run/scripts/night/20230516/ccdcalib-20230516-00180972-a0123456789-timing-$SLURM_JOBID.json

if [ $? -eq 0 ]; then
  echo SUCCESS: done at $(date)
else
  echo FAILED: done at $(date)
  exit 1
fi

Compared with the one that is currently being produced by main for the same date:

#!/bin/bash -l

#SBATCH -N 1
#SBATCH --qos regular
#SBATCH --constraint=cpu
#SBATCH --account desi
#SBATCH --job-name ccdcalib-20230516-00180972-a0123456789
#SBATCH --output /global/cfs/cdirs/desi/spectro/redux/banovetz/run/scripts/night/20230516/ccdcalib-20230516-00180972-a0123456789-%j.log
#SBATCH --time=00:25:00
#SBATCH --exclusive

export MPLBACKEND=agg
export OMP_NUM_THREADS=1
# using 128 cores on 1 nodes

echo Starting at $(date)

echo Running srun -N 1 -n 15 -c 17 --cpu-bind=cores desi_compute_dark_night --reference-night=20230516 --camword=a0123456789 --mpi
srun -N 1 -n 15 -c 17 --cpu-bind=cores desi_compute_dark_night --reference-night=20230516 --camword=a0123456789 --mpi

if [ $? -eq 0 ]; then
    echo darknight succeeded at $(date)
else
    echo FAILED: darknight failed, stopping at $(date)
    exit 1
fi

echo Running srun -N 1 -n 30 -c 8 --cpu-bind=cores desi_proc -n 20230516 --cameras a0123456789 -e 180972 --mpi
srun -N 1 -n 30 -c 8 --cpu-bind=cores desi_proc -n 20230516 --cameras a0123456789 -e 180972 --mpi

if [ $? -eq 0 ]; then
    echo badcolumn succeeded at $(date)
else
    echo FAILED: badcolumn failed, stopping at $(date)
    exit 1
fi

echo Running srun -N 1 -n 30 -c 8 --cpu-bind=cores desi_fit_cte_night -n 20230516 -c a0123456789 -e 180972,181010,181011
srun -N 1 -n 30 -c 8 --cpu-bind=cores desi_fit_cte_night -n 20230516 -c a0123456789 -e 180972,181010,181011

if [ $? -eq 0 ]; then
    echo ctecorr succeeded at $(date)
else
    echo FAILED: ctecorr failed, stopping at $(date)
    exit 1
fi


if [ $? -eq 0 ]; then
    echo All done at $(date)
else
    echo FAILED: Script failed, stopping at $(date)
    exit 1
fi

I am not sure if this is a by product of a fresh spec prod and only running for a single date (similar to #2593). I am not sure but maybe @akremin knows better?

[akremin]

@jbanovetz What command(s) are you using to launch these nights? It is complaining because it is using a non-flat to try and fit the CTE. That shouldn't happen, but I need to know what you're telling the pipeline to do before I can debug it.

[jbanovetz]

Sorry, forgot to post the command! Here is what I ran for this last one:

desi_proc_night --night 20230516

This was with a clean specprod (no other days within 30 days were processed).