Catch specific slurm errors?

[baxpr]

E.g. exit code of 0:53 from slurm means an I/O error, and we might not get an outlog to debug this. Any way we could capture this info to make it more visible? Goal would be to easily distinguish jobs that failed due to cluster problem and just need a re-run, vs jobs that failed due to internal issues and need to have the outlog reviewed.

If this is captured in the qcstatus on xnat, we could automatically reset based on qcstatus, or similar. Either command line tool, or some process (or extra cron job) that restarted cluster related failures automatically (up to X times, to avoid infinite submissions).

[duettwe]

I have a few questions/thoughts

1. The exit code is from the attempted launch of the slurm file?
2. Is anyone actually using the qcstatus of assessors?
3. Assuming we use qcstatus, what are we putting? We could do something like this:

{exit_code} - 1
{exit_code} - 2
{exit_code} - 3
JOB_FAILED/COMPLETE

4. What do we want the qcstatus to be if we DON'T want a relaunch but needs review? If it needs review, immediately set to JOB_FAILED?
5. Do we ever get any non-zero exit status other than 0:53? Those would need to be handled specifically per code?

[duettwe]

Actually, we pretty much already have a way to get the exit codes: https://github.com/VUIIS/dax_local/blob/master/slurm_failed_jobs.sh

All we need to do is add ExitCode on line 12. This will generate something like this in the nightly emails

HeckersESOP-x-2-001068-25-x-v000mo_261804-x-rois-hurst_v2-x-cbaf92b1.slurm 9895788 FAILED 0:53
HeckersESOP-x-2-001070-25-x-v000mo_261812-x-rois-hurst_v2-x-69f8d2d4.slurm 9885353 FAILED 0:53
HeckersESOP-x-2-001071-25-x-v000mo_261846-x-rois-hurst_v2-x-02605f1e.slurm 9885355 FAILED 0:53
HeckersESOP-x-2-001071-25-x-v000mo_261846-x-rois-hurst_v2-x-e3ba41f0.slurm 9895790 FAILED 0:53

We could use this, or something similar, to see which processors are I/O errors.... and it looks like all of them from the last 26 have the exitcode 0:53.

[baxpr]

Well that's handy. Can you go ahead and make this update to the nightly script?

Related, I heard today that ACCRE is putting in some guardrails to deal with the underlying issue as well. So maybe this will soon be less of a problem.

qcstatus is being used by at least a couple of teams that I know of. If dax only sets it when uploading a result back to XNAT, though, that wouldn't get in the way of the other uses because it wouldn't overwrite a relevant existing entry.

[duettwe]

https://github.com/VUIIS/dax_local/pull/237

[duettwe]

Merged and pulled. This might be good enough for now. Hopefully, we don't have so many that they couldn't be restarted manually.