Pipeline completed with errors ends with correct exit-code

[mewu3]

Don't know if this post is valid format, please let me know if i have to re-edit

New feature

I have noticed that nextflow pipeline exit-code does not match with the actual exit situation, is this a design on purpose? I personally find this rather inconvenient, if i want to have a quick look of the exit state of pipeline , i will have to look at the log thoroughly.
I would really like that the pipeline give an exit-code that match with the exit state, for example:
- "Workflow failed on launch: Launcher error" -> 1
- "Workflow failed on launch: Operation aborted" -> 1
- "Pipeline completed with errors" -> 1
- etc.

Use case

My use case is i have a shell script for retry a failed workflow for at most 3 times, so i have to judge the exit state, if i can't have the correct exit-code recorded, i will have to scramble the log file for key word or key phrases. which i find most unreliable - cause i will have to get to know all the key word

Suggested implementation

[bentsherman]

Can you provide a minimal reproducible example?

I assume you mean that a pipeline is failing but returning a zero exit code, that is not clear from your description

[mewu3]

Can you provide a minimal reproducible example?

I assume you mean that a pipeline is failing but returning a zero exit code, that is not clear from your description

Yes that is what i wanted to say, sorry for not being clear.

As you can see here, nextflow workflow actually did not run properly cause env variable TOWER_ACCESS_TOKEN is missing, yet the exit code of the process still remains 0.

I only manage to handle these two situations by re-assign the exitcode, because I have only encountered these two situations.

To reproduce you just have to run whatever pipeline you have with -with-tower without providing TOWER_ACCESS_TOKEN, and record the exitcode with $? and you will get 0.

The nextflow version that I used is 23.10.1

[bentsherman]

@jorgee do you think this would be solved by #6827 ? It looks like the same problem of errors from trace observers not being propagated to the top

[jorgee]

Yes, this is what I saw in my tests, and with #6827, it will fail and return exit code 1. However, I think errors in the notifications are intentionally ignored. In this case ( and I suppose in the case of the issue), the pipeline continued running and finished correctly.

[bentsherman]

It may be that exceptions from trace observers were ignored because they were historically considered "not essential" i.e. not worth failing the run over

But now they have become essential, as Nextflow runs are incorporated into larger software systems, the errors need to be surfaced as a pipeline failure or else the user won't notice it

So I think it's fair to just propagate all exceptions from trace observers -- then it's up to the plugin developer (or us) to decide whether an error in a trace observer is worth throwing up to the runtime

[jorgee]

What I have implemented in #6827 is that AbortRunExceptions are propagated others are just logged. I added a TOWER_ABORT_ON_ERROR to throw AbortRunExceptions instead of other exceptions to produce a failure. I just did it to not introduce side effects in other observers. However, based on what you commented (forward whatever exception), I think it is better to generalize this approach; we can use the env var NXF_ABORT_ON_NOTIFY_EVENT_ERROR to enable/disable failures and enable by default. If someone is getting a strange behaviour with event notification exceptions, he/she can easily disable it.