publishDir overwrite does not work during resumed runs

[zanieb]

Bug report

Expected behavior and actual behavior

publishDir "${params.outdir}/fastq", mode: 'copy', pattern: 'fastq-merge-lanes/*.fastq', overwrite: true, saveAs: { "${sample_name}.fastq" }`

should overwrite files in outdir but during resume a warning appears

WARN: Failed to publish file: ... to: ... [copy] -- See log file for details

The log clarifies that the file already exists in the output directory.

When -resume is not used, no errors occur. Nextflow appears to be overwriting the specified value for overwrite with its default.

https://www.nextflow.io/docs/latest/process.html#publishdir

Steps to reproduce the problem

1. Create a process that publishes with overwrite: true
2. Run nextflow
3. Run nextflow again with -resume
4. Failure to publish should ensue

Environment

• Nextflow version:

      N E X T F L O W
      version 19.04.1 build 5072
      last modified 03-05-2019 12:29 UTC (07:29 CDT)
      cite doi:10.1038/nbt.3820
      http://nextflow.io

• Java version:

openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

• Operating system: AWS Linux

[Z-Zen]

I am not sure if this is a bug unless I misunderstood your case, I apologize in advance...

The -resume flag is supposed to restart from the process it failed which means that all the previous processes that succeeded will be skipped.

He will resume on the process of where he failed. In that case, why would you need to overwrite the results since anyway the process hasn't succeeded and thus he didn't copy the files to the publishDir ?

I think the overwrite only if the pipeline is run from the start again...

[zanieb]

I'm a bit confused by your response, but I'll try to clarify.

I have two processes: process_a and process_b. They both write the same files to the same location but process_b has done additional work on the data. This works such that if I have disabled process_b in the settings, the files for a non-nextflow next step will still exist. This model requires the processes to always overwrite the files in the output. If I use -resume after a failed run (let's say it fails between a and b, process_b cannot overwrite the output from process_a in the first run. This is a bug in the sense that nextflow is not behaving as I have explicitly commanded it to by overriding the default value of overwrite. If the -resume behavior of overwrite: false is desired, then overwrite should not be overridden in your process. However, if overwrite is explicitly set to true, then it should be honored.

[pditommaso]

Please provide a self-contained test case.

[zanieb]

I attempted to construct a self-containted use-case and have failed to replicate although it is occurring in my actual pipeline. Here's my passing script:

params.skip_bar = false

process foo {
    publishDir "output", mode: 'copy', overwrite: true, saveAs: { "test_rename.tmp" }

    output:
    file 'test_foo.tmp'
    val true into ch_bar_flag

    script:
    """
    echo 'foo' > test_foo.tmp
    """
}

process bar {
    publishDir "output", mode: 'copy', overwrite: true, saveAs: { "test_rename.tmp" }

    input:
    val flag from ch_bar_flag

    when:
    !params.skip_bar

    output:
    file 'test_bar.tmp'

    script:
    """
    echo 'bar' > test_bar.tmp
    """
}

I also attempted having a failed process instead of a skipped process with the same results

    script:
    """
    if [ ${params.skip_bar} = "true" ]; then
        exit 125
    fi
    echo 'bar' > test_bar.tmp
    """
}

I suspect the problem may exist with more than 2 processes / conditional middle processes which will take longer to explore.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreas-wilm]

Hi all,

I'm experiencing the same problem (with v 21.04.0.5552), i.e. I get java.nio.file.FileAlreadyExistsException: target already exists. The target already exists because it's a resumed run, where the process in question defines a publishDir with mode: 'copy', overwrite: true and saveAs. I cannot reproduce in small self-contained example.

Andreas

[pditommaso]

Hi Andreas, feel free to re-open including the .nextflow.log file reporting this problem