Strelka2 stuck at 99% for over 24 hours (reopen)

[Tina04021997]

Dear developer,

Unfortunately, there has no feedback from Slack so far, and I would like to reopen the previous issue attached with my log file (Strelka2 stuck at 99% for over 24 hours #2123 ).
Additionally, all config files remain untouched (git cloned from Sarek).
Thank you again for your time!

Original issue:
Dear developer,

Thank you for working on this tool. I have a question regarding running Strelka.
I am test-running a pair of tumor-normal TCGA WES samples, and the process has been stuck at STRELKA SOMATIC for over 24 hours (it shows 99% for over 10-ish hours). There is no error message, and the pipeline continues running. However, I believe for TCGA WES, one pair should perhaps take less than 10 hours.
I have tried several times with v3.7.0 or v3.5.0, but all ended up being stuck at the Strelka2 process.

System: Slurm 23.02.7
Sarek version: 3.7.0
Memory and threads allocated: entire node with 64 threads, 1 T memory.
Command executed:

export TMPDIR=${tmp}/temp_F1_Sarek

NXF_OPTS='-Xms1g -Xmx4g'

nextflow run nf-core/sarek -r 3.7.0 -profile conda \
--input ./samplesheet.csv --step mapping --outdir ./results \
--aligner bwa-mem2 --genome GATK.GRCh38 \
--tools mutect2,strelka -c nextflow.config -with-report --dbsnp ${database}/GRCh38/resources_broad_hg38_v0_Homo_sapiens_assembly38.dbsnp138.vcf --known_indels ${database}/GRCh38/Homo_sapiens_assembly38.known_indels.vcf.gz --pon ${database}/GRCh38/PON.sorted.vcf.gz --pon_tbi${database}/GRCh38/PON.sorted.vcf.gz.tbi --germline_resource${database}/GRCh38/af-only-gnomad.hg38.vcf.gz --wes --intervals ${database}/GRCh38/GRCh38_exome.bed --only_paired_variant_calling

nextflow_sarek.log.zip

[FriederikeHanssen]

Duplicate of #2123

Did you try what Famke suggested in the previous issue?
What happens if you cancel and resume the job? How many samples are you processing?

It looks like you are not using the slurm scheduler but running everything on a local node. Is that correct?

[Tina04021997]

Hi! Yes, I try to run one tumor-normal matched TCGA WES without using the scheduler (everything on a local node).
I canceled and re-ran two times, but all ended up being stuck at STRELKA for 20+ hours.
Our cluster cannot do Docker, and I haven't tried Singularity yet.

[FriederikeHanssen]

I didn't see anywhere in your config that your using the scheduler. Could you share that config?

[Tina04021997]

Absolutely! I'm not using a scheduler but running everything on a local node (64 threads 1T).
Do you think this would be a scheduler issue or a conda issue? I am test-running on using a scheduler.
Thank you so much for your time.

config.zip