Issues with running on Slurm

[haraldgrove]

Hello

I'm trying to run Princess on a cluster managed by Slurm. I've followed the 4 steps indicated to change the configuration files and the minimap job has been submitted and is running.

However, despite the configuration file specifying 12 CPUs, it seems like the job only requested 3 CPUs on the cluster and the minimap command line only specifies 3 threads. Are there any more settings I need to change to increase the number of threads used and requested?

The job running on the cluster:
minimap2 -Y -R @RG\tSM:SAMPLE\tID:SAMPLE -ax map-ont /mnt/ScratchProjects/Causative/reference/bovine_ARS-UCD1.2/GCF_002263795.2_ARS-UCD1.3_genomic.fna.gz /mnt/ScratchProjects/Causative/bovine_11978/princess/filtlong_11978.fq.gz --MD -t 3 -y

The command to submit to the cluster:
sbatch --parsable --job-name=snakejob.minimap2 -n 3 --mem=20G --partition=smallmem --time=72:00:00 /net/fs-2/scale/OrionStore/ScratchProjects/Causative/bovine_11978/princess/.snakemake/tmp.tqfwec9x/snakejob.minimap2.3.sh

[MeHelmy]

Hi @haraldgrove,

Thank you for your comment.

There are two configuration files that you need to modify:

1. The first file is the job configuration on the cluster, which specifies the number of CPUs. You can find it here:

minimap2:
  queue: *long_queue
  time: "72:00:00"
  nCPUs: "12"
  mem: 20G

As you mentioned, it currently has "nCPUs: "12". You can change this value to the desired number of CPUs for your cluster.

2. The second file is the tool configuration, located at princess/config.yaml. In this file, you need to adjust the number of threads for the aligner (in this case, Minimap2). Here is the relevant section:

# Aligner
#########
aligner: "minimap" # minimap or ngmlr
aligner_threads: 3
#minimap_other_tags: ""
minimap_other_tags: "-y"
#########

The line aligner_threads: 3 currently specifies 3 threads for the aligner. You can modify this value to match the appropriate number of threads for your cluster. Additionally, if there is no methylation tags in your FASTA file I would suggest to use minimap_other_tags: "" instead of minimap_other_tags: "-y"

Feel free to reach out if you have any further questions.

Best regards,
Medhat

[haraldgrove]

Thank you for your response, I'd missed the tool configuration file.

Edit: I see I was wrong about my interpretation of the nCPUs parameter. The requested CPUs seems to follow the parameters in the 'config.yaml' file. In that case, what does the nCPUs parameter regulate?

However, maybe I don't understand how Slurm works, but when I check the allocated CPUs with sacct it only displays 3, even if the nCPUs parameter in the config file is set to 12. I assumed that was because of the -n 3 part of the sbatch command, and that should have been set to the number of nCPUs?

Best regards
Harald

[MeHelmy]

You didn't misunderstand. In the context of using Snakemake (which I am using implicitly in Princess) to spawn multiple jobs and control cluster submission, the job configuration takes precedence over the cluster configuration. Therefore, it will choose 3 based on the job configuration instead of 12 from the cluster configuration.

Best,
Medhat