Parallelize Preprocessing pipeline

[SofiaTorchia]

Hello,

I am working on the UKB Rap and I would like to run the preprocessing pipeline on all available .vcf.gz files. I was wondering if there is a way to easily parallelize the pipeline and run one job for .vcf.gz file so that the output of each job can be pieced together as if I ran the pipeline on all files with one job.

Thank you so much,
Sofia

[bfclarke]

Hi Sofia, sorry for the slow reply.

Unfortunately, because the UKB RAP doesn't support Snakemake, this isn't terribly straightforward. I think what you could do would be to first run the command in the preprocessing rule extract_samples to get the list of samples from the VCF files (it should be the same in every file, so you just need to run it on one of them).

Then, parallelize these steps, which are done for each VCF file separately:

1. Run each command in the QC pipeline (except process_individual_missingness), making sure to upload the outlet back to the RAP in the same working directory structure as Snakemake would use.
2. Run the rules normalize and sparsify in the preprocessing pipeline.

For parallelizing across files, you might find this script helpful: https://github.com/endast/DXSub

Sorry to not have a simpler solution, but do let me know if you have any questions.

[SofiaTorchia]

Hi Brian, thank you so much for your reply! I will try and will let you know!

Sofia