Module/gatk rnaseq/1.0 by Jwong684 · Pull Request #184 · LCR-BCCRC/lcr-modules

Jwong684 · 2021-05-07T22:29:40Z

Pull Request Checklists

Important: When opening a pull request, keep only the applicable checklist and delete all other sections.

Checklist for New Module

Required

If applicable

I added more granular output subdirectories.
I added rules to the reference_files workflow to generate any new reference files.
I added subdirectories with large intermediate files to the list of scratch_subdirectories in the default.yaml configuration file.
I updated the list of available wildcards for the input files in the default.yaml configuration file.

Checklist for Updated Module

To be completed.

Kdreval

Thanks Jasper! I left some comments. I have another general question: there is {pair_status} wildcard, but no reference to normal samples in the module. RNASeq would be unpaired, so do we need this wildcard to be included?

Kdreval · 2021-05-08T07:17:25Z

+    gatk_rnaseq:
+        inputs:
+            sample_bam: "data/{sample_id}.bam"
+            sample_bai: "data/{sample_id}.bam.bai"


This needs new line added at the end

It still shows no new line here, maybe I am looking at outdated file?

Sorry, I must have missed it. It's fixed

Kdreval · 2021-05-08T07:17:48Z

+                window: 35 # window size between SNPs in cluster
+                cluster_size: 3 # at least 3 SNPs in cluster
+                # hard filtering (filters OUT) based on metrics: 
+                    # FS (FisherStrand): Phred-scale probability that there is a strand bias from a Fisher's test. (default FS > 30.0)


Thanks for detailed documentation!

Kdreval · 2021-05-08T07:20:04Z

+            bcftools: "{MODSDIR}/envs/bcftools-1.10.2.yaml"
+
+        threads:
+            gatk_splitntrim: 12


It is possible to combine these keys and reuse them across rules. In other words, if several rules need the same number of threads, they can refer to the same key in config. Same can be applied to resources as well. I think this reduces the number of keys to specify/adjust if needed, and reduces complexity of the config. What do you think?

It's possible, but wouldn't it also cause confusion if there's ever a need to change the numbers? Also, would there be a unique name that could be applied to a subset of the rules ("thread_12" would be non-descriptive and wouldn't indicate which rules would use this parameter)?

You are right, let's keep the more detailed and informative names

Kdreval · 2021-05-08T07:25:14Z

@@ -0,0 +1,36 @@
+name: test-bcftools


This (and another environment file) should be rather in lcr-modules/envs/ folder, and symlinked in the module. This will make it easier to reuse same environments across modules, therefore reducing the need to build multiple environments. There is already an environment for bcftools with the same version, so you can just as well symlink lcr-modules/envs/bcftools/bcftools-1.10.2.yaml to this module

Kdreval · 2021-05-08T07:28:33Z

@@ -0,0 +1,352 @@
+name: bioinformatics


Is this environment just GATK or there is something else besides it that is required for this module to run? There is a GATK environment lcr-modules/envs/gatk/gatk-4.1.8.1.yaml, maybe it can be reused here? I see this one contains tools like, for example, bedtools, vcf2maf, tmux, and a lot of perl dependencies, but are they used in the module?

Hm that is a good question. I pulled it from a working environment. I'll test to see if it'll work with the gatk module

Yes it worked with gatk module. I will just use the symlinked version for both gatk and bcftools

Kdreval · 2021-05-08T07:33:07Z

+        stdout = CFG["logs"]["gatk_splitntrim"] + "{seq_type}--{genome_build}--{pair_status}/{sample_id}.gatk_splitntrim.stdout.log",
+        stderr = CFG["logs"]["gatk_splitntrim"] + "{seq_type}--{genome_build}--{pair_status}/{sample_id}.gatk_splitntrim.stderr.log"
+    params:
+        mem_mb = lambda wildcards, resources: int(resources.mem_mb / 1000)


For tools running java it is sometimes useful to include heat buffer, otherwise they might jump to excessive memory usage and will get killed by slurm. I mean the same approach as you did for the rule that merges vcf files. Also, here you can give -Xmx memory in m, so this calculation may not be necessary. Same can be applied to the rules below as well

Changed all -Xmx {params.mem_mb}G to -Xmx {params.mem_mb}m with same calculation (0.8 * assigned mem_mb)

Kdreval · 2021-05-08T07:39:41Z

+        **CFG["resources"]["gatk_base_recalibration"]
+    shell:
+        op.as_one_line("""
+        gatk --java-options "-Xmx{params.mem_mb}G" BaseRecalibrator 


I wonder if you know whether any rule is using java garbage collector, which may by default use all available threads? If so, it may create a lot of competitive usage when launched for many samples simultaneously. One way to get around this is to provide -XX:ConcGCThreads=1 option that basically limits java to only 1 thread for garbage collector. Please ignore this comment if there is no issue with garbage collector!

Hm I'm not sure about any rules specifically. I do see GATK documentation about some functions sometimes using too many resources. I will add -XX:ConcGCThreads=1 as a pre-emptive measure

Kdreval · 2021-05-08T07:40:41Z

+
+rule _gatk_base_recalibration:
+    input:
+        bam = rules._gatk_splitntrim.output,


can you please wrap the inputs referring to the outputs of other rules with str()?

Kdreval · 2021-05-08T07:44:42Z

+        bai = rules._gatk_applybqsr.output.bai,
+        fasta = reference_files("genomes/{genome_build}/genome_fasta/genome.fa")
+    output:
+        vcf = temp(CFG["dirs"]["gatk_variant_calling"] + "{seq_type}--{genome_build}--{pair_status}/{sample_id}.{chrom}.vcf.gz"),


Does HaplotypeCaller and VariantFiltration compress vcf files by default, or do you need to do that as a separate step? I was having impression that majority of variant calling tools do not compress by default, but may be very wrong!

Yes, the output files are gzipped (tested with: file results/gatk_rnaseq-1.0/06-gatk_variant_filtration/mrna--grch37--no_normal/Toledo.filtered.vcf.gz): output: gzip compressed data, extra field

Kdreval · 2021-05-08T07:55:00Z

+        bam = CFG["dirs"]["inputs"] + "bam/{seq_type}--{genome_build}/{sample_id}.bam",
+        fasta = reference_files("genomes/{genome_build}/genome_fasta/genome.fa")
+    output:
+        bam = temp(CFG["dirs"]["gatk_splitntrim"] +  "bam/{seq_type}--{genome_build}--{pair_status}/{sample_id}.split_reassign_mq.bam")


What do you think about making it seq_type}--{genome_build}/{sample_id}--{pair_status}? This would make it consistent eith the directory structure of other modules

I prefer making the final file name simple like: Toledo.ouput.filt.vcf.gz.

You're right about no-normal pair status. Pair status just comes out as "no_normal" in the output directories. I'm not sure if it'd be informative or not to keep it in, but I can remove it altogether too

Jwong684 · 2021-05-09T20:21:50Z

Removed pair-status as a wildcard

… fixed a final gnomad-filtration step.

Kdreval · 2022-01-22T00:10:30Z

@Jwong684 is this ready to go?

Jwong684 · 2022-01-22T00:22:05Z

This should be completed - I have some output tested on the BL/DLBCL cell lines in GAMBL from way before. I want to add a grouping resource so it doesn't get clogged up with temp bams in the middle of the pipeline

Kdreval · 2022-01-22T00:36:24Z

Great, thanks! It shows some merge conflicts with the recent update - can you resolve them and we'll get this on master?

Kdreval · 2024-07-26T16:42:32Z

@Jwong684 what should we do with this? Is this still in works? Should we close this PR? Any plans on continuing this?

Jwong684 · 2024-07-26T20:11:42Z

It is functional but not that great. The final vcf becomes way too big for this to be implemented on a large GAMBL-wide scale. There are just too many SNVs that are called with RNA-seq.

I don't have a particular interest in variants in RNAseq at the moment, but it might be useful as a verification tool. It does capture many of the same SNPs as you would see in WES in the same sample.

Jwong684 added 9 commits April 23, 2021 14:26

Add initial draft of gatk_rnaseq version 1.0

e8874a5

Split gatk variant calling into per chromosome to speed up runs

c1f7318

Split up haplotype calling per chrom then merge

f0310f4

pulling from master

b760336

removed an outdated env yaml file

e2b4623

removes unnecessary lines

83342ab

changelog updates

930fb25

Documented changelog

cc95c65

Updated with new lcr-module checks

12d743f

Jwong684 requested review from Kdreval and rdmorin May 7, 2021 22:29

Kdreval reviewed May 8, 2021

View reviewed changes

Updates to pull-request changes

361c0b7

Added a gnomad-filter step

d15b7d5

Kdreval mentioned this pull request May 14, 2021

RNAseq Variant Calling Pipeline #23

Closed

added one line to config

1a11e3f

Kdreval previously approved these changes May 21, 2021

View reviewed changes

Added GATK_params as an extra param option for flexibility. Added and…

dc4620f

… fixed a final gnomad-filtration step.

Jwong684 dismissed Kdreval’s stale review via dc4620f May 31, 2021 08:24

Added an RG rule before trimming

4abb545

fixing commiting too many files

f422933

Jwong684 force-pushed the module/gatk_rnaseq/1.0 branch from eb9875d to f422933 Compare February 18, 2022 22:32

Jwong684 added 3 commits February 18, 2022 14:38

trying to merge master into branch

4acda94

updated the demo WGS bam links

f697c8f

Updated defaults and added AD and biallelic snp filtering step

c299580

Uh oh!

Conversation

Jwong684 commented May 7, 2021

Pull Request Checklists

Checklist for New Module

Required

If applicable

Checklist for Updated Module

Uh oh!

Kdreval left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jwong684 commented May 9, 2021

Uh oh!

Kdreval commented Jan 22, 2022

Uh oh!

Jwong684 commented Jan 22, 2022

Uh oh!

Kdreval commented Jan 22, 2022

Uh oh!

Kdreval commented Jul 26, 2024

Uh oh!

Jwong684 commented Jul 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants