Filter wham-only DELs and scramble-only SVAs in CleanVcf & docs updates #740

epiercehoffman · 2024-10-29T22:57:54Z

Updates

Soft-filter wham-only deletions and scramble-only SVA insertions with HIGH_SR_BACKGROUND in INFO at the end of CleanVcf with the filter status HIGH_ALGORITHM_FP_RATE
Updates to documentation, including: add notebooks to joint calling documentation, delete joint calling Terra dashboard and just link to workspace, remove ApplyManualVariantFilter from canonical pipeline documentation (replaced by changes to CleanVcf in this PR), fix a few links, small tweaks here and there
Add a 156-sample single-batch Terra sample table in addition to the existing 2-batch 312-sample one
Add VisualizeCnvs Terra JSON and add workflow to dockstore.yml

Testing

Validated all WDLs and JSONs with womtool and Terra validation script
Tested CleanVcf on reference panel and checked for presence of HIGH_ALGORITHM_FP_RATE filter on valid variant sites. No scramble sites are present in the ref panel input VCF to CleanVcf so I also tested this task locally on a VCF with Scramble SVAs to ensure they were flagged as needed
Ran the website on my branch and spot-checked formatting and links

mwalker174

Thanks this is great! I have a few suggestions. I don't love the inline python but it's semi-temporary so will let it go.

mwalker174 · 2024-10-30T14:59:05Z

inputs/templates/terra_workspaces/cohort_mode/samples_1kgp_156.tsv.tmpl

@@ -0,0 +1,157 @@
+entity:sample_id	bam_or_cram_file
+NA10847	gs://fc-56ac46ea-efc4-4683-b6d5-6d95bed41c5e/CCDG_13607/Project_CCDG_13607_B01_GRM_WGS.cram.2019-02-06/Sample_NA10847/analysis/NA10847.final.cram


Can you sort these? So they are the same order as the ref panel jsons

Sure. This one matches the order of the 312-sample one - I could sort that too, although then it would mix the reference panel samples with the rest of the samples, and wouldn't match the batch assignment order

mwalker174 · 2024-10-30T15:38:21Z

website/docs/gs/calling_modes.md

+as joint calling with larger cohorts. Additionally, SV quality will be best when the case sample closely resembles the samples
+in the reference panel.


Suggested change

as joint calling with larger cohorts. Additionally, SV quality will be best when the case sample closely resembles the samples

in the reference panel.

as joint calling with larger cohorts. Additionally, input sequencing data should of similar quality to that of the reference panel, which was generated in [Byrska-Bishop et al. 2022](https://doi.org/10.1016/j.cell.2022.08.004) and is hosted in [this AnVIL workspace](https://app.terra.bio/#workspaces/anvil-datastorage/1000G-high-coverage-2019).

The reason I worded this without referring to our provided reference panel is because we do also give information on how users can build their own reference panel. I recognize most won't, but I felt like it was sufficient to state that samples should resemble the reference panel, and the page describes the provided sample above

Ok if this is redundant don't worry about it, but I think the word "resemble" is a bit vague so just make it clear you're referring to sample quality and sequencing methods.

mwalker174 · 2024-10-30T15:45:52Z

wdl/CleanVcfChromosome.wdl

+
+    python <<CODE
+import pysam
+fin = pysam.VariantFile("~{vcf}")


Suggested change

fin = pysam.VariantFile("~{vcf}")

with pysam.VariantFile("~{vcf}") as fin:

mwalker174 · 2024-10-30T15:46:45Z

wdl/CleanVcfChromosome.wdl

+fin = pysam.VariantFile("~{vcf}")
+header = fin.header
+header.add_line("##FILTER=<ID=HIGH_ALGORITHM_FP_RATE,Description=\"Categories of variants with low specificity including Wham-only deletions and certain Scramble SVAs\">")
+fo = pysam.VariantFile("~{prefix}.vcf.gz", 'w', header=header)


Suggested change

fo = pysam.VariantFile("~{prefix}.vcf.gz", 'w', header=header)

with pysam.VariantFile("~{prefix}.vcf.gz", 'w', header=header) as fo:

mwalker174 · 2024-10-30T15:48:15Z

wdl/CleanVcfChromosome.wdl

+import pysam
+fin = pysam.VariantFile("~{vcf}")
+header = fin.header
+header.add_line("##FILTER=<ID=HIGH_ALGORITHM_FP_RATE,Description=\"Categories of variants with low specificity including Wham-only deletions and certain Scramble SVAs\">")


Suggested change

header.add_line("##FILTER=<ID=HIGH_ALGORITHM_FP_RATE,Description=\"Categories of variants with low specificity including Wham-only deletions and certain Scramble SVAs\">")

header.add_line("##FILTER=<ID=HIGH_ALGORITHM_FDR,Description=\"Categories of variants with low precision including Wham-only deletions and certain Scramble SVAs\">")

I realize I probably suggested this name but FDR is more appropriate

mwalker174 · 2024-10-30T15:48:21Z

wdl/CleanVcfChromosome.wdl

+for record in fin:
+    if (record.info['ALGORITHMS'] == ('wham',) and record.info['SVTYPE'] == 'DEL') or \
+      (record.info['ALGORITHMS'] == ('scramble',) and record.info['HIGH_SR_BACKGROUND'] and record.alts == ('<INS:ME:SVA>',)):
+        record.filter.add('HIGH_ALGORITHM_FP_RATE')


Suggested change

record.filter.add('HIGH_ALGORITHM_FP_RATE')

record.filter.add('HIGH_ALGORITHM_FDR')

mwalker174 · 2024-10-30T15:51:56Z

wdl/CleanVcfChromosome.wdl

+
+  Float input_size = size(vcf, "GiB")
+  RuntimeAttr runtime_default = object {
+    mem_gb: 3.75 + input_size * 1.5,


Was this copy+paste from somewhere? 3.75 should be plenty I think.

Suggested change

mem_gb: 3.75 + input_size * 1.5,

mem_gb: 3.75,

Yeah most of the things you pointed out were from copy/paste, I'll do a better job of cleanup next time. Thanks for catching

mwalker174 · 2024-10-30T15:52:48Z

wdl/CleanVcfChromosome.wdl

+  Float input_size = size(vcf, "GiB")
+  RuntimeAttr runtime_default = object {
+    mem_gb: 3.75 + input_size * 1.5,
+    disk_gb: ceil(100.0 + input_size * 3.0),


Suggested change

disk_gb: ceil(100.0 + input_size * 3.0),

disk_gb: ceil(10 + input_size * 2.0),

…l similarity text

epiercehoffman added 6 commits October 29, 2024 12:59

filter wham-only del & scramble-only svas; docs updates

424c103

fix conflict

b13f6b0

add 1-batch terra sample table; add viscnvs to dockstore yml

84c5f85

tweaks

151b591

delete cohort mode dashboard and just link to website

e49c643

increase terra json count with viscnvs

f408d6d

mwalker174 requested changes Oct 30, 2024

View reviewed changes

reviewer comments; add package-lock.json to gitignore

a4b964f

mwalker174 approved these changes Oct 30, 2024

View reviewed changes

use v1.0 diagram; add package-lock.json to gitignore; update ref pane…

c878e59

…l similarity text

epiercehoffman merged commit aef6ac9 into main Oct 31, 2024
8 checks passed

epiercehoffman deleted the eph_high_alg_fp_filt branch October 31, 2024 14:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filter wham-only DELs and scramble-only SVAs in CleanVcf & docs updates #740

Filter wham-only DELs and scramble-only SVAs in CleanVcf & docs updates #740

epiercehoffman commented Oct 29, 2024 •

edited

Loading

mwalker174 left a comment

mwalker174 Oct 30, 2024

epiercehoffman Oct 30, 2024

mwalker174 Oct 30, 2024

epiercehoffman Oct 30, 2024

mwalker174 Oct 30, 2024

mwalker174 Oct 30, 2024

mwalker174 Oct 30, 2024

mwalker174 Oct 30, 2024

mwalker174 Oct 30, 2024

mwalker174 Oct 30, 2024

mwalker174 Oct 30, 2024

epiercehoffman Oct 30, 2024

mwalker174 Oct 30, 2024

		@@ -0,0 +1,157 @@
		entity:sample_id bam_or_cram_file
		NA10847 gs://fc-56ac46ea-efc4-4683-b6d5-6d95bed41c5e/CCDG_13607/Project_CCDG_13607_B01_GRM_WGS.cram.2019-02-06/Sample_NA10847/analysis/NA10847.final.cram

		as joint calling with larger cohorts. Additionally, SV quality will be best when the case sample closely resembles the samples
		in the reference panel.

	fin = pysam.VariantFile("~{vcf}")
	with pysam.VariantFile("~{vcf}") as fin:

	fo = pysam.VariantFile("~{prefix}.vcf.gz", 'w', header=header)
	with pysam.VariantFile("~{prefix}.vcf.gz", 'w', header=header) as fo:

	header.add_line("##FILTER=<ID=HIGH_ALGORITHM_FP_RATE,Description=\"Categories of variants with low specificity including Wham-only deletions and certain Scramble SVAs\">")
	header.add_line("##FILTER=<ID=HIGH_ALGORITHM_FDR,Description=\"Categories of variants with low precision including Wham-only deletions and certain Scramble SVAs\">")

	record.filter.add('HIGH_ALGORITHM_FP_RATE')
	record.filter.add('HIGH_ALGORITHM_FDR')

	disk_gb: ceil(100.0 + input_size * 3.0),
	disk_gb: ceil(10 + input_size * 2.0),

Filter wham-only DELs and scramble-only SVAs in CleanVcf & docs updates #740

Filter wham-only DELs and scramble-only SVAs in CleanVcf & docs updates #740

Conversation

epiercehoffman commented Oct 29, 2024 • edited Loading

Updates

Testing

mwalker174 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epiercehoffman commented Oct 29, 2024 •

edited

Loading