Filter outlier samples in TrainGCNV and GenerateBatchMetrics

[kjaisingh]

This PR addresses Issue #44.

Description

TrainGCNV

• Replaces Array[String]? sample_ids_training_subset with File? outlier_sample_ids, as the former was not being actively used before. The new parameter represents a text file containing outlier samples.
• Creates and calls task Utils.GetFilteredSubsampledIndices that uses outlier_sample_ids and/or outlier_sample_ids to subsample based on the following logic:
  • If only exclude_strings is provided, then only include samples that are not part of the outlier_sample_ids file.
  • If only subset_size is provided, then only include only n_samples_subsample random samples.
  • If both exclude_strings and subset_size are provided, then n_samples_subsample random samples that are not part of the outlier_sample_ids file.
  • If neither exclude_strings nor subset_size are provided, include all samples.
• Trains GCNV model exclusively using the subset of samples produced by Utils.GetFilteredSubsampledIndices.

GenerateBatchMetrics

• Creates new input parameter File? outlier_sample_ids to represent a text file containing outlier samples.
• Modifies existing PE, RD, BAF and SR tests such that they accept parameter--outlier-sample-ids, which provides a path to a file containing outlier samples. The metric calculation process in these tests are modified as follows:
  • Outlier samples are not considered to be called for variants where at least 1 non-outlier was called.
  • Outlier samples are never considered to be part of background samples.
• Finally, we populate the existing column is_outlier_specific in the output VCF with True if it corresponds to a site that only has outlier samples tagged to it, and with False if not.

FilterBatchSites
No changes included - we already filter out sites from the training process using the is_outlier_specific = True tag that we now populate in GenerateBatchMetrics.

Testing

• TrainGCNV: This Terra job passes an outlier file that includes a subset of 6 samples out of a cohort containing 156 samples. We can see in the count_entity_ids parameter for CNVGermlineCohortWorkflow that only 150 samples are passed into the model training step - none of which are listed in the outlier sample file.
• GenerateBatchMetrics: This Terra job passes an outlier file, while this Terra job calls the same WDL albeit without the outlier file. We can see in the .metrics and .stats files produced across tests due to the presence of outliers, the called variants have varying levels of support.
• Validated all WDLs with womtool.

Pre-Merge Changes Required

• Remove changes to .dockstore.yml.
• Output a exclude_sample_ids.txt file from the EvidenceQC notebook - samples part of this should not be dropped from the metrics table though.