All MarkDuplicates tasks run out of memory after addition of Xmx flags

[michaelgatzen]

Probably since the addition of the Xmx flags for Java tasks, all MarkDuplicates tasks run out of memory for WGS samples for me. Unfortunately, the way that they are specified, Cromwell's retry with more memory feature does not work because the value of the Xmx argument is only dependent on an input, not the actual (possibly increased) memory. This relates to #481.

[michaelgatzen]

Note that in addition to the failing workflows, this can lead to very high costs for users. If they have the retry with more memory feature enabled, Cromwell will detect that the task runs out of memory and will increase it, but the task will keep failing because the maximum memory that Java can consume won't change. Since this increase is automatic and exponential, Cromwell will end up requesting insane amounts of memory

[jessicaway]

@michaelgatzen I'll take a look at this. Thanks for pointing it out!

[droazen]

@michaelgatzen @jessicaway Isn't the fix just to make the -Xmx value relative to the total memory on the instance (eg., instance_memory - 1 GB), like we did recently for the HaplotypeCaller task in the DRAGEN-GATK pipeline?

[michaelgatzen]

@jessicaway Just letting you know, CollectRawWgsMetrics also runs out of memory for about 50% of samples for me

[jessicaway]

@michaelgatzen I have changed this back for now. We will try to reintroduce the Xmx for CollectRawWgsMetrics and MarkDuplicates later (with more testing!)

[gbggrant]

We think this is all set, with appropriate follow-up tickets created.