Expanding options for --tasks to include memory allocation

[abhijithbiji]

Could the task file in future updates include ways to also specify different --min-ram, --min-cores, and --disk-size options for subtasks? Eventually, could this also be done for most available dsub flags? Thank you

[wnojopra]

Hi @abhijithbiji , this is a pretty good idea and something we have thought about before. In particular, I've had workflows where my necessary disk-size per input would differ. But because the cost of just having more disk size is pretty small, we opted for that.

We haven't had much concrete use cases for this, can I hear more about your use case?

[abhijithbiji]

Hi @wnojopra. Thanks for the reply

In my case, all my use cases are related to workflows involving genomic data; where the disk size for larger chromosomes like 1 & 2 are close to 800 GB and that of smaller ones like 22 are close to 50 GB. Since these jobs can run for days, I can save significant costs by performing them with (1) variable --disk-size and --preemptible option and (2) having --retries option. Since I cannot have --disk-size in my parameter file, I can only do either (1) or (2), prolonging my analyses.

Same applies for --min-ram, --min-cores and even --machine-type