Setting Task resources dynamically within a workflow

[esadler-hbo]

Hello! I read through the docs and couldn't figure out an answer to this...

I would like to reuse tasks, however sometimes I will need to request different sized machines depending on the input data size. I could imagine a world where we pass in resource configurations to a workflow, but that would require overriding what is set in @task(...). Kubeflow pipelines has a way to do this dynamically, hoping Flyte does too.

Here is some sudo code:

@task
def preprocess(query: str) -> List[str]:
   ...

@dynamic
def workflow(query: str):
    
   # Is there a way to overwrite the resources here?
   transformed = preprocess(query)

I am very appreciative of any advice on how to do this in the 'Flyte' way!

[SmritiSatyanV]

Hi @esadler-hbo , is this something you are looking for-> flyteorg/flytesnacks#818