ContainerTask run error in copilot

[SandraGH5]

Bernhard Stadlbauer:

When using a ContainerTask to run some tasks, some of the inputs that the container receives are ~500mb. When running the Task, the flyte-copilot-downloader container in the pod fails with:

{"json":{},"level":"error","msg":"Failed to persist [<INPUT_NAME>], err failed to read Blob from storage: [LIMIT_EXCEEDED] limit exceeded. 479mb \u003e 2mb.","ts":"2021-07-16T09:06:19Z"}

When this happens, the Task does not fail directly (which would be nice), but only when we try to open the input file in the actual container on the pod.
Is there a way to increase this limit?

[SandraGH5]

Samuel Kogler managed to fix the issue by adding:

limits:
        maxDownloadMBs: 100000

This only needs to be updated for flyte-data-config in https://github.com/flyteorg/flyte/blob/master/helm/templates/_helpers.tpl#L207-L216.

Beyond that we are going to change a few things:

1. Ideally, there should be no limit for downloads in copilot (you can get terabytes of data if you want for example). To support this, we will be allowing a default unlimited download support here - https://github.com/flyteorg/flytestdlib/blob/master/storage/config.go#L103
2. Also, we will get rid of having to deploy flyte-data-config, as mounting a config map is expensive and configuring it is error prone. We will automatically configure data copilot from flytepropeller. https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/pluginmachinery/flytek8s/copilot.go This can be configured using command line args to the copilot init container and sidecar.

@evalsocket will be available for assistance.
Reference: Flyte CoPilot doc

@kumare3