Large Workflow Failures

[SandraGH5]

What might cause a workflow to fail with this error:

failed at Node[n1]. EventRecordingFailed: failed to record node event, caused by: ResourceExhausted: 
Events are sent too often, exceeded the rate limit, 
caused by [rpc error: code = ResourceExhausted desc = grpc: received message larger than
max (7848897 vs. 4194304)]

@jeevb

[SandraGH5]

The compiled workflow closures might be too big - especially the number of unique tasks.

In this case, the workflow only had 200 unique tasks, but each task had a list of around 300 strings, each 125 bytes long, resulting in a future file of about 7MB (200 * 300 * 125).

A solution could be to offload them as a FlyteFile or a new type transformer for an offloaded list - backed by a file.

@kumare3
@katrogan