Efficiently materializing a large set of partitions in kubernetes

[the4thamigo-uk]

What is the approved pattern to use in k8s, when we want to avoid creating many pods for materialising a large set of partitions for a given asset?

I have a process I need to run which determines which partitions need to be processed and initiates materialization for each of those partitions. The materialization is fairly quick and I would like to run these materializations without the overhead of starting a new pod for each one.

As a workaround, I have been using the materialize() function, which I call from a loop within an unpartitioned asset, which works quite well, except for the fact I frequently get misleading errors like this : #19554.