Scaling identical pipelines, different parameters

[JSteinbauer]

Hi there,
I'd like to know the best way (in your opinion) to apply dagster to the following, probably very common problem:
We are creating customer-specific ML models. That means that, for each customer, we want to

1. Run the same data-generation pipeline, but with different parameters (e.g. different urls for DB connections).
2. Take advantage of asset caching: When re-running the pipeline for a specific set of parameters, we neither want to re-materialize everything nor do we want to use assets from a different set of parameters.
3. Have the capability of launching such pipelines in parallel and
4. Monitor everything via the dagit UI.

Is it (currently) even possible to combine all of these points at the same time?
What would be the preferred way of launching individual runs (e.g. as docker containers, using sensors or GraphQL requests, etc.)? I tried sending different configurations with GraphQL requests to a single dagit instance, but asset caching didn't seem to work that way..
Is it be possible to spawn several pipeline containers and use a single dagit server to monitor all of it?

Thanks a lot for your reply,
Jakob

[sryza]

Hi @JSteinbauer - the first way that I would think to accomplish this would be with partitions. Have you looked at those?

https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions

[JSteinbauer]

Hi @sryza, thanks for your response! I didn't know about partitions. This might solve our problem, however I'm not sure about the following two points:

1. We will deal with graphs of connected assets. If we introduce partitions for all of these assets, will they align automatically? Example: If I have two assets:
   @asset(partitions_def=[1, 2])
def asset1():
...
   and
   @asset(partitions_def=[1, 2])
def asset2(asset1):
...
   will partition 1 of asset2 only materialize with partition 1 of asset1, or will it form some kind of "cross-product" (Which would clearly not be desired).
2. Partitions would only be interesting if they could be created dynamically. I've seen that there is dagster.dynamic_partitioned_config, but the description is not quite extensive. Is it recommended to use it in combination with Sensors, or how can external changes be used to create new partitions?

Thanks a lot for your response,
Jakob

[JSteinbauer]

@sryza Actually this issue seems to be related to #7943 - is there any update on this?
Best,
Jakob

[sryza]

We are hoping to add support for runtime asset partitions within the next quarter.