SubDAGs / Dependent DAGs(pipelines)

[ankxyz]

Hello.

Does Dagster allow to build SubDAGs / Dependent pipelines: pipeline B must be executed just when pipeline A will finished (A >> B in Airflow notation)?

I tried two approaches to execute sub-pipelines:

• inside another pipeline (inside function decorated by @pipeline1;
• inside task (solid) of another pipeline.

This is my result:

pipeline1.py

from dagster import solid, pipeline, execute_pipeline


@solid
def task1(context):
    context.log.info(context.solid.name)


@pipeline
def pipe1():
    task1()

pipeline2.py

from dagster import solid, pipeline, execute_pipeline


@solid
def task2(context):
    context.log.info(context.solid.name)


@pipeline
def pipe2():
    task2()

---

1st approach:
master_pipe1.py

from dagster import pipeline, execute_pipeline, repository, solid

from pipeline1 import pipe1
from pipeline2 import pipe2


@pipeline
def pipe():
    res1 = execute_pipeline(pipe1)
    res2 = execute_pipeline(pipe2)


@repository
def repo():
    return [pipe, pipe1, pipe2]

run dagit

dagit -f master_pipe1.py

Result: Dagit can not visualize such pipeline:

---

2nd approach:

master_pipe2.py

from dagster import pipeline, execute_pipeline, repository, solid

from pipeline1 import pipe1
from pipeline2 import pipe2


@solid
def run_pipes(context):
    
    res1 = execute_pipeline(pipe1)
    res2 = execute_pipeline(pipe2)


@pipeline
def pipe():
    run_pipes()


@repository
def repo():
    return [pipe, pipe1, pipe2]

Result: Dagit make visualization, but only solid run_pipes is displayed in pipeline pipe definition:

---

In both cases:

• I can execute pipeline pipe which in its turn executes other pipelines successfully;
• but just event logs are wrote (if I execute single pipeline, I get info logs wrote by context.log.info).

---

Conclusion:

• can I organize sub-pipelines: one pipeline executes (or waits) another?
• is there Dagit Python API: In Dagit I can click Launch Execution and pipeline will be run - can I do this programmatically?

Thank you for advance.

[ankxyz]

I consider solution answer #3048. Using DagsterInstnace I get expected result - pipeline executed inside another pipeline are reported into instance and Dagit displays sub-pipeline logs.