[dagster-airlift] simple migration example #23555
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Aug 9, 2024
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
3 times, most recently
from
a2e6fbb to
b1a1e1e
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
485bdd5 to
f969a17
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
b1a1e1e to
1822845
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
f969a17 to
8d46b49
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
1822845 to
6c7e1dd
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
8d46b49 to
d4c3c4e
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
2 times, most recently
from
6b0b3df to
7df6f3f
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
d4c3c4e to
d62b62d
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
7df6f3f to
198af5a
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
d62b62d to
87da942
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
198af5a to
6998ff9
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
8846dc2 to
d65f0f3
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
c68134c to
78b6db8
Compare
This was referenced Aug 12, 2024
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
d65f0f3 to
0abbf9f
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
78b6db8 to
2d3bba1
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
0abbf9f to
6c722d4
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
2d3bba1 to
c0b0d6b
Compare
schrockn
reviewed
Aug 13, 2024
Comment on lines
21
to
31
| t1 = PythonDefs( | ||
| name="simple__t1", | ||
| specs=[AssetSpec(key=A1)], | ||
| python_fn=t1_work, | ||
| ) | ||
|
|
||
| t2 = PythonDefs( | ||
| name="simple__t2", | ||
| specs=[ | ||
| AssetSpec(key=A2, deps=[AssetDep(asset=A1)]), | ||
| AssetSpec(key=A3, deps=[AssetDep(asset=A1)]), | ||
| ], | ||
| python_fn=t2_work, | ||
| ) | ||
|
|
||
| t3 = PythonDefs( | ||
| name="simple__t3", | ||
| specs=[ | ||
| AssetSpec(key=A4, deps=[AssetDep(asset=A2), AssetDep(asset=A3)]), | ||
| ], | ||
| python_fn=t3_work, | ||
| ) | ||
|
|
There was a problem hiding this comment.
Let's make this look like this:
# Asset graph
a1 = AssetSpec(key="a1")
a2 = AssetSpec(key="a2"), deps=[a1])
a3 = AssetSpec(key="a3"), deps=[a1])
a4 = AssetSpec(key="a4"), deps=[a2, a3])
# Factory instance per Airflow tasks
t1 = PythonDefs(name="simple__t1", specs=[a1])
t2 = PythonDefs(name="simple__t2", specs=[a2, a3])
t3 = PythonDefs(name="simple__t3", specs=[a4])
schrockn
previously requested changes
Aug 13, 2024
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
c0b0d6b to
8ceea9f
Compare
This was referenced Aug 13, 2024
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
8ceea9f to
82c5dd1
Compare
schrockn
approved these changes
Aug 13, 2024
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
3 times, most recently
from
ebd571c to
ca408aa
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
ca408aa to
28f23c8
Compare
PedramNavid
pushed a commit
that referenced
this pull request
Aug 14, 2024
Adds a simple example which shows what it looks like to have completed the peer, observe, and migrate steps, before having actually decommissioned the airflow instance. Also includes some necessary changes for the demo.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a simple example which shows what it looks like to have completed the peer, observe, and migrate steps, before having actually decommissioned the airflow instance.
Also includes some necessary changes for the demo.