[dagster-airlift] airflow op swizzler #23402
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dpeng817
changed the title
This was referenced Aug 3, 2024
This was referenced Aug 3, 2024
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
4325983 to
e9a752e
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
196c139 to
3adcf2d
Compare
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
e9a752e to
ebc04fe
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
3adcf2d to
3c646de
Compare
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
ebc04fe to
2e766d4
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
3c646de to
709b9fc
Compare
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
2e766d4 to
e12b6b5
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
709b9fc to
a115058
Compare
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
e12b6b5 to
40cea78
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
a115058 to
a6e6746
Compare
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
40cea78 to
f89e978
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
a6e6746 to
8f65ed1
Compare
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
f89e978 to
24f7927
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
8f65ed1 to
b7bc220
Compare
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
24f7927 to
d6a604f
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
b7bc220 to
f05d05a
Compare
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
d6a604f to
28c5820
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
f05d05a to
d955dd5
Compare
dpeng817
force-pushed
the
dpeng817/dagster_operator
branch
from
28c5820 to
d713198
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
1fec362 to
d2cf178
Compare
|
@schrockn made one more big change to this PR that introspects the baseoperator args, and uses that construct the DagsterOperator. The goal here is to be more resistant to changes in BaseOperator args. Lmk what you think. |
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
d2cf178 to
ae75b08
Compare
schrockn
previously requested changes
Aug 12, 2024
examples/experimental/dagster-airlift/dagster_airlift/in_airflow/dagster_operator.py
Outdated
Show resolved
Hide resolved
examples/experimental/dagster-airlift/dagster_airlift/in_airflow/mark_as_migrating.py
Outdated
Show resolved
Hide resolved
examples/experimental/dagster-airlift/dagster_airlift/migration_state.py
Outdated
Show resolved
Hide resolved
This was referenced Aug 12, 2024
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
ae75b08 to
8846dc2
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
8846dc2 to
d65f0f3
Compare
This was referenced Aug 12, 2024
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
d65f0f3 to
0abbf9f
Compare
dpeng817
force-pushed
the
dpeng817/airflow_op_switcher
branch
from
0abbf9f to
6c722d4
Compare
This was referenced Aug 13, 2024
PedramNavid
pushed a commit
that referenced
this pull request
Aug 14, 2024
Adds functionality for `mark_as_dagster_migrating` to swizzle out an operator for the dagster operator. Adds a test that runs against live airflow and dagster, and ensures that when a run of the dag marked as "migrated" is kicked off, that it results in an asset materialization appearing in dagster.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds functionality for
mark_as_dagster_migratingto swizzle out an operator for the dagster operator.Adds a test that runs against live airflow and dagster, and ensures that when a run of the dag marked as "migrated" is kicked off, that it results in an asset materialization appearing in dagster.