[dagster-airlift] mwaa example #23574
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
This was referenced Aug 12, 2024
This was referenced Aug 12, 2024
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
270a9b8 to
2bf3cd2
Compare
This was referenced Aug 12, 2024
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
2b89a72 to
c68134c
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
2bf3cd2 to
5d41734
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
c68134c to
78b6db8
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
5d41734 to
4beb211
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
78b6db8 to
2d3bba1
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
4beb211 to
34a1db3
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
2d3bba1 to
c0b0d6b
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
34a1db3 to
32dc1ca
Compare
schrockn
reviewed
Aug 13, 2024
| @@ -31,16 +32,17 @@ def get_session_info(region: str, env_name: str) -> Tuple[str, str]: | |||
|
|
|||
|
|
|||
| class MwaaSessionAuthBackend(AirflowAuthBackend): | |||
| def __init__(self, region: str, env_name: str): | |||
| def __init__(self, region: str, env_name: str, profile_name: Optional[str] = None): | |||
There was a problem hiding this comment.
Hmmm maybe we should just take in the boto.Session object directly as we might just replicate the entire interface here.
There was a problem hiding this comment.
I think we should just revisit the interface in general. I think we should be aiming to only auth the mwaa session once in general, not just retrieving the client I think.
There was a problem hiding this comment.
I bias towards getting this in so you guys can build on top of mwaa if needed, and then iterating on the API once it's under test
schrockn
reviewed
Aug 13, 2024
Comment on lines
+1
to
+20
| """This script uploads DAG files and requirements.txt to an S3 bucket in a directory | ||
| matching the name of the specified MWAA environment. It then attempts to update | ||
| the MWAA environment with the new S3 location. If the environment doesn't exist, | ||
| it provides instructions for creating one. | ||
|
|
||
| The script prompts for user confirmation before deleting any existing contents | ||
| in the target S3 directory, then uploads the new files, ensuring a clean deployment. | ||
| """ | ||
|
|
||
| import os | ||
| from typing import List | ||
|
|
||
| import boto3 | ||
| import click | ||
| from botocore.exceptions import ClientError | ||
|
|
||
|
|
||
| def get_s3_objects(s3_client: boto3.client, bucket: str, prefix: str) -> List[dict]: | ||
| """Get list of objects in the S3 bucket with the given prefix.""" | ||
| try: |
There was a problem hiding this comment.
This is a lot of code. Does MWAA not provide some of this stuff as utilities? This looks like we are reimplementing AWS tooling with ChatGPT or something.
There was a problem hiding this comment.
MWAA has pretty bad tooling for this stuff unfortunately. I pulled a lot of this from examples colton has shown me of previous mwaa deploy scripts (+ chat GPT to throw it in python land). The utilities here really are this bad unfortunately though; if you look at the AWS documentation it's exactly the steps we take here more or less: https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-folder.html#configuring-dag-folder-uploading
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
c0b0d6b to
8ceea9f
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
32dc1ca to
08cfa7f
Compare
This was referenced Aug 13, 2024
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
8ceea9f to
82c5dd1
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
08cfa7f to
3044ea5
Compare
schrockn
approved these changes
Aug 13, 2024
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
82c5dd1 to
11ecaf3
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
3044ea5 to
f7ad1a4
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
11ecaf3 to
ebd571c
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
f7ad1a4 to
96183a0
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
ebd571c to
ca408aa
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
96183a0 to
c6858a9
Compare
dpeng817
force-pushed
the
dpeng817/simple_migration_example
branch
from
ca408aa to
28f23c8
Compare
dpeng817
force-pushed
the
dpeng817/mwaa-example
branch
from
c6858a9 to
551bdb1
Compare
PedramNavid
pushed a commit
that referenced
this pull request
Aug 14, 2024
Adds example code for having dags running in mwaa and observing them running locally in dagster. The goal here is twofold: - Provide sample code for using the mwaa utility functions we build - Provide an easily stood up and reproducible test bed for deploying code to mwaa and testing underlying dagster functionality.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds example code for having dags running in mwaa and observing them running locally in dagster.
The goal here is twofold: