-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[dagster-airlift] Federation tutorial mirroring (#26020)
## Summary & Motivation This PR adds a script to mirror the federation tutorial to github, and fleshes out the readme.
- Loading branch information
Showing
3 changed files
with
87 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,19 @@ | ||
## Airlift Federation Tutorial Code | ||
|
||
Work in progress. | ||
This repo is the code that the Airlift federation tutorial is based off of. Follow along on the main docs site [here](https://docs.dagster.io/integrations/airlift). | ||
|
||
This example demonstrates how to use the `dagster-airlift` package to unify multiple Airflow instances into Dagster as a single control plane, and then federate execution between those Airflow instances. | ||
|
||
## Example Structure | ||
|
||
The following explains the structure of the repo. | ||
|
||
```plaintext | ||
airlift_federation_tutorial | ||
├── constants.py: Contains constant values used throughout both Airflow and Dagster | ||
├── dagster_defs: Contains Dagster definitions | ||
│ ├── definitions.py: Empty starter file for following along with the tutorial | ||
│ └── stages: Contains reference implementations for each stage of the migration process. | ||
├── metrics_airflow_dags: Contains the Airflow DAGs for the "downstream" airflow instance | ||
└── warehouse_airflow_dags: Contains the Airflow DAGs for the "upstream" airflow instance | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
64 changes: 64 additions & 0 deletions
64
examples/experimental/dagster-airlift/scripts/mirror_federation_tutorial.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
#!/bin/bash | ||
# There's a lot of shared work here with the `update_mirror` step of the OSS release process. | ||
# Code here can be consolidated. | ||
|
||
# Exit immediately if a command exits with a non-zero status | ||
set -e | ||
|
||
# Create a new temporary directory | ||
temp_dir=$(mktemp -d) | ||
echo "Created temporary directory: $temp_dir" | ||
|
||
# Change to the temporary directory | ||
cd "$temp_dir" | ||
|
||
# Initialize git, add remote, fetch, and reset | ||
echo "Initializing git repository..." | ||
git init | ||
echo "Adding remote..." | ||
git remote add origin [email protected]:dagster-io/airlift-federation-tutorial.git | ||
echo "Fetching and resetting..." | ||
git fetch origin | ||
git reset --soft origin/main | ||
|
||
# Go back to the original directory | ||
cd - | ||
|
||
echo "Copying files to temporary directory..." | ||
# Copy files from examples/tutorial_examples to the temporary directory | ||
rsync -av \ | ||
--exclude='.git' \ | ||
--exclude='*.tox' \ | ||
--exclude='.airflow_home' \ | ||
--exclude='.dagster_home' \ | ||
--exclude='*.egg-info' \ | ||
--exclude='airlift_federation_tutorial_tests' \ | ||
--exclude='conftest.py' \ | ||
../../airlift-federation-tutorial/ "$temp_dir/" | ||
|
||
# Make the version extraction script executable | ||
chmod +x "scripts/extract_pypi_version.sh" | ||
# Get the version number | ||
version=$(./scripts/extract_pypi_version.sh) | ||
|
||
# Change to the temporary directory again | ||
cd "$temp_dir" | ||
|
||
# Add all files to git | ||
git add . | ||
|
||
|
||
echo "Committing with version $version..." | ||
# Commit message is the version number | ||
git commit -m "$version" --allow-empty | ||
|
||
echo "Pushing to origin..." | ||
# Push to origin | ||
git push origin HEAD --force | ||
|
||
echo "Pushed successfully. Cleaning up..." | ||
# Clean up: remove the temporary directory | ||
cd .. | ||
rm -rf "$temp_dir" | ||
|
||
echo "Script completed successfully" |