Skip to content

Commit

Permalink
[dagster-airlift] Federation tutorial mirroring (#26020)
Browse files Browse the repository at this point in the history
## Summary & Motivation
This PR adds a script to mirror the federation tutorial to github, and
fleshes out the readme.
  • Loading branch information
dpeng817 committed Nov 21, 2024
1 parent e3e04d5 commit 2f90af2
Show file tree
Hide file tree
Showing 3 changed files with 87 additions and 1 deletion.
18 changes: 17 additions & 1 deletion examples/airlift-federation-tutorial/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
## Airlift Federation Tutorial Code

Work in progress.
This repo is the code that the Airlift federation tutorial is based off of. Follow along on the main docs site [here](https://docs.dagster.io/integrations/airlift).

This example demonstrates how to use the `dagster-airlift` package to unify multiple Airflow instances into Dagster as a single control plane, and then federate execution between those Airflow instances.

## Example Structure

The following explains the structure of the repo.

```plaintext
airlift_federation_tutorial
├── constants.py: Contains constant values used throughout both Airflow and Dagster
├── dagster_defs: Contains Dagster definitions
│ ├── definitions.py: Empty starter file for following along with the tutorial
│ └── stages: Contains reference implementations for each stage of the migration process.
├── metrics_airflow_dags: Contains the Airflow DAGs for the "downstream" airflow instance
└── warehouse_airflow_dags: Contains the Airflow DAGs for the "upstream" airflow instance
```
6 changes: 6 additions & 0 deletions examples/experimental/dagster-airlift/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ mirror_tutorial:
@chmod +x scripts/mirror_tutorial.sh
@./scripts/mirror_tutorial.sh

mirror_federation_tutorial:
@chmod +x scripts/mirror_federation_tutorial.sh
@./scripts/mirror_federation_tutorial.sh

# Runs the full release process for dagster-airlift.
# - Enforces that we're on master
# - Bumps the version in setup.py
Expand All @@ -26,6 +30,8 @@ adhoc_release:
@make adhoc_pypi
@echo "Mirroring tutorial..."
@make mirror_tutorial
@echo "Mirroring federation tutorial..."
@make mirror_federation_tutorial
@git add .
@git checkout -b airlift-$$(./scripts/extract_pypi_version.sh)
@git commit -m "[dagster-airlift] $$(./scripts/extract_pypi_version.sh)"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/bin/bash
# There's a lot of shared work here with the `update_mirror` step of the OSS release process.
# Code here can be consolidated.

# Exit immediately if a command exits with a non-zero status
set -e

# Create a new temporary directory
temp_dir=$(mktemp -d)
echo "Created temporary directory: $temp_dir"

# Change to the temporary directory
cd "$temp_dir"

# Initialize git, add remote, fetch, and reset
echo "Initializing git repository..."
git init
echo "Adding remote..."
git remote add origin [email protected]:dagster-io/airlift-federation-tutorial.git
echo "Fetching and resetting..."
git fetch origin
git reset --soft origin/main

# Go back to the original directory
cd -

echo "Copying files to temporary directory..."
# Copy files from examples/tutorial_examples to the temporary directory
rsync -av \
--exclude='.git' \
--exclude='*.tox' \
--exclude='.airflow_home' \
--exclude='.dagster_home' \
--exclude='*.egg-info' \
--exclude='airlift_federation_tutorial_tests' \
--exclude='conftest.py' \
../../airlift-federation-tutorial/ "$temp_dir/"

# Make the version extraction script executable
chmod +x "scripts/extract_pypi_version.sh"
# Get the version number
version=$(./scripts/extract_pypi_version.sh)

# Change to the temporary directory again
cd "$temp_dir"

# Add all files to git
git add .


echo "Committing with version $version..."
# Commit message is the version number
git commit -m "$version" --allow-empty

echo "Pushing to origin..."
# Push to origin
git push origin HEAD --force

echo "Pushed successfully. Cleaning up..."
# Clean up: remove the temporary directory
cd ..
rm -rf "$temp_dir"

echo "Script completed successfully"

0 comments on commit 2f90af2

Please sign in to comment.