[dagster-airlift][cleanup] Move airlift to python_modules (#26021)

## Summary & Motivation
Moves airlift to the python_modules/libraries directory. Matches other
top-level packages, means that it becomes part of the standard release
process.

This change required some legitimate changes to get the build to pass:
- reformatting proxied yaml files
- switching imports from relative to explicit
- fix the MANIFEST.in file which was totally busted before
- move py.typed to the `dagster_airlift` subdirectory.

This PR will be used as an accumulator for all the PRs prefixed with
[cleanup]

.buildkite/dagster-buildkite/dagster_buildkite/steps/packages.py

@@ -367,9 +367,6 @@ def k8s_extra_cmds(version: AvailablePythonVersion, _) -> List[str]:
     PackageSpec(
         "examples/experimental/dagster-blueprints",
     ),
-    PackageSpec(
-        "examples/experimental/dagster-airlift",
-    ),
     # Runs against live dbt cloud instance, we only want to run on commits and on the
     # nightly build
     PackageSpec(
@@ -393,18 +390,10 @@ def k8s_extra_cmds(version: AvailablePythonVersion, _) -> List[str]:
         timeout_in_minutes=30,
         queue=BuildkiteQueue.DOCKER,
     ),
-    PackageSpec(
-        "examples/experimental/dagster-airlift/perf-harness",
-        always_run_if=has_dagster_airlift_changes,
-    ),
     PackageSpec(
         "examples/airlift-migration-tutorial",
         always_run_if=has_dagster_airlift_changes,
     ),
-    PackageSpec(
-        "examples/experimental/dagster-airlift/kitchen-sink",
-        always_run_if=has_dagster_airlift_changes,
-    ),
     PackageSpec(
         "examples/experimental/dagster-dlift",
         name="dlift",
@@ -748,6 +737,14 @@ def tox_factors_for_folder(tests_folder_name: str) -> List[str]:
             AvailablePythonVersion.V3_12,
         ],
     ),
+    PackageSpec(
+        "python_modules/libraries/dagster-airlift/perf-harness",
+        always_run_if=has_dagster_airlift_changes,
+    ),
+    PackageSpec(
+        "python_modules/libraries/dagster-airlift/kitchen-sink",
+        always_run_if=has_dagster_airlift_changes,
+    ),
     PackageSpec(
         ".buildkite/dagster-buildkite",
         run_pytest=False,

docs/content/integrations/airlift/federation-tutorial/setup.mdx

@@ -8,19 +8,39 @@ In this step, we'll
 
 ## Installation & Project Structure
 
-First, clone the tutorial example repo locally, and enter the repo directory.
+First, we'll create a fresh virtual environment using `uv` and activate it.
 
 ```bash
-git clone [email protected]:dagster-io/airlift-migration-tutorial.git
-cd airlift-federation-tutorial
+pip install uv
+uv venv
+source .venv/bin/activate
 ```
 
-Next, we'll create a fresh virtual environment using `uv`.
+Next, we'll install Dagster, and verify that the dagster CLI is available.
 
 ```bash
-pip install uv
-uv venv
-source .venv/bin/activate
+uv pip install dagster
+dagster --version
+```
+
+Finally, we'll install the tutorial example code.
+
+```bash
+dagster project from-example --name airlift-federation-tutorial --example airlift-federation-tutorial
+```
+
+### Project Structure
+
+The following explains the structure of the repo.
+
+```plaintext
+airlift_federation_tutorial
+├── constants.py: Contains constant values used throughout both Airflow and Dagster
+├── dagster_defs: Contains Dagster definitions
+│   ├── definitions.py: Empty starter file for following along with the tutorial
+│   └── stages: Contains reference implementations for each stage of the migration process.
+├── metrics_airflow_dags: Contains the Airflow DAGs for the "downstream" airflow instance
+└── warehouse_airflow_dags: Contains the Airflow DAGs for the "upstream" airflow instance
 ```
 
 ## Running Airflow locally

docs/content/integrations/airlift/tutorial/setup.mdx

@@ -8,19 +8,42 @@ In this step, we'll
 
 ## Installation & Project Structure
 
-First, clone the tutorial example repo locally, and enter the repo directory.
+First, we'll create a fresh virtual environment using `uv`.
 
 ```bash
-git clone [email protected]:dagster-io/airlift-migration-tutorial.git
-cd airlift-migration-tutorial
+pip install uv
+uv venv
+source .venv/bin/activate
 ```
 
-Next, we'll create a fresh virtual environment using `uv`.
+Next, we'll install Dagster, and verify that the dagster CLI is available.
 
 ```bash
-pip install uv
-uv venv
-source .venv/bin/activate
+uv pip install dagster
+dagster --version
+```
+
+First, we'll create a fresh virtual environment using `uv`.
+
+```bash
+dagster project from-example --name airlift-migration-tutorial --example airlift-migration-tutorial
+```
+
+### Project Structure
+
+The following explains the structure of the repo.
+
+```plaintext
+tutorial_example
+├── shared: Contains shared Python & SQL code used Airflow and proxied Dagster code
+│
+├── dagster_defs: Contains Dagster definitions
+│   ├── stages: Contains reference implementations of each stage of the migration process
+│   ├── definitions.py: Empty starter file for following along with the tutorial
+│
+├── airflow_dags: Contains the Airflow DAG and associated files
+│   ├── proxied_state: Contains migration state files for each DAG, see migration step below
+│   ├── dags.py: The Airflow DAG definition
 ```
 
 ## Running Airflow locally

docs/sphinx/conf.py

@@ -14,7 +14,6 @@
 import sys
 
 ignored_folders = ["dagster-test"]
-additional_folders = ["../../examples/experimental/dagster-airlift"]
 base_path = "../../python_modules"
 python_modules_path = os.path.abspath(base_path)
 libraries_path = os.path.abspath(os.path.join(base_path, "libraries"))
@@ -35,8 +34,6 @@
         paths.append(folder_path)
 # Add the _ext folder
 paths.append(os.path.abspath("./_ext"))
-for path in additional_folders:
-    paths.append(os.path.abspath(path))
 
 for path in paths:
     sys.path.insert(0, path)

docs/tox.ini

@@ -48,7 +48,7 @@ deps =
   -e ../python_modules/libraries/dagster-sigma
   -e ../python_modules/libraries/dagster-tableau
   -e ../python_modules/libraries/dagster-powerbi
-  -e ../examples/experimental/dagster-airlift[tutorial,core,in-airflow,dbt]
+  -e ../python_modules/libraries/dagster-airlift[tutorial,core,in-airflow,dbt]
   -e ../examples/airlift-migration-tutorial
 
   sling

examples/airlift-federation-tutorial/README.md

@@ -1,6 +1,6 @@
 ## Airlift Federation Tutorial Code
 
-This repo is the code that the Airlift federation tutorial is based off of. Follow along on the main docs site [here](https://docs.dagster.io/integrations/airlift).
+This is the code that the Airlift federation tutorial is based off of.
 
 This example demonstrates how to use the `dagster-airlift` package to unify multiple Airflow instances into Dagster as a single control plane, and then federate execution between those Airflow instances.
 

examples/airlift-federation-tutorial/tox.ini

@@ -15,7 +15,7 @@ deps =
   -e ../../python_modules/dagster-pipes
   -e ../../python_modules/dagster-graphql
   -e ../../python_modules/libraries/dagster-dbt
-  -e ../experimental/dagster-airlift[core,dbt,test,in-airflow]
+  -e ../../python_modules/libraries/dagster-airlift[core,dbt,test,in-airflow]
   -e .
   pandas
 allowlist_externals =

examples/airlift-migration-tutorial/tox.ini

@@ -15,7 +15,7 @@ deps =
   -e ../../python_modules/dagster-pipes
   -e ../../python_modules/dagster-graphql
   -e ../../python_modules/libraries/dagster-dbt
-  -e ../experimental/dagster-airlift[core,dbt,test,in-airflow]
+  -e ../../python_modules/libraries/dagster-airlift[core,dbt,test,in-airflow]
   -e .
   pandas
 allowlist_externals =