From d15dc29939959dbf12366c9e23c723501bed2c90 Mon Sep 17 00:00:00 2001 From: Colton Padden Date: Sat, 14 Dec 2024 19:15:46 -0500 Subject: [PATCH 1/5] [docs-beta] migrate - code locations docs --- .../code-locations/dagster-cloud-yaml.md | 371 +++++++++++++++++- 1 file changed, 369 insertions(+), 2 deletions(-) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md index 7790f2cf5230a..1305f4c417d46 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md @@ -1,7 +1,374 @@ --- title: dagster_cloud.yaml reference sidebar_position: 200 -unlisted: true --- -{/* TODO move content from https://docs.dagster.io/dagster-plus/managing-deployments/dagster-cloud-yaml */} \ No newline at end of file +:::note +This reference is applicable to Dagster+. +::: + + + + + + + + + + + + + + + + + + + + + + + + +
+ Name + dagster_cloud.yaml
+ Status + Active
+ Required + Required for Dagster+
+ Description + + {" "} + Similar to the workspace.yaml in open source to define code + locations for Dagster+. +
+ Uses + + Defines multiple code locations for Dagster+. For Hybrid deployments, this file can be used + to manage + environment variables/secrets. +
    +
    + +## File location + +The `dagster_cloud.yaml` file should be placed in the root of your Dagster project. Below is an example of a file structure modified from the [Dagster+ ETL quickstart](https://github.com/dagster-io/dagster/tree/master/examples/quickstart_etl). + +```shell +quickstart_etl +├── README.md +├── quickstart_etl +│   ├── __init__.py +│   ├── assets +│   ├── docker_image +├── ml_project +│ ├── quickstart_ml +│ ├── __init__.py +│ ├── ml_assets +├── random_assets.py +├── quickstart_etl_tests +├── dagster_cloud.yaml +├── pyproject.toml +├── setup.cfg +└── setup.py +``` + +If your repository contains multiple Dagster projects in subdirectories - otherwise known as a monorepository - add the `dagster_cloud.yaml` file to the root of where the Dagster projects are stored. + +## File structure + +Settings are formatted using YAML. For example, using the file structure above as an example: + +```yaml +# dagster_cloud.yaml + +locations: + - location_name: data-eng-pipeline + code_source: + package_name: quickstart_etl + build: + directory: ./quickstart_etl + registry: localhost:5000/docker_image + - location_name: ml-pipeline + code_source: + package_name: quickstart_ml + working_directory: ./ml_project + executable_path: venvs/path/to/ml_tensorflow/bin/python + - location_name: my_random_assets + code_source: + python_file: random_assets.py + container_context: + k8s: + env_vars: + - database_name + - database_username=hooli_testing + env_secrets: + - database_password +``` + +## Settings + +The `dagster_cloud.yaml` file contains a single top-level key, `locations`. This key accepts a list of code locations; for each code location, you can configure the following: + +- [Location name](#location-name) +- [Code source](#code-source) +- [Working directory](#working-directory) +- [Build](#build) +- [Python executable](#python-executable) +- [Container context](#container-context) + +### Location name + +**This key is required.** The `location_name` key specifies the name of the code location. The location name will always be paired with a [code source](#code-source). + +```yaml +# dagster_cloud.yaml + +locations: + - location_name: data-eng-pipeline + code_source: + package_name: quickstart_etl +``` + +{/* TODO + + + The name of your code location that will appear in the Dagster UI Code + locations page +
      +
    • + Format - string +
      • +
    +     +     +*/} + +### Code source + +**This section is required.** The `code_source` defines how a code location is sourced. + +A `code_source` key must contain either a `module_name`, `package_name`, or `file_name` parameter that specifies where to find the definitions in the code location. + + + + +```yaml +# dagster_cloud.yaml + +locations: + - location_name: data-eng-pipeline + code_source: + package_name: quickstart_etl +``` + + + + +```yaml +# dagster_cloud.yaml + +locations: + - location_name: data-eng-pipeline + code_source: + package_name: quickstart_etl + - location_name: machine_learning + code_source: + python_file: ml/ml_model.py +``` + + + + +{/* TODO + + + The name of a package containing Dagster code +
      +
    • + Format - string(folder name) +
      • +
    +     + + The name of a Python file containing Dagster code, for example: + analytics_pipeline.py +
      +
    • + Format - string(.py file name) +
      • +
    +     + + The name of a Python module containing Dagster code, for example: + analytics_etl +
      +
    • + Format - string(module name) +
      • +
    +     +     +*/} + +### Working directory + +Use the `working_directory` setting to load Dagster code from a different directory than the root of your code repository. This setting allows you to specify the directory you want to load your code from. + +Consider the following project: + +```shell +quickstart_etl +├── README.md +├── project_directory +│   ├── quickstart_etl +│   ├── __init__.py +│   ├── assets +│   ├── quickstart_etl_tests +├── dagster_cloud.yaml +├── pyproject.toml +├── setup.cfg +└── setup.py +``` + +To load from `/project_directory`, the `dagster_cloud.yaml` code location would look like this: + +```yaml +# dagster_cloud.yaml + +locations: + - location_name: data-eng-pipeline + code_source: + package_name: quickstart_etl + working_directory: ./project_directory +``` + +{/* TODO + + + The path of the directory that Dagster should load the code source from +
      +
    • + Format - string(path) +
      • +
    +     +     +*/} + +### Build + +The `build` section contains two parameters: + +- `directory` - Setting a build directory is useful if your `setup.py` or `requirements.txt` is in a subdirectory instead of the project root. This is common if you have multiple Python modules within a single Dagster project. +- `registry` - **Applicable only to Hybrid deployments.** Specifies the Docker registry to push the code location to. + +In the example below, the Docker image for the code location is in the root directory and the registry and image defined: + +```yaml +# dagster_cloud.yaml + +locations: + - location_name: data-eng-pipeline + code_source: + package_name: quickstart_etl + build: + directory: ./ + registry: your-docker-image-registry/image-name # e.g. localhost:5000/myimage +``` + +{/* TODO + + + The path to the directory in your project that you want to deploy. If there + are subdirectories, you can specify the path to only deploy a specific + project directory. +
      +
    • + Format - string(path) +
      • +
    • + Default - . +
      • +
    +     + + Applicable to Hybrid deployments. The Docker registry to + push your code location to +
      +
    • + Format - string(docker registry) +
      • +
    +     +     +*/} + +### Python executable + +For Dagster+ Hybrid deployments, the Python executable that is installed globally in the image, or the default Python executable on the local system if you use the local agent, will be used. To use a different Python executable, specify it using the `executable_path` setting. It can be useful to have different Python executables for different code locations. + +For Dagster+ Serverless deployments, you can specify a different Python version by [following these instructions](/dagster-plus/deployment/serverless#using-a-different-python-version). + +```yaml +# dagster_cloud.yaml + +locations: + - location_name: data-eng-pipeline + code_source: + package_name: quickstart_etl + executable_path: venvs/path/to/dataengineering_spark_team/bin/python + - location_name: machine_learning + code_source: + python_file: ml_model.py + executable_path: venvs/path/to/ml_tensorflow/bin/python +``` + +{/* TODO + + + The file path of the Python executable to use +
      +
    • + Format - string(path) +
      • +
    +     +     +*/} + +### Container context + +If using Hybrid deployment, you can define additional configuration options for code locations using the `container_context` parameter. Depending on the Hybrid agent you're using, the configuration settings under `container_context` will vary. + +Refer to the configuration reference for your agent for more info: + +- [Docker agent configuration reference](/dagster-plus/deployment/agents/docker/configuration-reference) +- [Amazon ECS agent configuration reference](/dagster-plus/deployment/agents/amazon-ecs/configuration-reference) +- [Kubernetes agent configuration reference](/dagster-plus/deployment/agents/kubernetes/configuration-reference) From 65fb84e475e7d15fc1b1ee6223ed197a14e39c0e Mon Sep 17 00:00:00 2001 From: Colton Padden Date: Thu, 19 Dec 2024 10:57:05 -0500 Subject: [PATCH 2/5] update reference tables --- .../code-locations/dagster-cloud-yaml.md | 113 ++++-------------- 1 file changed, 20 insertions(+), 93 deletions(-) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md index 1305f4c417d46..0becfbddae54b 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md @@ -156,19 +156,9 @@ locations: package_name: quickstart_etl ``` -{/* TODO - - - The name of your code location that will appear in the Dagster UI Code - locations page -
      -
    • - Format - string -
      • -
    -     -     -*/} +| Property | Description | Format | +|-----------------|----------------------------------------------------------------------------------------|----------| +| `location_name` | The name of your code location that will appear in the Dagster UI Code locations page. | `string` | ### Code source @@ -206,36 +196,11 @@ locations: -{/* TODO - - - The name of a package containing Dagster code -
      -
    • - Format - string(folder name) -
      • -
    -     - - The name of a Python file containing Dagster code, for example: - analytics_pipeline.py -
      -
    • - Format - string(.py file name) -
      • -
    -     - - The name of a Python module containing Dagster code, for example: - analytics_etl -
      -
    • - Format - string(module name) -
      • -
    -     -     -*/} +| Property | Description | Format | +|----------------------------|-----------------------------------------------------------------------------------|--------------------------| +| `code_source.package_name` | The name of a package containing Dagster code | `string` (folder name) | +| `code_source.python_file` | The name of a Python file containing Dagster code (e.g. `analytics_pipeline.py` ) | `string` (.py file name) | +| `code_source.module_name` | The name of a Python module containing Dagster code (e.g. `analytics_etl`) | `string` (module name) | ### Working directory @@ -269,18 +234,9 @@ locations: working_directory: ./project_directory ``` -{/* TODO - - - The path of the directory that Dagster should load the code source from -
      -
    • - Format - string(path) -
      • -
    -     -     -*/} +| Property | Description | Format | +|---------------------|-------------------------------------------------------------------------|-----------------| +| `working_directory` | The path of the directory that Dagster should load the code source from | `string` (path) | ### Build @@ -303,32 +259,12 @@ locations: registry: your-docker-image-registry/image-name # e.g. localhost:5000/myimage ``` -{/* TODO - - - The path to the directory in your project that you want to deploy. If there - are subdirectories, you can specify the path to only deploy a specific - project directory. -
      -
    • - Format - string(path) -
      • -
    • - Default - . -
      • -
    -     - - Applicable to Hybrid deployments. The Docker registry to - push your code location to -
      -
    • - Format - string(docker registry) -
      • -
    -     -     -*/} + +| Property | Description | Format | Default | +|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------|---------| +| `build.directory` | The path to the directory in your project that you want to deploy. If there are subdirectories, you can specify the path to only deploy a specific project directory. | `string` (path) | `.` | +| `build.registry` | **Applicable to Hybrid deployments.** The Docker registry to push your code location to | `string` (docker registry) | | + ### Python executable @@ -350,18 +286,9 @@ locations: executable_path: venvs/path/to/ml_tensorflow/bin/python ``` -{/* TODO - - - The file path of the Python executable to use -
      -
    • - Format - string(path) -
      • -
    -     -     -*/} +| Property | Description | Format | +|-------------------|-----------------------------------------------|-----------------| +| `executable_path` | The file path of the Python executable to use | `string` (path) | ### Container context From ff558ecedb47289088db615e083c9fa192b04cff Mon Sep 17 00:00:00 2001 From: Colton Padden Date: Thu, 19 Dec 2024 10:59:32 -0500 Subject: [PATCH 3/5] add python runtime --- .../deployment/code-locations/dagster-cloud-yaml.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md index 0becfbddae54b..04122fc5a5615 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md @@ -270,7 +270,7 @@ locations: For Dagster+ Hybrid deployments, the Python executable that is installed globally in the image, or the default Python executable on the local system if you use the local agent, will be used. To use a different Python executable, specify it using the `executable_path` setting. It can be useful to have different Python executables for different code locations. -For Dagster+ Serverless deployments, you can specify a different Python version by [following these instructions](/dagster-plus/deployment/serverless#using-a-different-python-version). +For Dagster+ Serverless deployments, you can specify a different Python version by [following these instructions](/dagster-plus/deployment/deployment-types/serverless/runtime-environment#python-version). ```yaml # dagster_cloud.yaml From df4921d29617cecad1d9ec022c26e09618b87ee6 Mon Sep 17 00:00:00 2001 From: Colton Padden Date: Thu, 19 Dec 2024 13:29:59 -0500 Subject: [PATCH 4/5] todo links --- .../deployment/code-locations/dagster-cloud-yaml.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md index 04122fc5a5615..4e1400ae432a0 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md @@ -296,6 +296,9 @@ If using Hybrid deployment, you can define additional configuration options for Refer to the configuration reference for your agent for more info: -- [Docker agent configuration reference](/dagster-plus/deployment/agents/docker/configuration-reference) -- [Amazon ECS agent configuration reference](/dagster-plus/deployment/agents/amazon-ecs/configuration-reference) -- [Kubernetes agent configuration reference](/dagster-plus/deployment/agents/kubernetes/configuration-reference) +{/* - [Docker agent configuration reference](/dagster-plus/deployment/agents/docker/configuration-reference) */} +- [Docker agent configuration reference](/todo) +{/* - [Amazon ECS agent configuration reference](/dagster-plus/deployment/agents/amazon-ecs/configuration-reference) */} +- [Amazon ECS agent configuration reference](/todo) +{/* - [Kubernetes agent configuration reference](/dagster-plus/deployment/agents/kubernetes/configuration-reference) */} +- [Kubernetes agent configuration reference](/todo) From e3252d2631cd60be816d810af9631cd232b74ee3 Mon Sep 17 00:00:00 2001 From: Colton Padden Date: Thu, 19 Dec 2024 14:28:27 -0500 Subject: [PATCH 5/5] todo links --- .../deployment/code-locations/dagster-cloud-yaml.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md index 4e1400ae432a0..09ebc00dbc154 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/code-locations/dagster-cloud-yaml.md @@ -270,7 +270,8 @@ locations: For Dagster+ Hybrid deployments, the Python executable that is installed globally in the image, or the default Python executable on the local system if you use the local agent, will be used. To use a different Python executable, specify it using the `executable_path` setting. It can be useful to have different Python executables for different code locations. -For Dagster+ Serverless deployments, you can specify a different Python version by [following these instructions](/dagster-plus/deployment/deployment-types/serverless/runtime-environment#python-version). +{/* For Dagster+ Serverless deployments, you can specify a different Python version by [following these instructions](/dagster-plus/deployment/deployment-types/serverless/runtime-environment#python-version). */} +For Dagster+ Serverless deployments, you can specify a different Python version by [following these instructions](/todo). ```yaml # dagster_cloud.yaml