From b35fe5ab5344e96033bb3bbd56f1f0845f51d40d Mon Sep 17 00:00:00 2001 From: Erin Cochran Date: Tue, 16 Jul 2024 14:57:13 -0400 Subject: [PATCH 1/5] Update CI/CD reference --- .../references/ci-cd-file-reference.mdx | 89 +++++++++++-------- 1 file changed, 53 insertions(+), 36 deletions(-) diff --git a/docs/content/dagster-plus/references/ci-cd-file-reference.mdx b/docs/content/dagster-plus/references/ci-cd-file-reference.mdx index 5a87beb091e6c..50ac50e1b77ae 100644 --- a/docs/content/dagster-plus/references/ci-cd-file-reference.mdx +++ b/docs/content/dagster-plus/references/ci-cd-file-reference.mdx @@ -11,7 +11,7 @@ When you import a project into Dagster+ from GitHub or Gitlab, a few `.yml` file --- -## branch_deployments.yml +## deploy.yml Name - + - + @@ -76,7 +98,7 @@ When you import a project into Dagster+ from GitHub or Gitlab, a few `.yml` file --- -## deploy.yml +## branch_deployments.yml
branch_deployments.ymldeploy.yml
Required - Required to use{" "} - - Branch Deployments - - Required for Dagster+
Description - Defines the steps required to use Branch Deployments.
-
- Note: This file must be manually added to the - repository if using a{" "} - Hybrid deployment. + This file defines the steps required to: +
    +
  • + Deploy a project in Dagster+, including running checks, checking out + the project directory, and deploying the project. +
    • +
  • + Use{" "} + + Branch Deployments + {" "} + (For Dagster+ accounts created after [TODO]) +
    • +
+ Additionally, note the following: +
    +
  • + + If using a{" "} + Hybrid deployment + + , this file must be manually added to the repository. +
    • +
  • + If using dbt, some steps may need to be added to + successfully deploy your project. Refer to the{" "} + + Using dbt with Dagster+ guide + {" "} + for more information. +
    • +
Name - + - + From 83d2ac58fad35367f29e3bb094c0f79858dcf797 Mon Sep 17 00:00:00 2001 From: Erin Cochran Date: Tue, 16 Jul 2024 14:57:23 -0400 Subject: [PATCH 2/5] Update branch deps guide --- .../guides/dagster/branch_deployments.mdx | 26 +++++++++++++------ .../branch_deployments/clone_prod.yaml | 2 -- .../branch_deployments/drop_db_clone.yaml | 2 -- 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/docs/content/guides/dagster/branch_deployments.mdx b/docs/content/guides/dagster/branch_deployments.mdx index 4a38253bbe8f8..53303f07a6d47 100644 --- a/docs/content/guides/dagster/branch_deployments.mdx +++ b/docs/content/guides/dagster/branch_deployments.mdx @@ -272,13 +272,19 @@ defs = Definitions( ## Step 4: Create our database clone upon opening a branch - + + + + Heads up! If your Dagster+ account was created before [TODO], + refer to your repository's branch_deployments.yml instead of{" "} + deploy.yml. + -The `branch_deployments.yml` file located in `.github/workflows/branch_deployments.yml` defines a `dagster_cloud_build_push` job with a series of steps that launch a branch deployment. Because we want to queue a run of `clone_prod` within each deployment after it launches, we'll add an additional step at the end `dagster_cloud_build_push`. This job is triggered on multiple pull request events: `opened`, `synchronize`, `reopen`, and `closed`. This means that upon future pushes to the branch, we'll trigger a run of `clone_prod`. The `if` condition below ensures that `clone_prod` will not run if the pull request is closed: +The `deploy.yml` file located in `.github/workflows/deploy.yml` defines a `dagster_cloud_build_push` job with a series of steps that launch a branch deployment. Because we want to queue a run of `clone_prod` within each deployment after it launches, we'll add an additional step at the end `dagster_cloud_build_push`. -```yaml file=/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml -# .github/workflows/branch_deployments.yml +This job is triggered on multiple pull request events: `opened`, `synchronize`, `reopen`, and `closed`. This means that upon future pushes to the branch, we'll trigger a run of `clone_prod`. The `if` condition below ensures that `clone_prod` will not run if the pull request is closed: +```yaml file=/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml name: Dagster Branch Deployments on: pull_request: @@ -327,7 +333,7 @@ width={1431} height={537} /> - + The `.gitlab-ci.yaml` script contains a `deploy` job that defines a series of steps that launch a branch deployment. Because we want to queue a run of `clone_prod` within each deployment after it launches, we'll add an additional step at the end of `deploy`. This job is triggered on when a merge request is created or updated. This means that upon future pushes to the branch, we'll trigger a run of `clone_prod`. @@ -408,11 +414,15 @@ height={537} -Finally, we can add a step to our `branch_deployments.yml` file that queues a run of our `drop_prod_clone` job: + + Heads up! If your Dagster+ account was created before [TODO], + refer to your repository's branch_deployments.yml instead of{" "} + deploy.yml. + -```yaml file=/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml -# .github/workflows/branch_deployments.yml +Finally, we can add a step to our `deploy.yml` file that queues a run of our `drop_prod_clone` job: +```yaml file=/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml name: Dagster Branch Deployments on: pull_request: diff --git a/examples/docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml b/examples/docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml index d4cedc3c26d82..0c65c4d2c790d 100644 --- a/examples/docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml +++ b/examples/docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml @@ -1,5 +1,3 @@ -# .github/workflows/branch_deployments.yml - name: Dagster Branch Deployments on: pull_request: diff --git a/examples/docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml b/examples/docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml index 7cc7c7740455f..586356d51baa6 100644 --- a/examples/docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml +++ b/examples/docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml @@ -1,5 +1,3 @@ -# .github/workflows/branch_deployments.yml - name: Dagster Branch Deployments on: pull_request: From 6fce05a9db787a24a6249bd9b30b2933af070afc Mon Sep 17 00:00:00 2001 From: Erin Cochran Date: Tue, 16 Jul 2024 15:02:51 -0400 Subject: [PATCH 3/5] Update dbt and Dagster+ guide --- .../integrations/dbt/using-dbt-with-dagster-plus.mdx | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx b/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx index ad1bc1c8c9336..e7b6218804e8f 100644 --- a/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx +++ b/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx @@ -72,7 +72,7 @@ Once Dagster+ finishes importing the project, move onto the next step. The file structure of the repository will change the first time a project is deployed using Dagster+. For dbt projects, a few things will happen: - **A [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml) will be created.** This file defines the project as a Dagster+ code location. -- **A few `.yml` files, used for CI/CD, will be created in `.github/workflows`.** [These files](/dagster-plus/references/ci-cd-file-reference), named `branch_deployments.yml` and `deploy.yml`, manage the deployments of the repository. +- **CI/CD will be automatically set up**, using the contents of `.github/workflows`.\*\* [The files in this directory](/dagster-plus/references/ci-cd-file-reference) manage the deployments of the repository. - **For dbt-only projects being deployed for the first time**, Dagster+ will create a new Dagster project in the repository using the [`dagster-dbt scaffold`](/integrations/dbt/reference#scaffolding-a-dagster-project-from-a-dbt-project) command. This will result in a Dagster project that matches the dbt project. For example, a dbt project named `my_dbt_project` will contain a Dagster project in `my_dbt_project/my_dbt_project` after the process completes. Refer to the [Dagster project files reference](/guides/understanding-dagster-project-files) to learn more about the files in a Dagster project. **Use the following tabs** to see how the repository will change for a [dbt-only project](#dbt-only-projects) and a [dbt and Dagster project](#dbt-and-dagster-projects) being deployed for the first time. @@ -112,9 +112,8 @@ When the Dagster+ deployment process completes, the repository will now look lik ## after Dagster+ deployment my_dbt_project -├── .github ## CI/CD files +├── .github ## CI/CD setup │   ├── workflows -│   │   ├── branch_deployments.yml │   │   ├── deploy.yml ├── models │   ├── my_model.sql @@ -156,7 +155,6 @@ After the Dagster+ changes, a dbt and Dagster project will include the files req my_dbt_and_dagster_project ├── .github ## CI/CD files │   ├── workflows -│   │   ├── branch_deployments.yml │   │   ├── deploy.yml ├── dbt │   ├── models @@ -220,7 +218,7 @@ The last step is to update the [CI/CD files](/dagster-plus/references/ci-cd-file 5. Save the changes. -6. Open the `branch_deployments.yml` file and repeat steps 3 - 5. +6. **If your Dagster+ account was created before \[TODO]**, open the `branch_deployments.yml` file and repeat steps 3 - 5. 7. Commit the changes to the repository. From f5c000478d68243a09ddaf57779b67a685bb6b19 Mon Sep 17 00:00:00 2001 From: Erin Cochran Date: Tue, 16 Jul 2024 15:08:04 -0400 Subject: [PATCH 4/5] Update DU-dbt lesson 7 --- .../pages/dagster-dbt/lesson-7/3-setting-up-dagster-cloud.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/dagster-university/pages/dagster-dbt/lesson-7/3-setting-up-dagster-cloud.md b/docs/dagster-university/pages/dagster-dbt/lesson-7/3-setting-up-dagster-cloud.md index c94e3359a09ec..d7043257acb8b 100644 --- a/docs/dagster-university/pages/dagster-dbt/lesson-7/3-setting-up-dagster-cloud.md +++ b/docs/dagster-university/pages/dagster-dbt/lesson-7/3-setting-up-dagster-cloud.md @@ -32,6 +32,8 @@ When Dagster deploys the code, a few things happen: - `.github/workflows/deploy.yml` - This file sets up Continuous Deployment (CD) for the repository. We won’t talk through all the steps here, but a high-level summary is that every time a change is made to the `main` branch of your repository, this GitHub Action will build your Dagster project and deploy it to Dagster+. - `.github/workflows/branch_deployments.yml` - This file enables the use of [Branch Deployments](https://docs.dagster.io/dagster-cloud/managing-deployments/branch-deployments), a Dagster+ feature that automatically creates staging environments for your Dagster code with every pull request. We won’t work with Branch Deployments during this lesson, but we highly recommend trying them out! + **Note**: This file is only created if your Dagster+ account was created before [TODO]. If your account was created on or after this date, the `branch_deployments.yml` file will be consolidated into `deploy.yml`. + --- ## Checking deployment status From 0e97f909c105f4da636eeade25c28ecbda871d85 Mon Sep 17 00:00:00 2001 From: Erin Cochran Date: Tue, 16 Jul 2024 15:12:35 -0400 Subject: [PATCH 5/5] Fix typos --- docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx b/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx index e7b6218804e8f..2b4b732ac01bf 100644 --- a/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx +++ b/docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx @@ -72,7 +72,7 @@ Once Dagster+ finishes importing the project, move onto the next step. The file structure of the repository will change the first time a project is deployed using Dagster+. For dbt projects, a few things will happen: - **A [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml) will be created.** This file defines the project as a Dagster+ code location. -- **CI/CD will be automatically set up**, using the contents of `.github/workflows`.\*\* [The files in this directory](/dagster-plus/references/ci-cd-file-reference) manage the deployments of the repository. +- **CI/CD will be automatically set up**, using the contents of `.github/workflows`. [The files in this directory](/dagster-plus/references/ci-cd-file-reference) manage the deployments of the repository. - **For dbt-only projects being deployed for the first time**, Dagster+ will create a new Dagster project in the repository using the [`dagster-dbt scaffold`](/integrations/dbt/reference#scaffolding-a-dagster-project-from-a-dbt-project) command. This will result in a Dagster project that matches the dbt project. For example, a dbt project named `my_dbt_project` will contain a Dagster project in `my_dbt_project/my_dbt_project` after the process completes. Refer to the [Dagster project files reference](/guides/understanding-dagster-project-files) to learn more about the files in a Dagster project. **Use the following tabs** to see how the repository will change for a [dbt-only project](#dbt-only-projects) and a [dbt and Dagster project](#dbt-and-dagster-projects) being deployed for the first time.
deploy.ymlbranch_deployments.yml
Required Required for Dagster+ + Required to use{" "} + + Branch Deployments + +
Description - Defines the steps required to deploy a project in Dagster+, including - running checks, checking out the project directory, and deploying the - project. Additionally, note the following: -
    -
  • - - If using a{" "} - Hybrid deployment - - , this file must be manually added to the repository. -
    • -
  • - If using dbt, some steps may need to be added to - successfully deploy your project. Refer to the{" "} - - Using dbt with Dagster+ guide - {" "} - for more information. -
    • -
+ Defines the steps required to use Branch Deployments.{" "} + For Dagster+ accounts created after [TODO], the + contents of this file will be consolidated into the{" "} + deploy.yml file. +
+
+ Note: This file must be manually added to the repository + if using a + Hybrid deployment + .