Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] - Update Dagster+ CI/CD files into single deploy.yml #23035

Closed
wants to merge 5 commits into from
Closed

[docs] - Update Dagster+ CI/CD files into single deploy.yml #23035

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b35fe5a
Update CI/CD reference
erinkcochran87 Jul 16, 2024
83d2ac5
Update branch deps guide
erinkcochran87 Jul 16, 2024
6fce05a
Update dbt and Dagster+ guide
erinkcochran87 Jul 16, 2024
f5c0004
Update DU-dbt lesson 7
erinkcochran87 Jul 16, 2024
0e97f90
Fix typos
erinkcochran87 Jul 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 53 additions & 36 deletions docs/content/dagster-plus/references/ci-cd-file-reference.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ When you import a project into Dagster+ from GitHub or Gitlab, a few `.yml` file

---

## branch_deployments.yml
## deploy.yml

<table
className="table"
Expand All @@ -28,7 +28,7 @@ When you import a project into Dagster+ from GitHub or Gitlab, a few `.yml` file
>
<strong>Name</strong>
</td>
<td>branch_deployments.yml</td>
<td>deploy.yml</td>
</tr>
<tr>
<td
Expand All @@ -48,12 +48,7 @@ When you import a project into Dagster+ from GitHub or Gitlab, a few `.yml` file
>
<strong>Required</strong>
</td>
<td>
Required to use{" "}
<a href="/dagster-plus/managing-deployments/branch-deployments">
Branch Deployments
</a>
</td>
<td>Required for Dagster+</td>
</tr>
<tr>
<td
Expand All @@ -64,19 +59,46 @@ When you import a project into Dagster+ from GitHub or Gitlab, a few `.yml` file
<strong>Description</strong>
</td>
<td>
Defines the steps required to use Branch Deployments. <br />
<br />
<strong>Note</strong>: This file must be manually added to the
repository if using a{" "}
<a href="/dagster-plus/deployment/hybrid">Hybrid deployment</a>.
This file defines the steps required to:
<ul>
<li>
Deploy a project in Dagster+, including running checks, checking out
the project directory, and deploying the project.
</li>
<li>
Use{" "}
<a href="/dagster-plus/managing-deployments/branch-deployments">
Branch Deployments
</a>{" "}
(For Dagster+ accounts created after [TODO])
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace with date when we have it

</li>
</ul>
Additionally, note the following:
<ul>
<li>
<strong>
If using a{" "}
<a href="/dagster-plus/deployment/hybrid">Hybrid deployment</a>
</strong>
, this file must be manually added to the repository.
</li>
<li>
<strong>If using dbt</strong>, some steps may need to be added to
successfully deploy your project. Refer to the{" "}
<a href="/integrations/dbt/using-dbt-with-dagster-plus#step-3-update-the-cicd-files">
Using dbt with Dagster+ guide
</a>{" "}
for more information.
</li>
</ul>
</td>
</tr>
</tbody>
</table>

---

## deploy.yml
## branch_deployments.yml

<table
className="table"
Expand All @@ -93,7 +115,7 @@ When you import a project into Dagster+ from GitHub or Gitlab, a few `.yml` file
>
<strong>Name</strong>
</td>
<td>deploy.yml</td>
<td>branch_deployments.yml</td>
</tr>
<tr>
<td
Expand All @@ -113,7 +135,12 @@ When you import a project into Dagster+ from GitHub or Gitlab, a few `.yml` file
>
<strong>Required</strong>
</td>
<td>Required for Dagster+</td>
<td>
Required to use{" "}
<a href="/dagster-plus/managing-deployments/branch-deployments">
Branch Deployments
</a>
</td>
</tr>
<tr>
<td
Expand All @@ -124,26 +151,16 @@ When you import a project into Dagster+ from GitHub or Gitlab, a few `.yml` file
<strong>Description</strong>
</td>
<td>
Defines the steps required to deploy a project in Dagster+, including
running checks, checking out the project directory, and deploying the
project. Additionally, note the following:
<ul>
<li>
<strong>
If using a{" "}
<a href="/dagster-plus/deployment/hybrid">Hybrid deployment</a>
</strong>
, this file must be manually added to the repository.
</li>
<li>
<strong>If using dbt</strong>, some steps may need to be added to
successfully deploy your project. Refer to the{" "}
<a href="/integrations/dbt/using-dbt-with-dagster-plus#step-3-update-the-cicd-files">
Using dbt with Dagster+ guide
</a>{" "}
for more information.
</li>
</ul>
Defines the steps required to use Branch Deployments.{" "}
<strong>For Dagster+ accounts created after [TODO]</strong>, the
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace with date when we have it

contents of this file will be consolidated into the{" "}
<code>deploy.yml</code> file.
<br />
<br />
<strong>Note</strong>: This file must be manually added to the repository
if using a <a href="/dagster-plus/deployment/hybrid">
Hybrid deployment
</a>.
</td>
</tr>
</tbody>
Expand Down
26 changes: 18 additions & 8 deletions docs/content/guides/dagster/branch_deployments.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,13 +272,19 @@ defs = Definitions(
## Step 4: Create our database clone upon opening a branch

<TabGroup>
<TabItem name="Using GitHub Actions">
<TabItem name="Using GitHub Actions">

<Note>
<strong>Heads up!</strong> If your Dagster+ account was created before [TODO],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace with date when we have it

refer to your repository's <code>branch_deployments.yml</code> instead of{" "}
<code>deploy.yml</code>.
</Note>

The `branch_deployments.yml` file located in `.github/workflows/branch_deployments.yml` defines a `dagster_cloud_build_push` job with a series of steps that launch a branch deployment. Because we want to queue a run of `clone_prod` within each deployment after it launches, we'll add an additional step at the end `dagster_cloud_build_push`. This job is triggered on multiple pull request events: `opened`, `synchronize`, `reopen`, and `closed`. This means that upon future pushes to the branch, we'll trigger a run of `clone_prod`. The `if` condition below ensures that `clone_prod` will not run if the pull request is closed:
The `deploy.yml` file located in `.github/workflows/deploy.yml` defines a `dagster_cloud_build_push` job with a series of steps that launch a branch deployment. Because we want to queue a run of `clone_prod` within each deployment after it launches, we'll add an additional step at the end `dagster_cloud_build_push`.

```yaml file=/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml
# .github/workflows/branch_deployments.yml
This job is triggered on multiple pull request events: `opened`, `synchronize`, `reopen`, and `closed`. This means that upon future pushes to the branch, we'll trigger a run of `clone_prod`. The `if` condition below ensures that `clone_prod` will not run if the pull request is closed:

```yaml file=/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml
name: Dagster Branch Deployments
on:
pull_request:
Expand Down Expand Up @@ -327,7 +333,7 @@ width={1431}
height={537}
/> </TabItem>

<TabItem name="Using Gitlab CI/CD">
<TabItem name="Using Gitlab CI/CD">

The `.gitlab-ci.yaml` script contains a `deploy` job that defines a series of steps that launch a branch deployment. Because we want to queue a run of `clone_prod` within each deployment after it launches, we'll add an additional step at the end of `deploy`. This job is triggered on when a merge request is created or updated. This means that upon future pushes to the branch, we'll trigger a run of `clone_prod`.

Expand Down Expand Up @@ -408,11 +414,15 @@ height={537}
<TabGroup>
<TabItem name="Using GitHub Actions">

Finally, we can add a step to our `branch_deployments.yml` file that queues a run of our `drop_prod_clone` job:
<Note>
<strong>Heads up!</strong> If your Dagster+ account was created before [TODO],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace with date when we have it

refer to your repository's <code>branch_deployments.yml</code> instead of{" "}
<code>deploy.yml</code>.
</Note>

```yaml file=/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml
# .github/workflows/branch_deployments.yml
Finally, we can add a step to our `deploy.yml` file that queues a run of our `drop_prod_clone` job:

```yaml file=/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml
name: Dagster Branch Deployments
on:
pull_request:
Expand Down
8 changes: 3 additions & 5 deletions docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Once Dagster+ finishes importing the project, move onto the next step.
The file structure of the repository will change the first time a project is deployed using Dagster+. For dbt projects, a few things will happen:

- **A [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml) will be created.** This file defines the project as a Dagster+ code location.
- **A few `.yml` files, used for CI/CD, will be created in `.github/workflows`.** [These files](/dagster-plus/references/ci-cd-file-reference), named `branch_deployments.yml` and `deploy.yml`, manage the deployments of the repository.
- **CI/CD will be automatically set up**, using the contents of `.github/workflows`.\*\* [The files in this directory](/dagster-plus/references/ci-cd-file-reference) manage the deployments of the repository.
- **For dbt-only projects being deployed for the first time**, Dagster+ will create a new Dagster project in the repository using the [`dagster-dbt scaffold`](/integrations/dbt/reference#scaffolding-a-dagster-project-from-a-dbt-project) command. This will result in a Dagster project that matches the dbt project. For example, a dbt project named `my_dbt_project` will contain a Dagster project in `my_dbt_project/my_dbt_project` after the process completes. Refer to the [Dagster project files reference](/guides/understanding-dagster-project-files) to learn more about the files in a Dagster project.

**Use the following tabs** to see how the repository will change for a [dbt-only project](#dbt-only-projects) and a [dbt and Dagster project](#dbt-and-dagster-projects) being deployed for the first time.
Expand Down Expand Up @@ -112,9 +112,8 @@ When the Dagster+ deployment process completes, the repository will now look lik
## after Dagster+ deployment

my_dbt_project
├── .github ## CI/CD files
├── .github ## CI/CD setup
│   ├── workflows
│   │   ├── branch_deployments.yml
│   │   ├── deploy.yml
├── models
│   ├── my_model.sql
Expand Down Expand Up @@ -156,7 +155,6 @@ After the Dagster+ changes, a dbt and Dagster project will include the files req
my_dbt_and_dagster_project
├── .github ## CI/CD files
│   ├── workflows
│   │   ├── branch_deployments.yml
│   │   ├── deploy.yml
├── dbt
│   ├── models
Expand Down Expand Up @@ -220,7 +218,7 @@ The last step is to update the [CI/CD files](/dagster-plus/references/ci-cd-file

5. Save the changes.

6. Open the `branch_deployments.yml` file and repeat steps 3 - 5.
6. **If your Dagster+ account was created before \[TODO]**, open the `branch_deployments.yml` file and repeat steps 3 - 5.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace with date when we have it


7. Commit the changes to the repository.

Expand Down
2 changes: 2 additions & 0 deletions docs/dagster-university/pages/dagster-dbt/lesson-7/3-setting-up-dagster-cloud.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ When Dagster deploys the code, a few things happen:
- `.github/workflows/deploy.yml` - This file sets up Continuous Deployment (CD) for the repository. We won’t talk through all the steps here, but a high-level summary is that every time a change is made to the `main` branch of your repository, this GitHub Action will build your Dagster project and deploy it to Dagster+.
- `.github/workflows/branch_deployments.yml` - This file enables the use of [Branch Deployments](https://docs.dagster.io/dagster-cloud/managing-deployments/branch-deployments), a Dagster+ feature that automatically creates staging environments for your Dagster code with every pull request. We won’t work with Branch Deployments during this lesson, but we highly recommend trying them out!

**Note**: This file is only created if your Dagster+ account was created before [TODO]. If your account was created on or after this date, the `branch_deployments.yml` file will be consolidated into `deploy.yml`.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace with date when we have it


---

## Checking deployment status
Expand Down
2 changes: 0 additions & 2 deletions ...docs_snippets/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# .github/workflows/branch_deployments.yml

name: Dagster Branch Deployments
on:
pull_request:
Expand Down
2 changes: 0 additions & 2 deletions ...s_snippets/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# .github/workflows/branch_deployments.yml

name: Dagster Branch Deployments
on:
pull_request:
Expand Down
Loading