Skip to content

Commit

Permalink
[docs] - Move Dagster+ reserved env vars to dedicated reference (#21484)
Browse files Browse the repository at this point in the history
## Summary & Motivation

This PR moves the built-in/reserved env vars to a dedicated reference
page. We've had some feedback that these were difficult to find, even in
search. This makes the content visible in the sidenav; additionally, the
now H1-level heading should improve search results.

<img width="1505" alt="Screenshot 2024-04-29 at 12 17 13 PM"
src="https://github.com/dagster-io/dagster/assets/16615212/c9b57619-626b-4c29-95a6-0603517208ae">

## How I Tested These Changes

eyes, local

---------

Co-authored-by: Colton Padden <[email protected]>
  • Loading branch information
erinkcochran87 and cmpadden authored Sep 19, 2024
1 parent 219ed91 commit ecf9367
Show file tree
Hide file tree
Showing 8 changed files with 105 additions and 63 deletions.
4 changes: 4 additions & 0 deletions docs/content/_navigation.json
Original file line number Diff line number Diff line change
Expand Up @@ -658,6 +658,10 @@
{
"title": "Setup for Hybrid agents",
"path": "/dagster-plus/managing-deployments/setting-environment-variables-agents"
},
{
"title": "Reserved environment variables",
"path": "/dagster-plus/managing-deployments/reserved-environment-variables"
}
]
},
Expand Down
4 changes: 4 additions & 0 deletions docs/content/dagster-plus.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,10 @@ Learn how to deploy your code to Dagster+, use command line tools, set up CI/CD,
title="Setting up environment variables using Hybrid agent configuration"
href="/dagster-plus/managing-deployments/setting-environment-variables-agents"
></ArticleListItem>
<ArticleListItem
title="Reserved environment variables"
href="/dagster-plus/managing-deployments/reserved-environment-variables"
></ArticleListItem>
</ArticleList>

### Branch deployments
Expand Down
4 changes: 4 additions & 0 deletions docs/content/dagster-plus/managing-deployments.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,10 @@ Learn how to deploy your code to Dagster+, use command line tools, set up CI/CD,
title="Setting up environment variables using Hybrid agent configuration"
href="/dagster-plus/managing-deployments/setting-environment-variables-agents"
></ArticleListItem>
<ArticleListItem
title="Reserved environment variables"
href="/dagster-plus/managing-deployments/reserved-environment-variables"
></ArticleListItem>
</ArticleList>

---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ height={311}

To ensure the best experience when using Branch Deployments, we recommend:

- **Configuring jobs based on environment**. Dagster automatically sets [environment variables](/dagster-plus/managing-deployments/environment-variables-and-secrets#built-in-environment-variables) containing deployment metadata, allowing you to parameterize jobs based on the executing environment. Use these variables in your jobs to configure things like connection credentials, databases, and so on. This practice will allow you to use Branch Deployments without impacting production data.
- **Configuring jobs based on environment**. Dagster automatically sets [environment variables](/dagster-plus/managing-deployments/reserved-environment-variables) containing deployment metadata, allowing you to parameterize jobs based on the executing environment. Use these variables in your jobs to configure things like connection credentials, databases, and so on. This practice will allow you to use Branch Deployments without impacting production data.

- **Creating jobs to automate output cleanup.** As Branch Deployments don't automatically remove [the output they create](#output-handling), you may want to create an additional Dagster job to perform the cleanup.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ In this example, the value of `SNOWFLAKE_PASSWORD` would be `production_password
variables.
</Note>

[Built-in (system) Dagster+ environment variables](#built-in-environment-variables) are reserved and therefore unavailable for use. [An error will surface in Dagster+](#troubleshooting) if a built-in variable name is used.
[Built-in (system) Dagster+ environment variables](/dagster-plus/managing-deployments/reserved-environment-variables) are reserved and therefore unavailable for use. [An error will surface in Dagster+](#troubleshooting) if a built-in variable name is used.

---

Expand Down Expand Up @@ -352,65 +352,9 @@ Ready to start using environment variables in your Dagster code? Refer to the [U

---

## Built-in environment variables

[Dagster+](/dagster-plus) provides a set of built-in, automatically populated environment variables, such as the name of a deployment or details about a branch deployment commit, that can be used to modify behavior based on environment.

### All deployment variables

The following variables are available in every deployment of your Dagster+ instance, including full (e.g., `prod`) and branch deployments.

<ReferenceTable>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_DEPLOYMENT_NAME">
The name of the Dagster+ deployment. For example, <code>prod</code>.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT">
If <code>1</code>, the deployment is a{" "}
<a href="/dagster-plus/managing-deployments/branch-deployments">
branch deployment
</a>
. Refer to the <a href="#branch-deployment-variables">
Branch Deployment variables section
</a> for a list of variables available in branch deployments.
</ReferenceTableItem>
</ReferenceTable>

### Branch Deployment variables

The following environment variables are currently available only in a [branch deployment](/dagster-plus/managing-deployments/branch-deployments).

For every commit made to a branch, the following metadata is available:

<ReferenceTable>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_SHA">
The SHA of the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_TIMESTAMP">
The time the commit occurred.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_AUTHOR_EMAIL">
The email of the git user who authored the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_AUTHOR_NAME">
The name of the git user who authored the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_MESSAGE">
The message associated with the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_BRANCH">
The name of the branch associated with the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_REPO">
The name of the repository associated with the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_PULL_REQUEST_ID">
The ID of the pull request associated with the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_PULL_REQUEST_STATUS">
The status of the pull request at the time of the commit. Possible values
are <code>OPEN</code>, <code>CLOSED</code>, and <code>MERGED</code>.
</ReferenceTableItem>
</ReferenceTable>
## Reserved environment variables

[Dagster+](/dagster-plus) provides a set of built-in, automatically populated environment variables, such as the name of a deployment or details about a branch deployment commit, that can be used to modify behavior based on environment. Refer to the [Reserved Dagster+ environment variables reference](/dagster-plus/managing-deployments/reserved-environment-variables) for more information.

---

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
title: "Reserved Dagster+ environment variables | Dagster Docs"
---

# Reserved Dagster+ environment variables

[Dagster+](/dagster-plus) provides a set of built-in, automatically populated environment variables, such as the name of a deployment or details about a branch deployment commit, that can be used to modify behavior based on environment.

---

## Reserved deployment variables

The following variables are available in every deployment of your Dagster+ instance, including full (e.g., `prod`) and branch deployments.

<ReferenceTable>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_DEPLOYMENT_NAME">
The name of the Dagster+ deployment. For example, <code>prod</code>.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT">
If <code>1</code>, the deployment is a{" "}
<a href="/dagster-plus/managing-deployments/branch-deployments">
branch deployment
</a>
. Refer to the <a href="#reserved-branch-deployment-variables">
Branch Deployment variables section
</a> for a list of variables available in branch deployments.
</ReferenceTableItem>
</ReferenceTable>

---

## Reserved Branch Deployment variables

The following environment variables are currently available only in a [branch deployment](/dagster-plus/managing-deployments/branch-deployments).

For every commit made to a branch, the following metadata is available:

<ReferenceTable>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_SHA">
The SHA of the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_TIMESTAMP">
The time the commit occurred.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_AUTHOR_EMAIL">
The email of the git user who authored the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_AUTHOR_NAME">
The name of the git user who authored the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_MESSAGE">
The message associated with the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_BRANCH">
The name of the branch associated with the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_GIT_REPO">
The name of the repository associated with the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_PULL_REQUEST_ID">
The ID of the pull request associated with the commit.
</ReferenceTableItem>
<ReferenceTableItem propertyName="DAGSTER_CLOUD_PULL_REQUEST_STATUS">
The status of the pull request at the time of the commit. Possible values
are <code>OPEN</code>, <code>CLOSED</code>, and <code>MERGED</code>.
</ReferenceTableItem>
</ReferenceTable>

---

## Related

<ArticleList>
<ArticleListItem
title="Dagster+ environment variables and secrets"
href="/dagster-plus/managing-deployments/environment-variables-and-secrets"
></ArticleListItem>
<ArticleListItem
title="Setting environment variables for Hybrid agents"
href="/dagster-plus/managing-deployments/setting-environment-variables-agents"
></ArticleListItem>
<ArticleListItem
title="Managing Dagster+ deployments"
href="/dagster-plus/managing-deployments"
></ArticleListItem>
</ArticleList>
2 changes: 1 addition & 1 deletion docs/content/guides/dagster/branch_deployments.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ At runtime, we’d like to determine which environment our code is running in: b

To ensure we can't accidentally write to production from within our branch deployment, we’ll use a different set of credentials from production and write to our database clone.

Dagster automatically sets certain [environment variables](/dagster-plus/managing-deployments/environment-variables-and-secrets#built-in-environment-variables) containing deployment metadata, allowing us to read these environment variables to discern between deployments. We can access the `DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT` environment variable to determine the currently executing environment.
Dagster automatically sets certain [environment variables](/dagster-plus/managing-deployments/reserved-environment-variables) containing deployment metadata, allowing us to read these environment variables to discern between deployments. We can access the `DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT` environment variable to determine the currently executing environment.

Because we want to configure our assets to write to Snowflake using a different set of credentials and database in each environment, we’ll configure a separate I/O manager for each environment:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ import os
database_name = os.getenv("DATABASE_NAME")
```

This approach also works for accessing [built-in environment variables for Dagster+](/dagster-plus/managing-deployments/environment-variables-and-secrets#built-in-environment-variables):
This approach also works for accessing [built-in environment variables for Dagster+](/dagster-plus/managing-deployments/reserved-environment-variables):

```python
import os
Expand Down

1 comment on commit ecf9367

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-1nyjmmbwu-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit ecf9367.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.