-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[docs-beta] migrate - docker agent docs (#26506)
## Summary & Motivation Broken links converted to `/todo`: ``` - Broken link on source page path = /dagster-plus/deployment/deployment-types/hybrid/docker/configuration: -> linking to /dagster-plus/deployment/agents/docker/configuring-running-docker-agent -> linking to /dagster-plus/managing-deployments/environment-variables-and-secrets -> linking to /guides/dagster/using-environment-variables-and-secrets - Broken link on source page path = /dagster-plus/deployment/deployment-types/hybrid/docker/setup: -> linking to /dagster-plus/account/managing-users -> linking to /dagster-plus/managing-deployments/code-locations ``` ## How I Tested These Changes ## Changelog NOCHANGELOG
- Loading branch information
Showing
3 changed files
with
156 additions
and
4 deletions.
There are no files selected for viewing
38 changes: 36 additions & 2 deletions
38
...ta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/configuration.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,41 @@ | ||
--- | ||
title: Docker agent configuration | ||
sidebar_position: 200 | ||
unlisted: true | ||
--- | ||
|
||
{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/docker/configuration-reference */} | ||
:::note | ||
This guide is applicable to Dagster+. | ||
::: | ||
|
||
{/* This reference describes the various configuration options Dagster+ currently supports for [Docker agents](/dagster-plus/deployment/agents/docker/configuring-running-docker-agent). */} | ||
This reference describes the various configuration options Dagster+ currently supports for [Docker agents](/todo). | ||
|
||
--- | ||
|
||
## Environment variables and secrets | ||
|
||
Using the `container_context.docker.env_vars` property, you can include environment variables and secrets in the Docker container associated with a specific code location. For example: | ||
|
||
```yaml | ||
# dagster_cloud.yaml | ||
locations: | ||
- location_name: cloud-examples | ||
image: dagster/dagster-cloud-examples:latest | ||
code_source: | ||
package_name: dagster_cloud_examples | ||
container_context: | ||
docker: | ||
env_vars: | ||
- DATABASE_NAME | ||
- DATABASE_USERNAME=hooli_testing | ||
``` | ||
The `container_context.docker.env_vars` property is a list, where each item can be either `KEY` or `KEY=VALUE`. If only `KEY` is specified, the value will be pulled from the local environment. | ||
|
||
Refer to the following guides for more info about environment variables: | ||
|
||
{/* - [Dagster+ environment variables and secrets](/dagster-plus/managing-deployments/environment-variables-and-secrets) */} | ||
- [Dagster+ environment variables and secrets](/todo) | ||
{/* - [Using environment variables and secrets in Dagster code](/guides/dagster/using-environment-variables-and-secrets) */} | ||
- [Using environment variables and secrets in Dagster code](/todo) | ||
|
122 changes: 120 additions & 2 deletions
122
.../docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/setup.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,125 @@ | ||
--- | ||
title: Docker agent setup | ||
sidebar_position: 100 | ||
unlisted: true | ||
--- | ||
|
||
{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/docker/configuring-running-docker-agent */} | ||
:::note | ||
This guide is applicable to Dagster+. | ||
::: | ||
|
||
In this guide, you'll configure and run a Docker agent. Docker agents are used to launch your code in Docker containers. | ||
|
||
## Prerequisites | ||
|
||
To complete the steps in this guide, you'll need: | ||
|
||
{/* - **Permissions in Dagster+ that allow you to manage agent tokens**. Refer to the [User permissions documentation](/dagster-plus/account/managing-users) for more info. */} | ||
- **Permissions in Dagster+ that allow you to manage agent tokens**. Refer to the [User permissions documentation](/todo) for more info. | ||
- **To have Docker installed** | ||
- **Access to a container registry to which you can push images with Dagster code.** Additionally, your Docker agent must have the permissions required to pull images from the registry. | ||
|
||
This can be: | ||
|
||
- A self-hosted registry, | ||
- A public registry such as [DockerHub](https://hub.docker.com/), or | ||
- A managed offering such as [Amazon ECR](https://aws.amazon.com/ecr/), [Azure CR](https://azure.microsoft.com/en-us/services/container-registry/#overview), or [Google CR](https://cloud.google.com/container-registry) | ||
|
||
## Step 1: Generate a Dagster+ agent token | ||
|
||
In this step, you'll generate a token for the Dagster+ agent. The Dagster+ agent will use this to authenticate to the agent API. | ||
|
||
1. Sign in to your Dagster+ instance. | ||
2. Click the **user menu (your icon) > Organization Settings**. | ||
3. In the **Organization Settings** page, click the **Tokens** tab. | ||
4. Click the **+ Create agent token** button. | ||
5. After the token has been created, click **Reveal token**. | ||
|
||
Keep the token somewhere handy - you'll need it to complete the setup. | ||
|
||
## Step 2: Create a Docker agent | ||
|
||
1. Create a Docker network for your agent: | ||
|
||
```shell | ||
docker network create dagster_cloud_agent | ||
``` | ||
|
||
2. Create a `dagster.yaml` file: | ||
|
||
```yaml | ||
instance_class: | ||
module: dagster_cloud.instance | ||
class: DagsterCloudAgentInstance | ||
|
||
dagster_cloud_api: | ||
agent_token: <YOUR_AGENT_TOKEN> | ||
branch_deployments: true # enables branch deployments | ||
deployment: prod | ||
|
||
user_code_launcher: | ||
module: dagster_cloud.workspace.docker | ||
class: DockerUserCodeLauncher | ||
config: | ||
networks: | ||
- dagster_cloud_agent | ||
``` | ||
3. In the file, fill in the following: | ||
- `agent_token` - Add the agent token you created in [Step 1](#step-1-generate-a-dagster-agent-token) | ||
- `deployment` - Enter the deployment associated with this instance of the agent. | ||
|
||
In the above example, we specified `prod` as the deployment. This is present when Dagster+ organizations are first created. | ||
|
||
4. Save the file. | ||
|
||
## Step 3: Start the agent | ||
|
||
Next, you'll start the agent as a container. Run the following command in the same folder as your `dagster.yaml` file: | ||
|
||
```shell | ||
docker run \ | ||
--network=dagster_cloud_agent \ | ||
--volume $PWD/dagster.yaml:/opt/dagster/app/dagster.yaml:ro \ | ||
--volume /var/run/docker.sock:/var/run/docker.sock \ | ||
-it docker.io/dagster/dagster-cloud-agent:latest \ | ||
dagster-cloud agent run /opt/dagster/app | ||
``` | ||
|
||
This command: | ||
|
||
- Starts the agent with your local `dagster.yaml` mounted as a volume | ||
- Starts the system Docker socket mounted as a volume, allowing the agent to launch containers. | ||
|
||
To view the agent in Dagster+, click the Dagster icon in the top left to navigate to the **Status** page and click the **Agents** tab. You should see the agent running in the **Agent statuses** section: | ||
|
||
![Instance Status](/images/dagster-cloud/agents/dagster-cloud-instance-status.png) | ||
|
||
## Credential Helpers | ||
|
||
If your images are stored in a private registry, configuring a [Docker credentials helper](https://docs.docker.com/engine/reference/commandline/login/#credential-helpers) allows the agent to log in to your registry. The agent image comes with several popular credentials helpers preinstalled: | ||
|
||
- [docker-credential-ecr-login](https://github.com/awslabs/amazon-ecr-credential-helper) | ||
- [docker-credential-gcr](https://github.com/GoogleCloudPlatform/docker-credential-gcr) | ||
|
||
These credential helpers generally are configured in `~/.docker.config.json`. To use one, make sure you mount that file as a volume when you start your agent: | ||
|
||
```shell | ||
docker run \ | ||
--network=dagster_cloud_agent \ | ||
--volume $PWD/dagster.yaml:/opt/dagster/app/dagster.yaml:ro \ | ||
--volume /var/run/docker.sock:/var/run/docker.sock \ | ||
--volume ~/.docker/config.json:/root/.docker/config.json:ro \ | ||
-it docker.io/dagster/dagster-cloud-agent:latest \ | ||
dagster-cloud agent run /opt/dagster/app | ||
``` | ||
|
||
## Next steps | ||
|
||
Now that you've got your agent running, what's next? | ||
|
||
{/* - **If you're getting Dagster+ set up**, the next step is to [add a code location](/dagster-plus/managing-deployments/code-locations) using the agent. */} | ||
- **If you're getting Dagster+ set up**, the next step is to [add a code location](/todo) using the agent. | ||
|
||
{/* - **If you're ready to load your Dagster code**, refer to the [Adding Code to Dagster+](/dagster-plus/managing-deployments/code-locations) guide for more info. */} | ||
- **If you're ready to load your Dagster code**, refer to the [Adding Code to Dagster+](/todo) guide for more info. |
Binary file added
BIN
+151 KB
.../docs-beta/static/images/dagster-cloud/agents/dagster-cloud-instance-status.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.