diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/configuration.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/configuration.md index 1294b30313cfb..99111c0fa0616 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/configuration.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/configuration.md @@ -1,7 +1,41 @@ --- title: Docker agent configuration sidebar_position: 200 -unlisted: true --- -{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/docker/configuration-reference */} \ No newline at end of file +:::note +This guide is applicable to Dagster+. +::: + +{/* This reference describes the various configuration options Dagster+ currently supports for [Docker agents](/dagster-plus/deployment/agents/docker/configuring-running-docker-agent). */} +This reference describes the various configuration options Dagster+ currently supports for [Docker agents](/todo). + +--- + +## Environment variables and secrets + +Using the `container_context.docker.env_vars` property, you can include environment variables and secrets in the Docker container associated with a specific code location. For example: + +```yaml +# dagster_cloud.yaml +locations: + - location_name: cloud-examples + image: dagster/dagster-cloud-examples:latest + code_source: + package_name: dagster_cloud_examples + container_context: + docker: + env_vars: + - DATABASE_NAME + - DATABASE_USERNAME=hooli_testing +``` + +The `container_context.docker.env_vars` property is a list, where each item can be either `KEY` or `KEY=VALUE`. If only `KEY` is specified, the value will be pulled from the local environment. + +Refer to the following guides for more info about environment variables: + +{/* - [Dagster+ environment variables and secrets](/dagster-plus/managing-deployments/environment-variables-and-secrets) */} +- [Dagster+ environment variables and secrets](/todo) +{/* - [Using environment variables and secrets in Dagster code](/guides/dagster/using-environment-variables-and-secrets) */} +- [Using environment variables and secrets in Dagster code](/todo) + diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/setup.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/setup.md index f16e0dc8e25a9..9144b7cb623cf 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/setup.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/setup.md @@ -1,7 +1,125 @@ --- title: Docker agent setup sidebar_position: 100 -unlisted: true --- -{/* TODO copy from https://docs.dagster.io/dagster-plus/deployment/agents/docker/configuring-running-docker-agent */} \ No newline at end of file +:::note +This guide is applicable to Dagster+. +::: + +In this guide, you'll configure and run a Docker agent. Docker agents are used to launch your code in Docker containers. + +## Prerequisites + +To complete the steps in this guide, you'll need: + +{/* - **Permissions in Dagster+ that allow you to manage agent tokens**. Refer to the [User permissions documentation](/dagster-plus/account/managing-users) for more info. */} +- **Permissions in Dagster+ that allow you to manage agent tokens**. Refer to the [User permissions documentation](/todo) for more info. +- **To have Docker installed** +- **Access to a container registry to which you can push images with Dagster code.** Additionally, your Docker agent must have the permissions required to pull images from the registry. + + This can be: + + - A self-hosted registry, + - A public registry such as [DockerHub](https://hub.docker.com/), or + - A managed offering such as [Amazon ECR](https://aws.amazon.com/ecr/), [Azure CR](https://azure.microsoft.com/en-us/services/container-registry/#overview), or [Google CR](https://cloud.google.com/container-registry) + +## Step 1: Generate a Dagster+ agent token + +In this step, you'll generate a token for the Dagster+ agent. The Dagster+ agent will use this to authenticate to the agent API. + +1. Sign in to your Dagster+ instance. +2. Click the **user menu (your icon) > Organization Settings**. +3. In the **Organization Settings** page, click the **Tokens** tab. +4. Click the **+ Create agent token** button. +5. After the token has been created, click **Reveal token**. + +Keep the token somewhere handy - you'll need it to complete the setup. + +## Step 2: Create a Docker agent + +1. Create a Docker network for your agent: + + ```shell + docker network create dagster_cloud_agent + ``` + +2. Create a `dagster.yaml` file: + + ```yaml + instance_class: + module: dagster_cloud.instance + class: DagsterCloudAgentInstance + + dagster_cloud_api: + agent_token: + branch_deployments: true # enables branch deployments + deployment: prod + + user_code_launcher: + module: dagster_cloud.workspace.docker + class: DockerUserCodeLauncher + config: + networks: + - dagster_cloud_agent + ``` + +3. In the file, fill in the following: + + - `agent_token` - Add the agent token you created in [Step 1](#step-1-generate-a-dagster-agent-token) + - `deployment` - Enter the deployment associated with this instance of the agent. + + In the above example, we specified `prod` as the deployment. This is present when Dagster+ organizations are first created. + +4. Save the file. + +## Step 3: Start the agent + +Next, you'll start the agent as a container. Run the following command in the same folder as your `dagster.yaml` file: + +```shell +docker run \ + --network=dagster_cloud_agent \ + --volume $PWD/dagster.yaml:/opt/dagster/app/dagster.yaml:ro \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + -it docker.io/dagster/dagster-cloud-agent:latest \ + dagster-cloud agent run /opt/dagster/app +``` + +This command: + +- Starts the agent with your local `dagster.yaml` mounted as a volume +- Starts the system Docker socket mounted as a volume, allowing the agent to launch containers. + +To view the agent in Dagster+, click the Dagster icon in the top left to navigate to the **Status** page and click the **Agents** tab. You should see the agent running in the **Agent statuses** section: + +![Instance Status](/images/dagster-cloud/agents/dagster-cloud-instance-status.png) + +## Credential Helpers + +If your images are stored in a private registry, configuring a [Docker credentials helper](https://docs.docker.com/engine/reference/commandline/login/#credential-helpers) allows the agent to log in to your registry. The agent image comes with several popular credentials helpers preinstalled: + +- [docker-credential-ecr-login](https://github.com/awslabs/amazon-ecr-credential-helper) +- [docker-credential-gcr](https://github.com/GoogleCloudPlatform/docker-credential-gcr) + +These credential helpers generally are configured in `~/.docker.config.json`. To use one, make sure you mount that file as a volume when you start your agent: + +```shell + docker run \ + --network=dagster_cloud_agent \ + --volume $PWD/dagster.yaml:/opt/dagster/app/dagster.yaml:ro \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + --volume ~/.docker/config.json:/root/.docker/config.json:ro \ + -it docker.io/dagster/dagster-cloud-agent:latest \ + dagster-cloud agent run /opt/dagster/app +``` + +## Next steps + +Now that you've got your agent running, what's next? + +{/* - **If you're getting Dagster+ set up**, the next step is to [add a code location](/dagster-plus/managing-deployments/code-locations) using the agent. */} +- **If you're getting Dagster+ set up**, the next step is to [add a code location](/todo) using the agent. + +{/* - **If you're ready to load your Dagster code**, refer to the [Adding Code to Dagster+](/dagster-plus/managing-deployments/code-locations) guide for more info. */} +- **If you're ready to load your Dagster code**, refer to the [Adding Code to Dagster+](/todo) guide for more info. diff --git a/docs/docs-beta/static/images/dagster-cloud/agents/dagster-cloud-instance-status.png b/docs/docs-beta/static/images/dagster-cloud/agents/dagster-cloud-instance-status.png new file mode 100644 index 0000000000000..507a67f8f8ad9 Binary files /dev/null and b/docs/docs-beta/static/images/dagster-cloud/agents/dagster-cloud-instance-status.png differ