diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/index.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/index.md index ade5c5b935ffa..0ab0a059c490b 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/index.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/index.md @@ -1,6 +1,6 @@ --- title: Amazon ECS agent -sidebar_position: 50 +sidebar_position: 30 --- import DocCardList from '@theme/DocCardList'; diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/architecture.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/architecture.md index 7c21d41a38430..8afa1ecb5e76a 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/architecture.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/architecture.md @@ -5,25 +5,13 @@ sidebar_position: 10 The Hybrid architecture is the most flexible and secure way to deploy Dagster+. It allows you to run your user code in your environment while leveraging Dagster+'s infrastructure for orchestration and metadata management -
- Pre-requisites - -Before you begin, you should have: - -- A [Dagster+ account](/dagster-plus/getting-started) -- [Basic familiarity with Dagster](/getting-started/quickstart) - -
- ---- - ## Hybrid architecture overview A **hybrid deployment** utilizes a combination of your infrastructure and Dagster-hosted backend services. -The Dagster backend services - including the web frontend, GraphQL API, metadata database, and daemons (responsible for executing schedules and sensors) - are hosted in Dagster+. You are responsible for running an [agent](/todo) in your environment. +The Dagster backend services - including the web frontend, GraphQL API, metadata database, and daemons (responsible for executing schedules and sensors) - are hosted in Dagster+. You are responsible for running an [agent](index.md#dagster-hybrid-agents) in your environment. -![Dagster+ Hybrid deployment architecture](/img/placeholder.svg) +![Dagster+ Hybrid deployment architecture](/images/dagster-cloud/deployment/hybrid-architecture.png) Work is enqueued for your agent when: @@ -35,27 +23,31 @@ The agent polls the agent API to see if any work needs to be done and launches u All user code runs within your environment, in isolation from Dagster system code. ---- - ## The agent Because the agent communicates with the Dagster+ control plane over the agent API, it's possible to support agents that operate in arbitrary compute environments. This means that over time, Dagster+'s support for different user deployment environments will expand and custom agents can take advantage of bespoke compute environments such as HPC. -Refer to the [Agents documentation](/todo) for more info, including the agents that are currently supported. - ---- +See the [setup page](index.md#dagster-hybrid-agents) for a list of agents that are currently supported. ## Security -This section describes how Dagster+ interacts with user code. To summarize: +Dagster+ Hybrid relies on a shared security model. + +The Dagster+ control plane is SOC 2 Type II certified and follows best practices such as: +- encrypting data at rest (AES 256) and in transit (TLS 1.2+) +- highly available, with disaster recovery and backup strategies +- only manages metadata such as pipeline names, execution status, and run duration + +The execution environment is managed by the customer: +- Dagster+ doesn't have access to user code—your code never leaves your environment. Metadata about the code is fetched over constrained APIs. +- All connections to databases, file systems, and other resources are made from your environment. +- The execution environment only requires egress access to Dagster+. No ingress is required from Dagster+ to user environments. -- No ingress is required from Dagster+ to user environments -- Dagster+ doesn't have access to user code. Metadata about the code is fetched over constrained APIs. -- The Dagster+ agent is [open source and auditable](https://github.com/dagster-io/dagster-cloud) +Additionally, the Dagster+ agent is [open source and auditable](https://github.com/dagster-io/dagster-cloud) -These highlights are described in more detail below: +The following highlights are described in more detail below: - [Interactions and queries](#interactions-and-queries) - [Runs](#runs) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/index.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/index.md index cb00c69f21296..81ebfa807f20a 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/index.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/docker/index.md @@ -1,6 +1,6 @@ --- title: Docker agent -sidebar_position: 30 +sidebar_position: 40 --- import DocCardList from '@theme/DocCardList'; diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/index.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/index.md index 3d96293d0c923..0b23dc899ee39 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/index.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/index.md @@ -6,25 +6,26 @@ sidebar_position: 20 In a Dagster+ Hybrid deployment, the orchestration control plane is run by Dagster+ while your Dagster code is executed within your environment. -[comment]: <> (TODO: Architecture diagram) +:::note +For an overview of the Hybrid design, including security considerations, see [Dagster+ Hybrid architecture](architecture.md). +::: ## Get started -To get started with a Hybrid deployment you'll need to: +To get started with a Hybrid deployment, you'll need to: 1. Create a [Dagster+ organization](https://dagster.cloud/signup) -2. Install a Dagster+ Hybrid Agent +2. [Install a Dagster+ Hybrid agent](#dagster-hybrid-agents) 3. [Add a code location](/dagster-plus/deployment/code-locations), typically using a Git repository and CI/CD ## Dagster+ Hybrid agents -The Dagster+ agent is a long-lived process that polls Dagster+'s API servers for new work. +The Dagster+ agent is a long-lived process that polls Dagster+'s API servers for new work. Currently supported agents include: -See the following guides for setting up an agent: - [Kubernetes](/dagster-plus/deployment/deployment-types/hybrid/kubernetes) - [AWS ECS](/dagster-plus/deployment/deployment-types/hybrid/amazon-ecs/new-vpc) - [Docker](/dagster-plus/deployment/deployment-types/hybrid/docker) - - [Locally](/dagster-plus/deployment/deployment-types/hybrid/local) + - [Local agent](/dagster-plus/deployment/deployment-types/hybrid/local) ## What you'll see in your environment @@ -44,20 +45,10 @@ When a run needs to be launched, Dagster+ enqueues instructions for your agent t Your agent will send Dagster+ metadata letting us know the run has been launched. Your run's container will also send Dagster+ metadata informing us of how the run is progressing. The Dagster+ backend services will monitor this stream of metadata to make additional orchestration decisions, monitor for failure, or send alerts. -## Security +## Best practices -Dagster+ hybrid relies on a shared security model. +### Security -The Dagster+ control plane is SOC 2 Type II certified and follows best practices such as: -- encrypting data at rest (AES 256) and in transit (TLS 1.2+) -- highly available, with disaster recovery and backup strategies -- only manages metadata such as pipeline names, execution status, and run duration - -The execution environment is managed by the customer: -- your code never leaves your environment -- all connections to databases, file systems, and other resources are made from your environment -- the execution environment only requires egress access to Dagster+ - -Common security considerations in Dagster+ hybrid include: -- [disabling log forwarding](/todo) -- [managing tokens](/todo) +You can do the following to make your Dagster+ Hybrid deployment more secure: +- [Disable log forwarding](/dagster-plus/deployment/management/settings/customizing-agent-settings#disabling-compute-logs) +- [Manage tokens](/dagster-plus/deployment/management/tokens/agent-tokens) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/kubernetes/index.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/kubernetes/index.md index d744a447c3b87..3b4e765984d84 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/kubernetes/index.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/kubernetes/index.md @@ -1,6 +1,6 @@ --- title: Kubernetes agent -sidebar_position: 40 +sidebar_position: 20 --- import DocCardList from '@theme/DocCardList'; diff --git a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/local.md b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/local.md index 53c43ba09d468..dc199cc03e4a8 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/local.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/deployment-types/hybrid/local.md @@ -1,6 +1,6 @@ --- title: Running a local agent -sidebar_position: 20 +sidebar_position: 50 sidebar_label: Local agent --- diff --git a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md index df595ee48e153..87ad2fab53a4e 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md @@ -4,4 +4,8 @@ sidebar_position: 80 unlisted: true --- -{/* TODO move from https://docs.dagster.io/dagster-plus/deployment/agents/customizing-configuration */} \ No newline at end of file +{/* TODO move from https://docs.dagster.io/dagster-plus/deployment/agents/customizing-configuration */} + +## Disabling compute logs + +{/* NOTE this is a placeholder section so the Hybrid deployment index page has somewhere to link to */} \ No newline at end of file diff --git a/docs/docs-beta/static/images/dagster-cloud/deployment/hybrid-architecture.png b/docs/docs-beta/static/images/dagster-cloud/deployment/hybrid-architecture.png new file mode 100644 index 0000000000000..89ce1b90c0b59 Binary files /dev/null and b/docs/docs-beta/static/images/dagster-cloud/deployment/hybrid-architecture.png differ