Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs-beta] migrate - dagster+ settings #26607

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/docs-beta/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,9 @@ Before:

After:

| Key | Value |
|---|---|
| `DAGSTER_CLOUD_DEPLOYMENT_NAME` | The name of the Dagster+ deployment. <br/><br/> **Example:** `prod`. |
| Key | Value |
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All formatting not related to docs/docs-beta/docs/dagster-plus/deployment/management/settings/ is from yarn lint

| ------------------------------------ | --------------------------------------------------------------------------------------------------------- |
| `DAGSTER_CLOUD_DEPLOYMENT_NAME` | The name of the Dagster+ deployment. <br/><br/> **Example:** `prod`. |
| `DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT` | `1` if the deployment is a [branch deployment](/dagster-plus/features/ci-cd/branch-deployments/index.md). |

### Whitespace via `{" "}`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,132 @@
unlisted: true
---

{/* TODO move from https://docs.dagster.io/dagster-plus/deployment/agents/customizing-configuration */}
:::note
This guide is applicable to Dagster+.
:::

{/* /deployment/dagster-instance */}
The Dagster+ Agent is a special variant of the Dagster instance used in [Dagster Open Source](/todo.md) and is configured through the same `dagster.yaml` file. You can customize your agent with these settings.

:::note
{/* /dagster-plus/deployment/agents/kubernetes/configuring-running-kubernetes-agent */}
For [Kubernetes agents](/todo.md) deployed with the Dagster+ Helm chart, you'll need to refer to the Helm chart's config map for customizing the agent.
:::

## Enabling user code server TTL

User code servers support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven’t served requests recently and will spin them back up the next time they’re needed. Configuring TTL can save compute cost because user code servers will spend less time sitting idle.

Check warning on line 21 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.chars-non-standard-quotes] Use standard single quotes or double quotes only. Do not use left or right quotes. Raw Output: {"message": "[Dagster.chars-non-standard-quotes] Use standard single quotes or double quotes only. Do not use left or right quotes.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md", "range": {"start": {"line": 21, "column": 119}}}, "severity": "WARNING"}

Check warning on line 21 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.chars-non-standard-quotes] Use standard single quotes or double quotes only. Do not use left or right quotes. Raw Output: {"message": "[Dagster.chars-non-standard-quotes] Use standard single quotes or double quotes only. Do not use left or right quotes.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md", "range": {"start": {"line": 21, "column": 192}}}, "severity": "WARNING"}
dehume marked this conversation as resolved.
Show resolved Hide resolved

To configure TTL:
```yaml
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance

dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod

user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher
config:
server_ttl:
enabled: true
ttl_seconds: 7200 #2 hours
```

## Streaming compute logs

You can set up streaming compute logs by configuring the log upload interval (in seconds).

```yaml
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance

dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod

user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher

compute_logs:
module: dagster_cloud
class: CloudComputeLogManager
config:
upload_interval: 60
```

## Disabling compute logs

{/* NOTE this is a placeholder section so the Hybrid deployment index page has somewhere to link to */}
You can disable forwarding compute logs to Dagster+ by configuring the `NoOpComputeLogManager` setting:

```yaml
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance

dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod

user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher

compute_logs:
module: dagster.core.storage.noop_compute_log_manager
class: NoOpComputeLogManager
```

## Writing compute logs to AWS S3

{/* /api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager */}

Check warning on line 96 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Terms.engineering] Use 'Python' instead of 'python'. Raw Output: {"message": "[Terms.engineering] Use 'Python' instead of 'python'.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md", "range": {"start": {"line": 96, "column": 10}}}, "severity": "WARNING"}

Check failure on line 96 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Terms] Use 'AWS' instead of 'aws'. Raw Output: {"message": "[Vale.Terms] Use 'AWS' instead of 'aws'.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md", "range": {"start": {"line": 96, "column": 51}}}, "severity": "ERROR"}
You can write compute logs to an AWS S3 bucket by configuring the [dagster_aws.s3.compute_log_manager](/todo.md) module.

You are also able to stream partial compute log files by configuring the log upload interval (in seconds) using the `upload_interval` parameter.

Note: Dagster Labs will neither have nor use your AWS credentials. The Dagster+ UI will be able to show the URLs linking to the compute log files in your S3 bucket when you set the `show_url_only` parameter to `true`.

```yaml
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance

dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod

user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher

compute_logs:
module: dagster_aws.s3.compute_log_manager
class: S3ComputeLogManager
config:
bucket: "mycorp-dagster-compute-logs"
local_dir: "/tmp/cool"
prefix: "dagster-test-"
use_ssl: true
verify: true
verify_cert_path: "/path/to/cert/bundle.pem"
endpoint_url: "http://alternate-s3-host.io"
skip_empty_files: true
upload_interval: 30
upload_extra_args:
ServerSideEncryption: "AES256"
show_url_only: true
region: "us-west-1"
```
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,125 @@
unlisted: true
---

{/* TODO move from https://docs.dagster.io/dagster-plus/managing-deployments/deployment-settings-reference */}
# Dagster+ deployment settings reference

:::note
This guide is applicable to Dagster+.
:::

{/* /dagster-plus */}
This reference describes the settings that can be configured for full deployments in [Dagster+](/todo.md).

{/* /dagster-plus/managing-deployments/managing-deployments#configuring-deployment-settings */}
Refer to the [Managing deployments in Dagster+ guide](/todo.md) for info about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI.

## Settings schema

Settings are formatted using YAML. For example:

```yaml
run_queue:
max_concurrent_runs: 10
tag_concurrency_limits:
- key: "database"
value: "redshift"
limit: 5

run_monitoring:
start_timeout_seconds: 1200
cancel_timeout_seconds: 1200
max_runtime_seconds: 7200

run_retries:
max_retries: 0

sso_default_role: EDITOR
```

## Settings

For each deployment, you can configure settings for:

- [Run queue](#run-queue-run_queue)
- [Run monitoring](#run-monitoring-run_monitoring)
- [Run retries](#run-retries-run_retries)
- [SSO default role](#sso-default-role)
- [Non-isolated runs](#non-isolated-runs)

### Run queue (run_queue)

Check failure on line 52 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'run_queue' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'run_queue' spelled correctly?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 52, "column": 16}}}, "severity": "ERROR"}

Check failure on line 52 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'run_queue'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'run_queue'?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 52, "column": 16}}}, "severity": "ERROR"}

The `run_queue` settings allow you to specify how many runs can execute concurrently in the deployment.

```yaml
run_queue:
max_concurrent_runs: 10
tag_concurrency_limits:
- key: "database"
value: "redshift"
limit: 5
```

| Property | Descripton |

Check failure on line 65 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'Descripton' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'Descripton' spelled correctly?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 65, "column": 14}}}, "severity": "ERROR"}

Check failure on line 65 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Descripton'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Descripton'?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 65, "column": 14}}}, "severity": "ERROR"}
dehume marked this conversation as resolved.
Show resolved Hide resolved
| --- | --- |
| run_queue.max_concurrent_runs | The maximum number of runs that are allowed to be in progress at once. Set to 0 to stop any runs from launching. Negative values aren't permitted. <ul><li>**Default** - `10` (20 minutes)</li><li>**Maximum** - `500` <a href="/todo.md">(Hybrid)</a>, `50` <a href="/todo.md">(Serverless)</a></li></ul> |

Check warning on line 67 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / deploy

Do not use an `<a>` element to navigate. Use the `<Link />` component from `@docusaurus/Link` instead. See: https://docusaurus.io/docs/docusaurus-core#link

Check warning on line 67 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / deploy

Do not use an `<a>` element to navigate. Use the `<Link />` component from `@docusaurus/Link` instead. See: https://docusaurus.io/docs/docusaurus-core#link
| run_queue.tag_concurrency_limits | A list of limits applied to runs with particular tags. <ul><li>**Defaults** - `[]`</li></ul> Each list item may have the following properties: <ul><li>`key`</li><li>`value`</li><ul><li>If defined, the `limit` is applied only to the `key-value` pair.</li><li>If set to a dict with applyLimitPerUniqueValue: true, the `limit` is applied to the number of unique values for the `key`.</li><li>If set to a dict with `applyLimitPerUniqueValue: true`, the limit is applied to the number of unique values for the `key`.</li></ul><li>`limit`</li></ul> |

### Run monitoring (run_monitoring)

Check failure on line 70 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'run_monitoring'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'run_monitoring'?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 70, "column": 21}}}, "severity": "ERROR"}

Check failure on line 70 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'run_monitoring' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'run_monitoring' spelled correctly?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 70, "column": 21}}}, "severity": "ERROR"}

The `run_monitoring` settings allow you to define how long Dagster+ should wait for runs to start before making them as failed, or to terminate before marking them as canceled.

```yaml
run_monitoring:
start_timeout_seconds: 1200
cancel_timeout_seconds: 1200
max_runtime_seconds: 7200
```

| Property | Descripton |

Check failure on line 81 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'Descripton' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'Descripton' spelled correctly?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 81, "column": 14}}}, "severity": "ERROR"}

Check failure on line 81 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Descripton'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Descripton'?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 81, "column": 14}}}, "severity": "ERROR"}
dehume marked this conversation as resolved.
Show resolved Hide resolved
| --- | --- |
| run_monitoring.start_timeout_seconds | The number of seconds that Dagster+ will wait after a run is launched for the process or container to start executing. After the timeout, the run will fail. This prevents runs from hanging in `STARTING` indefinitely when the process or container doesn't start. <ul><li>**Default** - `1200` (20 minutes)</li></ul> |
| run_monitoring.cancel_timeout_seconds | The number of seconds that Dagster+ will wait after a run termination is initiated for the process or container to terminate. After the timeout, the run will move into a `CANCELED` state. This prevents runs from hanging in `CANCELING` indefinitely when the process or container doesn't terminate cleanly. <ul><li>**Default** - `1200` (20 minutes)</li></ul> |
| run_monitoring.max_runtime_seconds | The number of seconds that Dagster+ will wait after a run is moved into a `STARTED` state for the run to complete. After the timeout, the run will be terminated and moved into a `FAILURE` state. This prevents runs from hanging in `STARTED` indefinitely if the process is hanging. <ul><li>**Default** - `No limit`</li></ul> |

### Run retries (run_retries)

The `run_retries` settings allow you to define how Dagster+ handles retrying failed runs in the deployment.

```yaml
run_retries:
max_retries: 0
```

| Property | Descripton |
| --- | --- |
| run_retries.max_retries | The maximum number of times Dagster+ should attempt to retry a failed run. Dagster+ will use the default if this setting is undefined. <ul><li>**Default** - `0`</li></ul> |
| run_retries.retry_on_asset_or_op_failure | Whether to retry runs that failed due to assets or ops in the run failing. Set this to false if you only want to retry failures that occur due to the run worker crashing or unexpectedly terminating, and instead rely on op or asset-level retry policies to retry assert or op failures. Setting this field to false will only change retry behavior for runs on dagster version 1.6.7 or greater. <ul><li>**Default** - `0`</li></ul> |

### SSO default role

{/* dagster-plus/account/managing-users/managing-user-roles-permissions#user-permissions-reference */}
The `sso_default_role` setting lets you configure the default role on the deployment which is granted to new users that log in via SSO. For more information on available roles, see the [Dagster+ permissions reference](/todo.md).

Check warning on line 104 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.latin] Use 'with', 'through', or 'by using' instead of 'via', but consider rewriting the sentence. Raw Output: {"message": "[Dagster.latin] Use 'with', 'through', or 'by using' instead of 'via', but consider rewriting the sentence.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 104, "column": 128}}}, "severity": "WARNING"}

```yaml
sso_default_role: EDITOR
```

| Property | Descripton |
| --- | --- |
| sso_default_role | If SAML SSO is enabled, this is the default role that will be assigned to Dagster+ users for this deployment. If SAML SSO is not enabled, this setting is ignored. <ul><li>**Default** - `Viewer`</li></ul> |

### Non-isolated runs

{/* /dagster-plus/deployment/serverless */}
Configure [non-isolated runs](/todo.md) on your deployment.

```yaml
non_isolated_runs:
enabled: True
max_concurrent_non_isolated_runs: 1
```

| Property | Descripton |
| --- | --- |
| enabled | If enabled, the `Isolate run environment` checkbox will appear in the Launchpad. <ul><li>**Default** - `true`</li></ul> |
| max_concurrent_non_isolated_runs | A limit for how many non-isolated runs to launch at once. Once this limit is reached, the checkbox will be greyed out and all runs will be isolated. This helps to avoid running out of RAM on the code location server. <ul><li>**Default** - `1`</li></ul> |
2 changes: 1 addition & 1 deletion docs/docs-beta/docusaurus.config.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import {themes as prismThemes} from 'prism-react-renderer';
import type {Config} from '@docusaurus/types';
import type * as Preset from '@docusaurus/preset-classic';
import { groupCollapsed } from 'console';
import {groupCollapsed} from 'console';

const config: Config = {
title: 'Dagster Docs - Beta',
Expand Down
Loading
Loading