From 334dee5a2b5c4667542b9c44257eda155fd5e984 Mon Sep 17 00:00:00 2001 From: Dennis Hume Date: Thu, 19 Dec 2024 13:24:15 -0600 Subject: [PATCH 1/5] [docs-beta] migrate - dagster+ settings --- docs/docs-beta/CONTRIBUTING.md | 6 +- .../settings/customizing-agent-settings.md | 38 +++++- .../settings/deployment-settings.md | 119 +++++++++++++++++- docs/docs-beta/docusaurus.config.ts | 2 +- docs/docs-beta/sidebars.ts | 49 ++++---- docs/docs-beta/src/theme/MDXComponents.tsx | 2 +- .../management/settings/aws_compute_logs.yaml | 31 +++++ .../management/settings/code_server_ttl.yaml | 17 +++ .../settings/disable_compute_logs.yaml | 17 +++ .../settings/streaming_compute_logs.yaml | 19 +++ 10 files changed, 269 insertions(+), 31 deletions(-) create mode 100644 examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/aws_compute_logs.yaml create mode 100644 examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/code_server_ttl.yaml create mode 100644 examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/disable_compute_logs.yaml create mode 100644 examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/streaming_compute_logs.yaml diff --git a/docs/docs-beta/CONTRIBUTING.md b/docs/docs-beta/CONTRIBUTING.md index c0b8e52ba5712..bd9c06cafa322 100644 --- a/docs/docs-beta/CONTRIBUTING.md +++ b/docs/docs-beta/CONTRIBUTING.md @@ -97,9 +97,9 @@ Before: After: -| Key | Value | -|---|---| -| `DAGSTER_CLOUD_DEPLOYMENT_NAME` | The name of the Dagster+ deployment.

**Example:** `prod`. | +| Key | Value | +| ------------------------------------ | --------------------------------------------------------------------------------------------------------- | +| `DAGSTER_CLOUD_DEPLOYMENT_NAME` | The name of the Dagster+ deployment.

**Example:** `prod`. | | `DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT` | `1` if the deployment is a [branch deployment](/dagster-plus/features/ci-cd/branch-deployments/index.md). | ### Whitespace via `{" "}` diff --git a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md index 87ad2fab53a4e..84b91b53d253e 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md @@ -4,8 +4,42 @@ sidebar_position: 80 unlisted: true --- -{/* TODO move from https://docs.dagster.io/dagster-plus/deployment/agents/customizing-configuration */} +:::note +This guide is applicable to Dagster+. +::: + +The Dagster+ Agent is a special variant of the Dagster instance used in [Dagster Open Source](/deployment/dagster-instance) and is configured through the same `dagster.yaml` file. You can customize your agent with these settings. + +:::note +For [Kubernetes agents](/dagster-plus/deployment/agents/kubernetes/configuring-running-kubernetes-agent) deployed with the Dagster+ Helm chart, you'll need to refer to the Helm chart's config map for customizing the agent. +::: + +## Enabling user code server TTL + +User code servers support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven’t served requests recently and will spin them back up the next time they’re needed. Configuring TTL can save compute cost because user code servers will spend less time sitting idle. + +To configure TTL: + + + +## Streaming compute logs + +You can set up streaming compute logs by configuring the log upload interval (in seconds). + + ## Disabling compute logs -{/* NOTE this is a placeholder section so the Hybrid deployment index page has somewhere to link to */} \ No newline at end of file +You can disable forwarding compute logs to Dagster+ by configuring the `NoOpComputeLogManager` setting: + + + +## Writing compute logs to AWS S3 + +You can write compute logs to an AWS S3 bucket by configuring the [dagster_aws.s3.compute_log_manager](/api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager) module. + +You are also able to stream partial compute log files by configuring the log upload interval (in seconds) using the `upload_interval` parameter. + +Note: Dagster Labs will neither have nor use your AWS credentials. The Dagster+ UI will be able to show the URLs linking to the compute log files in your S3 bucket when you set the `show_url_only` parameter to `true`. + + \ No newline at end of file diff --git a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md index c0cc4dbecca60..481437fa1b920 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md @@ -4,4 +4,121 @@ sidebar_position: 200 unlisted: true --- -{/* TODO move from https://docs.dagster.io/dagster-plus/managing-deployments/deployment-settings-reference */} +# Dagster+ deployment settings reference + +:::note +This guide is applicable to Dagster+. +::: + +This reference describes the settings that can be configured for full deployments in [Dagster+](/dagster-plus). + +Refer to the [Managing deployments in Dagster+ guide](/dagster-plus/managing-deployments/managing-deployments#configuring-deployment-settings) for info about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI. + +## Settings schema + +Settings are formatted using YAML. For example: + +```yaml +run_queue: + max_concurrent_runs: 10 + tag_concurrency_limits: + - key: "database" + value: "redshift" + limit: 5 + +run_monitoring: + start_timeout_seconds: 1200 + cancel_timeout_seconds: 1200 + max_runtime_seconds: 7200 + +run_retries: + max_retries: 0 + +sso_default_role: EDITOR +``` + +## Settings + +For each deployment, you can configure settings for: + +- [Run queue](#run-queue-run_queue) +- [Run monitoring](#run-monitoring-run_monitoring) +- [Run retries](#run-retries-run_retries) +- [SSO default role](#sso-default-role) +- [Non-isolated runs](#non-isolated-runs) + +### Run queue (run_queue) + +The `run_queue` settings allow you to specify how many runs can execute concurrently in the deployment. + +```yaml +run_queue: + max_concurrent_runs: 10 + tag_concurrency_limits: + - key: "database" + value: "redshift" + limit: 5 +``` + +| Property | Descripton | +| --- | --- | +| run_queue.max_concurrent_runs | The maximum number of runs that are allowed to be in progress at once. Set to 0 to stop any runs from launching. Negative values aren't permitted.
  • **Default** - `10` (20 minutes)
  • **Maximum** - `500` (Hybrid)[/dagster-plus/deployment/hybrid], `50` (Serverless)[/dagster-plus/deployment/serverless]
| +| run_queue.tag_concurrency_limits | A list of limits applied to runs with particular tags.
  • **Defaults** - `[]`
Each list item may have the following properties:
  • `key`
  • `value`
    • If defined, the `limit` is applied only to the `key-value` pair.
    • If set to a dict with applyLimitPerUniqueValue: true, the `limit` is applied to the number of unique values for the `key`.
    • If set to a dict with `applyLimitPerUniqueValue: true`, the limit is applied to the number of unique values for the `key`.
  • `limit`
| + +### Run monitoring (run_monitoring) + +The `run_monitoring` settings allow you to define how long Dagster+ should wait for runs to start before making them as failed, or to terminate before marking them as canceled. + +```yaml +run_monitoring: + start_timeout_seconds: 1200 + cancel_timeout_seconds: 1200 + max_runtime_seconds: 7200 +``` + +| Property | Descripton | +| --- | --- | +| run_monitoring.start_timeout_seconds | The number of seconds that Dagster+ will wait after a run is launched for the process or container to start executing. After the timeout, the run will fail. This prevents runs from hanging in `STARTING` indefinitely when the process or container doesn't start.
  • **Default** - `1200` (20 minutes)
| +| run_monitoring.cancel_timeout_seconds | The number of seconds that Dagster+ will wait after a run termination is initiated for the process or container to terminate. After the timeout, the run will move into a `CANCELED` state. This prevents runs from hanging in `CANCELING` indefinitely when the process or container doesn't terminate cleanly.
  • **Default** - `1200` (20 minutes)
| +| run_monitoring.max_runtime_seconds | The number of seconds that Dagster+ will wait after a run is moved into a `STARTED` state for the run to complete. After the timeout, the run will be terminated and moved into a `FAILURE` state. This prevents runs from hanging in `STARTED` indefinitely if the process is hanging.
  • **Default** - `No limit`
| + +### Run retries (run_retries) + +The `run_retries` settings allow you to define how Dagster+ handles retrying failed runs in the deployment. + +```yaml +run_retries: + max_retries: 0 +``` + +| Property | Descripton | +| --- | --- | +| run_retries.max_retries | The maximum number of times Dagster+ should attempt to retry a failed run. Dagster+ will use the default if this setting is undefined.
  • **Default** - `0`
| +| run_retries.retry_on_asset_or_op_failure | Whether to retry runs that failed due to assets or ops in the run failing. Set this to false if you only want to retry failures that occur due to the run worker crashing or unexpectedly terminating, and instead rely on op or asset-level retry policies to retry assert or op failures. Setting this field to false will only change retry behavior for runs on dagster version 1.6.7 or greater.
  • **Default** - `0`
| + +### SSO default role + +The `sso_default_role` setting lets you configure the default role on the deployment which is granted to new users that log in via SSO. For more information on available roles, see the [Dagster+ permissions reference](/dagster-plus/account/managing-users/managing-user-roles-permissions#user-permissions-reference). + +```yaml +sso_default_role: EDITOR +``` + +| Property | Descripton | +| --- | --- | +| sso_default_role | If SAML SSO is enabled, this is the default role that will be assigned to Dagster+ users for this deployment. If SAML SSO is not enabled, this setting is ignored.
  • **Default** - `Viewer`
| + +### Non-isolated runs + +Configure [non-isolated runs](/dagster-plus/deployment/serverless) on your deployment. + +```yaml +non_isolated_runs: + enabled: True + max_concurrent_non_isolated_runs: 1 +``` + +| Property | Descripton | +| --- | --- | +| enabled | If enabled, the `Isolate run environment` checkbox will appear in the Launchpad.
  • **Default** - `true`
| +| max_concurrent_non_isolated_runs | A limit for how many non-isolated runs to launch at once. Once this limit is reached, the checkbox will be greyed out and all runs will be isolated. This helps to avoid running out of RAM on the code location server.
  • **Default** - `1`
| \ No newline at end of file diff --git a/docs/docs-beta/docusaurus.config.ts b/docs/docs-beta/docusaurus.config.ts index 7a3f8dd9f75af..eab1dc1c27b85 100644 --- a/docs/docs-beta/docusaurus.config.ts +++ b/docs/docs-beta/docusaurus.config.ts @@ -1,7 +1,7 @@ import {themes as prismThemes} from 'prism-react-renderer'; import type {Config} from '@docusaurus/types'; import type * as Preset from '@docusaurus/preset-classic'; -import { groupCollapsed } from 'console'; +import {groupCollapsed} from 'console'; const config: Config = { title: 'Dagster Docs - Beta', diff --git a/docs/docs-beta/sidebars.ts b/docs/docs-beta/sidebars.ts index 03ca26fa27bcb..3ebf22cbc96b0 100644 --- a/docs/docs-beta/sidebars.ts +++ b/docs/docs-beta/sidebars.ts @@ -5,7 +5,12 @@ const sidebars: SidebarsConfig = { type: 'category', label: 'Getting Started', collapsed: false, - items: ['intro', 'getting-started/quickstart', 'getting-started/installation', 'getting-started/glossary'], + items: [ + 'intro', + 'getting-started/quickstart', + 'getting-started/installation', + 'getting-started/glossary', + ], }, { type: 'category', @@ -21,8 +26,8 @@ const sidebars: SidebarsConfig = { items: [ { type: 'autogenerated', - dirName: 'guides/build' - } + dirName: 'guides/build', + }, ], }, { @@ -33,8 +38,8 @@ const sidebars: SidebarsConfig = { items: [ { type: 'autogenerated', - dirName: 'guides/automate' - } + dirName: 'guides/automate', + }, ], }, { @@ -45,7 +50,7 @@ const sidebars: SidebarsConfig = { items: [ { type: 'autogenerated', - dirName: 'guides/monitor' + dirName: 'guides/monitor', }, ], }, @@ -57,8 +62,8 @@ const sidebars: SidebarsConfig = { items: [ { type: 'autogenerated', - dirName: 'guides/test' - } + dirName: 'guides/test', + }, ], }, { @@ -69,8 +74,8 @@ const sidebars: SidebarsConfig = { items: [ { type: 'autogenerated', - dirName: 'guides/deploy' - } + dirName: 'guides/deploy', + }, ], }, { @@ -90,9 +95,7 @@ const sidebars: SidebarsConfig = { type: 'category', label: 'Guides', collapsed: false, - items: [ - 'integrations/guides/multi-asset-integration' - ] + items: ['integrations/guides/multi-asset-integration'], }, { type: 'category', @@ -102,10 +105,10 @@ const sidebars: SidebarsConfig = { items: [ { type: 'autogenerated', - dirName: 'integrations/libraries' - } - ] - } + dirName: 'integrations/libraries', + }, + ], + }, ], dagsterPlus: [ 'dagster-plus/index', @@ -117,9 +120,9 @@ const sidebars: SidebarsConfig = { items: [ { type: 'autogenerated', - dirName: 'dagster-plus/features' - } - ] + dirName: 'dagster-plus/features', + }, + ], }, { type: 'category', @@ -128,8 +131,8 @@ const sidebars: SidebarsConfig = { items: [ { type: 'autogenerated', - dirName: 'dagster-plus/deployment' - } + dirName: 'dagster-plus/deployment', + }, ], }, ], @@ -147,7 +150,7 @@ const sidebars: SidebarsConfig = { dirName: 'api/python-api', }, ], - } + }, ], }; diff --git a/docs/docs-beta/src/theme/MDXComponents.tsx b/docs/docs-beta/src/theme/MDXComponents.tsx index d26d4fe3c81ca..d780d971c00ac 100644 --- a/docs/docs-beta/src/theme/MDXComponents.tsx +++ b/docs/docs-beta/src/theme/MDXComponents.tsx @@ -1,6 +1,6 @@ // Import the original mapper import MDXComponents from '@theme-original/MDXComponents'; -import { PyObject } from '../components/PyObject'; +import {PyObject} from '../components/PyObject'; import CodeExample from '../components/CodeExample'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; diff --git a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/aws_compute_logs.yaml b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/aws_compute_logs.yaml new file mode 100644 index 0000000000000..73be7fe037c96 --- /dev/null +++ b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/aws_compute_logs.yaml @@ -0,0 +1,31 @@ +# dagster.yaml +instance_class: + module: dagster_cloud.instance + class: DagsterCloudAgentInstance + +dagster_cloud_api: + agent_token: + env: DAGSTER_CLOUD_AGENT_TOKEN + deployment: prod + +user_code_launcher: + module: dagster_cloud.workspace.docker + class: DockerUserCodeLauncher + +compute_logs: + module: dagster_aws.s3.compute_log_manager + class: S3ComputeLogManager + config: + bucket: "mycorp-dagster-compute-logs" + local_dir: "/tmp/cool" + prefix: "dagster-test-" + use_ssl: true + verify: true + verify_cert_path: "/path/to/cert/bundle.pem" + endpoint_url: "http://alternate-s3-host.io" + skip_empty_files: true + upload_interval: 30 + upload_extra_args: + ServerSideEncryption: "AES256" + show_url_only: true + region: "us-west-1" \ No newline at end of file diff --git a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/code_server_ttl.yaml b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/code_server_ttl.yaml new file mode 100644 index 0000000000000..ba65835936118 --- /dev/null +++ b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/code_server_ttl.yaml @@ -0,0 +1,17 @@ +# dagster.yaml +instance_class: + module: dagster_cloud.instance + class: DagsterCloudAgentInstance + +dagster_cloud_api: + agent_token: + env: DAGSTER_CLOUD_AGENT_TOKEN + deployment: prod + +user_code_launcher: + module: dagster_cloud.workspace.docker + class: DockerUserCodeLauncher + config: + server_ttl: + enabled: true + ttl_seconds: 7200 #2 hours \ No newline at end of file diff --git a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/disable_compute_logs.yaml b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/disable_compute_logs.yaml new file mode 100644 index 0000000000000..4964d8d213790 --- /dev/null +++ b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/disable_compute_logs.yaml @@ -0,0 +1,17 @@ +# dagster.yaml +instance_class: + module: dagster_cloud.instance + class: DagsterCloudAgentInstance + +dagster_cloud_api: + agent_token: + env: DAGSTER_CLOUD_AGENT_TOKEN + deployment: prod + +user_code_launcher: + module: dagster_cloud.workspace.docker + class: DockerUserCodeLauncher + +compute_logs: + module: dagster.core.storage.noop_compute_log_manager + class: NoOpComputeLogManager \ No newline at end of file diff --git a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/streaming_compute_logs.yaml b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/streaming_compute_logs.yaml new file mode 100644 index 0000000000000..09606881a90e2 --- /dev/null +++ b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/streaming_compute_logs.yaml @@ -0,0 +1,19 @@ +# dagster.yaml +instance_class: + module: dagster_cloud.instance + class: DagsterCloudAgentInstance + +dagster_cloud_api: + agent_token: + env: DAGSTER_CLOUD_AGENT_TOKEN + deployment: prod + +user_code_launcher: + module: dagster_cloud.workspace.docker + class: DockerUserCodeLauncher + +compute_logs: + module: dagster_cloud + class: CloudComputeLogManager + config: + upload_interval: 60 \ No newline at end of file From a964fdd8d695923e130cf71d89c626257f9f00ae Mon Sep 17 00:00:00 2001 From: Dennis Hume Date: Fri, 20 Dec 2024 09:26:29 -0600 Subject: [PATCH 2/5] Update links --- .../settings/customizing-agent-settings.md | 106 ++++++++++++++++-- .../settings/deployment-settings.md | 14 ++- .../management/settings/aws_compute_logs.yaml | 31 ----- .../management/settings/code_server_ttl.yaml | 17 --- .../settings/disable_compute_logs.yaml | 17 --- .../settings/streaming_compute_logs.yaml | 19 ---- 6 files changed, 107 insertions(+), 97 deletions(-) delete mode 100644 examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/aws_compute_logs.yaml delete mode 100644 examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/code_server_ttl.yaml delete mode 100644 examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/disable_compute_logs.yaml delete mode 100644 examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/streaming_compute_logs.yaml diff --git a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md index 84b91b53d253e..e4997a87ed5fc 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md @@ -8,10 +8,12 @@ unlisted: true This guide is applicable to Dagster+. ::: -The Dagster+ Agent is a special variant of the Dagster instance used in [Dagster Open Source](/deployment/dagster-instance) and is configured through the same `dagster.yaml` file. You can customize your agent with these settings. +{/* /deployment/dagster-instance */} +The Dagster+ Agent is a special variant of the Dagster instance used in [Dagster Open Source](/todo.md) and is configured through the same `dagster.yaml` file. You can customize your agent with these settings. :::note -For [Kubernetes agents](/dagster-plus/deployment/agents/kubernetes/configuring-running-kubernetes-agent) deployed with the Dagster+ Helm chart, you'll need to refer to the Helm chart's config map for customizing the agent. +{/* /dagster-plus/deployment/agents/kubernetes/configuring-running-kubernetes-agent */} +For [Kubernetes agents](/todo.md) deployed with the Dagster+ Helm chart, you'll need to refer to the Helm chart's config map for customizing the agent. ::: ## Enabling user code server TTL @@ -19,27 +21,115 @@ For [Kubernetes agents](/dagster-plus/deployment/agents/kubernetes/configuring-r User code servers support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven’t served requests recently and will spin them back up the next time they’re needed. Configuring TTL can save compute cost because user code servers will spend less time sitting idle. To configure TTL: - - +```yaml +# dagster.yaml +instance_class: + module: dagster_cloud.instance + class: DagsterCloudAgentInstance + +dagster_cloud_api: + agent_token: + env: DAGSTER_CLOUD_AGENT_TOKEN + deployment: prod + +user_code_launcher: + module: dagster_cloud.workspace.docker + class: DockerUserCodeLauncher + config: + server_ttl: + enabled: true + ttl_seconds: 7200 #2 hours +``` ## Streaming compute logs You can set up streaming compute logs by configuring the log upload interval (in seconds). - +```yaml +# dagster.yaml +instance_class: + module: dagster_cloud.instance + class: DagsterCloudAgentInstance + +dagster_cloud_api: + agent_token: + env: DAGSTER_CLOUD_AGENT_TOKEN + deployment: prod + +user_code_launcher: + module: dagster_cloud.workspace.docker + class: DockerUserCodeLauncher + +compute_logs: + module: dagster_cloud + class: CloudComputeLogManager + config: + upload_interval: 60 +``` ## Disabling compute logs You can disable forwarding compute logs to Dagster+ by configuring the `NoOpComputeLogManager` setting: - +```yaml +# dagster.yaml +instance_class: + module: dagster_cloud.instance + class: DagsterCloudAgentInstance + +dagster_cloud_api: + agent_token: + env: DAGSTER_CLOUD_AGENT_TOKEN + deployment: prod + +user_code_launcher: + module: dagster_cloud.workspace.docker + class: DockerUserCodeLauncher + +compute_logs: + module: dagster.core.storage.noop_compute_log_manager + class: NoOpComputeLogManager +``` ## Writing compute logs to AWS S3 -You can write compute logs to an AWS S3 bucket by configuring the [dagster_aws.s3.compute_log_manager](/api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager) module. +{/* /api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager */} +You can write compute logs to an AWS S3 bucket by configuring the [dagster_aws.s3.compute_log_manager](/todo.md) module. You are also able to stream partial compute log files by configuring the log upload interval (in seconds) using the `upload_interval` parameter. Note: Dagster Labs will neither have nor use your AWS credentials. The Dagster+ UI will be able to show the URLs linking to the compute log files in your S3 bucket when you set the `show_url_only` parameter to `true`. - \ No newline at end of file +```yaml +# dagster.yaml +instance_class: + module: dagster_cloud.instance + class: DagsterCloudAgentInstance + +dagster_cloud_api: + agent_token: + env: DAGSTER_CLOUD_AGENT_TOKEN + deployment: prod + +user_code_launcher: + module: dagster_cloud.workspace.docker + class: DockerUserCodeLauncher + +compute_logs: + module: dagster_aws.s3.compute_log_manager + class: S3ComputeLogManager + config: + bucket: "mycorp-dagster-compute-logs" + local_dir: "/tmp/cool" + prefix: "dagster-test-" + use_ssl: true + verify: true + verify_cert_path: "/path/to/cert/bundle.pem" + endpoint_url: "http://alternate-s3-host.io" + skip_empty_files: true + upload_interval: 30 + upload_extra_args: + ServerSideEncryption: "AES256" + show_url_only: true + region: "us-west-1" +``` \ No newline at end of file diff --git a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md index 481437fa1b920..202171d7fdb23 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md @@ -10,9 +10,11 @@ unlisted: true This guide is applicable to Dagster+. ::: -This reference describes the settings that can be configured for full deployments in [Dagster+](/dagster-plus). +{/* /dagster-plus */} +This reference describes the settings that can be configured for full deployments in [Dagster+](/todo.md). -Refer to the [Managing deployments in Dagster+ guide](/dagster-plus/managing-deployments/managing-deployments#configuring-deployment-settings) for info about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI. +{/* /dagster-plus/managing-deployments/managing-deployments#configuring-deployment-settings */} +Refer to the [Managing deployments in Dagster+ guide](/todo.md) for info about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI. ## Settings schema @@ -62,7 +64,7 @@ run_queue: | Property | Descripton | | --- | --- | -| run_queue.max_concurrent_runs | The maximum number of runs that are allowed to be in progress at once. Set to 0 to stop any runs from launching. Negative values aren't permitted.
  • **Default** - `10` (20 minutes)
  • **Maximum** - `500` (Hybrid)[/dagster-plus/deployment/hybrid], `50` (Serverless)[/dagster-plus/deployment/serverless]
| +| run_queue.max_concurrent_runs | The maximum number of runs that are allowed to be in progress at once. Set to 0 to stop any runs from launching. Negative values aren't permitted. | | run_queue.tag_concurrency_limits | A list of limits applied to runs with particular tags.
  • **Defaults** - `[]`
Each list item may have the following properties:
  • `key`
  • `value`
    • If defined, the `limit` is applied only to the `key-value` pair.
    • If set to a dict with applyLimitPerUniqueValue: true, the `limit` is applied to the number of unique values for the `key`.
    • If set to a dict with `applyLimitPerUniqueValue: true`, the limit is applied to the number of unique values for the `key`.
  • `limit`
| ### Run monitoring (run_monitoring) @@ -98,7 +100,8 @@ run_retries: ### SSO default role -The `sso_default_role` setting lets you configure the default role on the deployment which is granted to new users that log in via SSO. For more information on available roles, see the [Dagster+ permissions reference](/dagster-plus/account/managing-users/managing-user-roles-permissions#user-permissions-reference). +{/* dagster-plus/account/managing-users/managing-user-roles-permissions#user-permissions-reference */} +The `sso_default_role` setting lets you configure the default role on the deployment which is granted to new users that log in via SSO. For more information on available roles, see the [Dagster+ permissions reference](/todo.md). ```yaml sso_default_role: EDITOR @@ -110,7 +113,8 @@ sso_default_role: EDITOR ### Non-isolated runs -Configure [non-isolated runs](/dagster-plus/deployment/serverless) on your deployment. +{/* /dagster-plus/deployment/serverless */} +Configure [non-isolated runs](/todo.md) on your deployment. ```yaml non_isolated_runs: diff --git a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/aws_compute_logs.yaml b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/aws_compute_logs.yaml deleted file mode 100644 index 73be7fe037c96..0000000000000 --- a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/aws_compute_logs.yaml +++ /dev/null @@ -1,31 +0,0 @@ -# dagster.yaml -instance_class: - module: dagster_cloud.instance - class: DagsterCloudAgentInstance - -dagster_cloud_api: - agent_token: - env: DAGSTER_CLOUD_AGENT_TOKEN - deployment: prod - -user_code_launcher: - module: dagster_cloud.workspace.docker - class: DockerUserCodeLauncher - -compute_logs: - module: dagster_aws.s3.compute_log_manager - class: S3ComputeLogManager - config: - bucket: "mycorp-dagster-compute-logs" - local_dir: "/tmp/cool" - prefix: "dagster-test-" - use_ssl: true - verify: true - verify_cert_path: "/path/to/cert/bundle.pem" - endpoint_url: "http://alternate-s3-host.io" - skip_empty_files: true - upload_interval: 30 - upload_extra_args: - ServerSideEncryption: "AES256" - show_url_only: true - region: "us-west-1" \ No newline at end of file diff --git a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/code_server_ttl.yaml b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/code_server_ttl.yaml deleted file mode 100644 index ba65835936118..0000000000000 --- a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/code_server_ttl.yaml +++ /dev/null @@ -1,17 +0,0 @@ -# dagster.yaml -instance_class: - module: dagster_cloud.instance - class: DagsterCloudAgentInstance - -dagster_cloud_api: - agent_token: - env: DAGSTER_CLOUD_AGENT_TOKEN - deployment: prod - -user_code_launcher: - module: dagster_cloud.workspace.docker - class: DockerUserCodeLauncher - config: - server_ttl: - enabled: true - ttl_seconds: 7200 #2 hours \ No newline at end of file diff --git a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/disable_compute_logs.yaml b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/disable_compute_logs.yaml deleted file mode 100644 index 4964d8d213790..0000000000000 --- a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/disable_compute_logs.yaml +++ /dev/null @@ -1,17 +0,0 @@ -# dagster.yaml -instance_class: - module: dagster_cloud.instance - class: DagsterCloudAgentInstance - -dagster_cloud_api: - agent_token: - env: DAGSTER_CLOUD_AGENT_TOKEN - deployment: prod - -user_code_launcher: - module: dagster_cloud.workspace.docker - class: DockerUserCodeLauncher - -compute_logs: - module: dagster.core.storage.noop_compute_log_manager - class: NoOpComputeLogManager \ No newline at end of file diff --git a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/streaming_compute_logs.yaml b/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/streaming_compute_logs.yaml deleted file mode 100644 index 09606881a90e2..0000000000000 --- a/examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/settings/streaming_compute_logs.yaml +++ /dev/null @@ -1,19 +0,0 @@ -# dagster.yaml -instance_class: - module: dagster_cloud.instance - class: DagsterCloudAgentInstance - -dagster_cloud_api: - agent_token: - env: DAGSTER_CLOUD_AGENT_TOKEN - deployment: prod - -user_code_launcher: - module: dagster_cloud.workspace.docker - class: DockerUserCodeLauncher - -compute_logs: - module: dagster_cloud - class: CloudComputeLogManager - config: - upload_interval: 60 \ No newline at end of file From 66e5bce4472e2f4dd00ea7cfbfe64fe1a114ef49 Mon Sep 17 00:00:00 2001 From: Dennis Hume Date: Fri, 20 Dec 2024 10:49:00 -0600 Subject: [PATCH 3/5] Update docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md Co-authored-by: Nikki Everett --- .../management/settings/customizing-agent-settings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md index e4997a87ed5fc..726017a31251e 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md @@ -18,7 +18,7 @@ For [Kubernetes agents](/todo.md) deployed with the Dagster+ Helm chart, you'll ## Enabling user code server TTL -User code servers support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven’t served requests recently and will spin them back up the next time they’re needed. Configuring TTL can save compute cost because user code servers will spend less time sitting idle. +User code servers support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven't served requests recently and will spin them back up the next time they're needed. Configuring TTL can save compute cost because user code servers will spend less time sitting idle. To configure TTL: ```yaml From 386638ade94b24cea52ba8a514924d0812483646 Mon Sep 17 00:00:00 2001 From: Dennis Hume Date: Fri, 20 Dec 2024 10:49:08 -0600 Subject: [PATCH 4/5] Update docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md Co-authored-by: Nikki Everett --- .../deployment/management/settings/deployment-settings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md index 202171d7fdb23..9b8f1e669a9d7 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md @@ -62,7 +62,7 @@ run_queue: limit: 5 ``` -| Property | Descripton | +| Property | Description | | --- | --- | | run_queue.max_concurrent_runs | The maximum number of runs that are allowed to be in progress at once. Set to 0 to stop any runs from launching. Negative values aren't permitted. | | run_queue.tag_concurrency_limits | A list of limits applied to runs with particular tags.
  • **Defaults** - `[]`
Each list item may have the following properties:
  • `key`
  • `value`
    • If defined, the `limit` is applied only to the `key-value` pair.
    • If set to a dict with applyLimitPerUniqueValue: true, the `limit` is applied to the number of unique values for the `key`.
    • If set to a dict with `applyLimitPerUniqueValue: true`, the limit is applied to the number of unique values for the `key`.
  • `limit`
| From 587a4e8bc98c73ca0c562a93932342fd08c528a5 Mon Sep 17 00:00:00 2001 From: Dennis Hume Date: Fri, 20 Dec 2024 10:49:15 -0600 Subject: [PATCH 5/5] Update docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md Co-authored-by: Nikki Everett --- .../deployment/management/settings/deployment-settings.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md index 9b8f1e669a9d7..0881eeafba5ce 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md @@ -78,7 +78,7 @@ run_monitoring: max_runtime_seconds: 7200 ``` -| Property | Descripton | +| Property | Description | | --- | --- | | run_monitoring.start_timeout_seconds | The number of seconds that Dagster+ will wait after a run is launched for the process or container to start executing. After the timeout, the run will fail. This prevents runs from hanging in `STARTING` indefinitely when the process or container doesn't start.
  • **Default** - `1200` (20 minutes)
| | run_monitoring.cancel_timeout_seconds | The number of seconds that Dagster+ will wait after a run termination is initiated for the process or container to terminate. After the timeout, the run will move into a `CANCELED` state. This prevents runs from hanging in `CANCELING` indefinitely when the process or container doesn't terminate cleanly.
  • **Default** - `1200` (20 minutes)
|