Skip to content

Commit

Permalink
Update links
Browse files Browse the repository at this point in the history
  • Loading branch information
dehume committed Dec 20, 2024
1 parent 334dee5 commit a964fdd
Show file tree
Hide file tree
Showing 6 changed files with 107 additions and 97 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8,38 +8,128 @@ unlisted: true
This guide is applicable to Dagster+.
:::

The Dagster+ Agent is a special variant of the Dagster instance used in [Dagster Open Source](/deployment/dagster-instance) and is configured through the same `dagster.yaml` file. You can customize your agent with these settings.
{/* /deployment/dagster-instance */}
The Dagster+ Agent is a special variant of the Dagster instance used in [Dagster Open Source](/todo.md) and is configured through the same `dagster.yaml` file. You can customize your agent with these settings.

:::note
For [Kubernetes agents](/dagster-plus/deployment/agents/kubernetes/configuring-running-kubernetes-agent) deployed with the Dagster+ Helm chart, you'll need to refer to the Helm chart's config map for customizing the agent.
{/* /dagster-plus/deployment/agents/kubernetes/configuring-running-kubernetes-agent */}
For [Kubernetes agents](/todo.md) deployed with the Dagster+ Helm chart, you'll need to refer to the Helm chart's config map for customizing the agent.
:::

## Enabling user code server TTL

User code servers support a configurable time-to-live (TTL). The agent will spin down any user code servers that haven’t served requests recently and will spin them back up the next time they’re needed. Configuring TTL can save compute cost because user code servers will spend less time sitting idle.

Check warning on line 21 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.chars-non-standard-quotes] Use standard single quotes or double quotes only. Do not use left or right quotes. Raw Output: {"message": "[Dagster.chars-non-standard-quotes] Use standard single quotes or double quotes only. Do not use left or right quotes.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md", "range": {"start": {"line": 21, "column": 119}}}, "severity": "WARNING"}

Check warning on line 21 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.chars-non-standard-quotes] Use standard single quotes or double quotes only. Do not use left or right quotes. Raw Output: {"message": "[Dagster.chars-non-standard-quotes] Use standard single quotes or double quotes only. Do not use left or right quotes.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md", "range": {"start": {"line": 21, "column": 192}}}, "severity": "WARNING"}

To configure TTL:

<CodeExample filePath="./examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/code_server_ttl.yaml" />
```yaml
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance

dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod

user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher
config:
server_ttl:
enabled: true
ttl_seconds: 7200 #2 hours
```
## Streaming compute logs
You can set up streaming compute logs by configuring the log upload interval (in seconds).
<CodeExample filePath="./examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/streaming_compute_logs.yaml" />
```yaml
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance

dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod

user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher

compute_logs:
module: dagster_cloud
class: CloudComputeLogManager
config:
upload_interval: 60
```
## Disabling compute logs
You can disable forwarding compute logs to Dagster+ by configuring the `NoOpComputeLogManager` setting:

<CodeExample filePath="./examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/disable_compute_logs.yaml" />
```yaml
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance
dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod
user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher
compute_logs:
module: dagster.core.storage.noop_compute_log_manager
class: NoOpComputeLogManager
```

## Writing compute logs to AWS S3

You can write compute logs to an AWS S3 bucket by configuring the [dagster_aws.s3.compute_log_manager](/api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager) module.
{/* /api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager */}

Check warning on line 96 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Terms.engineering] Use 'Python' instead of 'python'. Raw Output: {"message": "[Terms.engineering] Use 'Python' instead of 'python'.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md", "range": {"start": {"line": 96, "column": 10}}}, "severity": "WARNING"}

Check failure on line 96 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Terms] Use 'AWS' instead of 'aws'. Raw Output: {"message": "[Vale.Terms] Use 'AWS' instead of 'aws'.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md", "range": {"start": {"line": 96, "column": 51}}}, "severity": "ERROR"}
You can write compute logs to an AWS S3 bucket by configuring the [dagster_aws.s3.compute_log_manager](/todo.md) module.

You are also able to stream partial compute log files by configuring the log upload interval (in seconds) using the `upload_interval` parameter.

Note: Dagster Labs will neither have nor use your AWS credentials. The Dagster+ UI will be able to show the URLs linking to the compute log files in your S3 bucket when you set the `show_url_only` parameter to `true`.

<CodeExample filePath="./examples/docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/management/aws_compute_logs.yaml" />
```yaml
# dagster.yaml
instance_class:
module: dagster_cloud.instance
class: DagsterCloudAgentInstance
dagster_cloud_api:
agent_token:
env: DAGSTER_CLOUD_AGENT_TOKEN
deployment: prod
user_code_launcher:
module: dagster_cloud.workspace.docker
class: DockerUserCodeLauncher
compute_logs:
module: dagster_aws.s3.compute_log_manager
class: S3ComputeLogManager
config:
bucket: "mycorp-dagster-compute-logs"
local_dir: "/tmp/cool"
prefix: "dagster-test-"
use_ssl: true
verify: true
verify_cert_path: "/path/to/cert/bundle.pem"
endpoint_url: "http://alternate-s3-host.io"
skip_empty_files: true
upload_interval: 30
upload_extra_args:
ServerSideEncryption: "AES256"
show_url_only: true
region: "us-west-1"
```
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@ unlisted: true
This guide is applicable to Dagster+.
:::

This reference describes the settings that can be configured for full deployments in [Dagster+](/dagster-plus).
{/* /dagster-plus */}
This reference describes the settings that can be configured for full deployments in [Dagster+](/todo.md).

Refer to the [Managing deployments in Dagster+ guide](/dagster-plus/managing-deployments/managing-deployments#configuring-deployment-settings) for info about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI.
{/* /dagster-plus/managing-deployments/managing-deployments#configuring-deployment-settings */}
Refer to the [Managing deployments in Dagster+ guide](/todo.md) for info about configuring settings in the Dagster+ interface or using the `dagster-cloud` CLI.

## Settings schema

Expand Down Expand Up @@ -62,7 +64,7 @@ run_queue:

| Property | Descripton |

Check failure on line 65 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'Descripton' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'Descripton' spelled correctly?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 65, "column": 14}}}, "severity": "ERROR"}

Check failure on line 65 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'Descripton'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'Descripton'?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 65, "column": 14}}}, "severity": "ERROR"}
| --- | --- |
| run_queue.max_concurrent_runs | The maximum number of runs that are allowed to be in progress at once. Set to 0 to stop any runs from launching. Negative values aren't permitted. <ul><li>**Default** - `10` (20 minutes)</li><li>**Maximum** - `500` (Hybrid)[/dagster-plus/deployment/hybrid], `50` (Serverless)[/dagster-plus/deployment/serverless]</li></ul> |
| run_queue.max_concurrent_runs | The maximum number of runs that are allowed to be in progress at once. Set to 0 to stop any runs from launching. Negative values aren't permitted. <ul><li>**Default** - `10` (20 minutes)</li><li>**Maximum** - `500` <a href="/todo.md">(Hybrid)</a>, `50` <a href="/todo.md">(Serverless)</a></li></ul> |

Check warning on line 67 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / deploy

Do not use an `<a>` element to navigate. Use the `<Link />` component from `@docusaurus/Link` instead. See: https://docusaurus.io/docs/docusaurus-core#link

Check warning on line 67 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / deploy

Do not use an `<a>` element to navigate. Use the `<Link />` component from `@docusaurus/Link` instead. See: https://docusaurus.io/docs/docusaurus-core#link
| run_queue.tag_concurrency_limits | A list of limits applied to runs with particular tags. <ul><li>**Defaults** - `[]`</li></ul> Each list item may have the following properties: <ul><li>`key`</li><li>`value`</li><ul><li>If defined, the `limit` is applied only to the `key-value` pair.</li><li>If set to a dict with applyLimitPerUniqueValue: true, the `limit` is applied to the number of unique values for the `key`.</li><li>If set to a dict with `applyLimitPerUniqueValue: true`, the limit is applied to the number of unique values for the `key`.</li></ul><li>`limit`</li></ul> |

### Run monitoring (run_monitoring)

Check failure on line 70 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'run_monitoring'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'run_monitoring'?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 70, "column": 21}}}, "severity": "ERROR"}

Check failure on line 70 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'run_monitoring' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'run_monitoring' spelled correctly?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 70, "column": 21}}}, "severity": "ERROR"}
Expand Down Expand Up @@ -98,7 +100,8 @@ run_retries:

### SSO default role

The `sso_default_role` setting lets you configure the default role on the deployment which is granted to new users that log in via SSO. For more information on available roles, see the [Dagster+ permissions reference](/dagster-plus/account/managing-users/managing-user-roles-permissions#user-permissions-reference).
{/* dagster-plus/account/managing-users/managing-user-roles-permissions#user-permissions-reference */}
The `sso_default_role` setting lets you configure the default role on the deployment which is granted to new users that log in via SSO. For more information on available roles, see the [Dagster+ permissions reference](/todo.md).

Check warning on line 104 in docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.latin] Use 'with', 'through', or 'by using' instead of 'via', but consider rewriting the sentence. Raw Output: {"message": "[Dagster.latin] Use 'with', 'through', or 'by using' instead of 'via', but consider rewriting the sentence.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/settings/deployment-settings.md", "range": {"start": {"line": 104, "column": 128}}}, "severity": "WARNING"}

```yaml
sso_default_role: EDITOR
Expand All @@ -110,7 +113,8 @@ sso_default_role: EDITOR

### Non-isolated runs

Configure [non-isolated runs](/dagster-plus/deployment/serverless) on your deployment.
{/* /dagster-plus/deployment/serverless */}
Configure [non-isolated runs](/todo.md) on your deployment.

```yaml
non_isolated_runs:
Expand Down

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

0 comments on commit a964fdd

Please sign in to comment.