Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs-beta] migrate - managing compute logs and error messages #26500

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,106 @@
unlisted: true
---

{/* TODO move from https://docs.dagster.io/dagster-plus/managing-deployments/controlling-logs */}
import ThemedImage from '@theme/ThemedImage';

:::note
This guide is applicable to Dagster+.
:::

In this guide, we'll cover how to adjust where Dagster+ compute logs are stored and manage masking of error messages in the Dagster+ UI.

{/* By default, Dagster+ ingests [structured event logs and compute logs](/concepts/logging#log-types) from runs and surfaces error messages from [code locations](/guides/deploy/code-locations/) in the UI. */}
By default, Dagster+ ingests [structured event logs and compute logs](/todo) from runs and surfaces error messages from [code locations](/todo) in the UI.

Depending on your organization's needs, you may want to retain these logs in your own infrastructure or mask error message contents.

---

## Modifying compute log storage

Dagster's compute logs are handled by the configured [`ComputeLogManager`](/api/python-api/internals#compute-log-manager). By default, Dagster+ utilizes the `CloudComputeLogManager` which stores logs in a Dagster+-managed Amazon S3 bucket, but you can customize this behavior to store logs in a destination of your choice.

### Writing to your own S3 bucket

If using the Kubernetes agent, you can instead forward logs to your own S3 bucket by using the [`S3ComputeLogManager`](/api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager).

{/* You can configure the `S3ComputeLogManager` in your [`dagster.yaml` file](/dagster-plus/deployment/agents/customizing-configuration): */}
You can configure the `S3ComputeLogManager` in your [`dagster.yaml` file](/todo):

```yaml
cmpadden marked this conversation as resolved.
Show resolved Hide resolved
compute_logs:
module: dagster_aws.s3.compute_log_manager
class: S3ComputeLogManager
config:
show_url_only: true
bucket: your-compute-log-storage-bucket
region: your-bucket-region
```

If you are using Helm to deploy the Kubernetes agent, you can provide the following configuration in your `values.yaml` file:

```yaml
computeLogs:
enabled: true
custom:
module: dagster_aws.s3.compute_log_manager
class: S3ComputeLogManager
config:
show_url_only: true
bucket: your-compute-log-storage-bucket
region: your-bucket-region
```

### Disabling compute log upload

If your organization has its own logging solution which ingests `stdout` and `stderr` from your compute environment, you may want to disable compute log upload entirely. You can do this with the <PyObject module="dagster._core.storage.noop_compute_log_manager" object="NoOpComputeLogManager" />.

{/* You can configure the `NoOpComputeLogManager` in your [`dagster.yaml` file](/dagster-plus/deployment/agents/customizing-configuration): */}
You can configure the `NoOpComputeLogManager` in your [`dagster.yaml` file](/todo):

```yaml
compute_logs:
module: dagster.core.storage.noop_compute_log_manager
class: NoOpComputeLogManager
```

If you are using Helm to deploy the Kubernetes agent, use the `enabled` flag to disable compute log upload:

```yaml
computeLogs:
enabled: false
```

### Other compute log storage options

{/* For a full list of available compute log storage options, refer to the [Compute log storage docs](/deployment/dagster-instance#compute-log-storage). */}
For a full list of available compute log storage options, refer to the [Compute log storage docs](/todo).

---

## Masking error messages

By default, Dagster+ surfaces error messages from your code locations in the UI, including when runs fail, sensors or schedules throw an exception, or code locations fail to load. You can mask these error messages in the case that their contents are sensitive.

To mask error messages in a Dagster+ Deployment, set the environment variable `DAGSTER_REDACT_USER_CODE_ERRORS` equal to `1` using the [**Environment variables** page](/dagster-plus/deployment/management/environment-variables/) in the UI:

<ThemedImage
alt="Environment variable UI showing DAGSTER_REDACT_USER_CODE_ERRORS set to 1"
style={{width:'100%', height: 'auto'}}
sources={{
light: '/images/dagster-cloud/logs-error-messages/configure-redact-env-var.png',
dark: '/images/dagster-cloud/logs-error-messages/configure-redact-env-var.png',
}}
/>

Once set, error messages from your code locations will be masked in the UI. A unique error ID will be generated, which you can use to look up the error message in your own logs. This error ID will appear in place of the error message in UI dialogs or in a run's event logs.

Check failure on line 98 in docs/docs-beta/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'dialogs' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'dialogs' spelled correctly?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md", "range": {"start": {"line": 98, "column": 241}}}, "severity": "ERROR"}

Check failure on line 98 in docs/docs-beta/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'dialogs'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'dialogs'?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md", "range": {"start": {"line": 98, "column": 241}}}, "severity": "ERROR"}


<ThemedImage
alt="Error message in Dagster event logs showing unique error ID"
style={{width:'100%', height: 'auto'}}
sources={{
light: '/images/dagster-cloud/logs-error-messages/masked-err-message.png',
dark: '/images/dagster-cloud/logs-error-messages/masked-err-message.png',
}}
/>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading