Skip to content

Commit

Permalink
[docs-beta] migrate - managing compute logs and error messages
Browse files Browse the repository at this point in the history
  • Loading branch information
cmpadden committed Dec 14, 2024
1 parent 3252fda commit 6d3c51e
Show file tree
Hide file tree
Showing 3 changed files with 99 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,102 @@ title: Managing compute logs and error messages
unlisted: true
---

{/* TODO move from https://docs.dagster.io/dagster-plus/managing-deployments/controlling-logs */}
import ThemedImage from '@theme/ThemedImage';

:::note
This guide is applicable to Dagster+.
:::

In this guide, we'll cover how to adjust where Dagster+ compute logs are stored and manage masking of error messages in the Dagster+ UI.

By default, Dagster+ ingests [structured event logs and compute logs](/concepts/logging#log-types) from runs and surfaces error messages from [code locations](/guides/deploy/code-locations/) in the UI.

Depending on your organization's needs, you may want to retain these logs in your own infrastructure or mask error message contents.

---

## Modifying compute log storage

Dagster's compute logs are handled by the configured [`ComputeLogManager`](/api/python-api/internals#compute-log-manager). By default, Dagster+ utilizes the `CloudComputeLogManager` which stores logs in a Dagster+-managed Amazon S3 bucket, but you can customize this behavior to store logs in a destination of your choice.

### Writing to your own S3 bucket

If using the Kubernetes agent, you can instead forward logs to your own S3 bucket by using the [`S3ComputeLogManager`](/\_apidocs/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager).

You can configure the `S3ComputeLogManager` in your [`dagster.yaml` file](/dagster-plus/deployment/agents/customizing-configuration):

```yaml
compute_logs:
module: dagster_aws.s3.compute_log_manager
class: S3ComputeLogManager
config:
show_url_only: true
bucket: your-compute-log-storage-bucket
region: your-bucket-region
```
If you are using Helm to deploy the Kubernetes agent, you can provide the following configuration in your `values.yaml` file:

```yaml
computeLogs:
enabled: true
custom:
module: dagster_aws.s3.compute_log_manager
class: S3ComputeLogManager
config:
show_url_only: true
bucket: your-compute-log-storage-bucket
region: your-bucket-region
```

### Disabling compute log upload

If your organization has its own logging solution which ingests `stdout` and `stderr` from your compute environment, you may want to disable compute log upload entirely. You can do this with the <PyObject module="dagster._core.storage.noop_compute_log_manager" object="NoOpComputeLogManager" />.

You can configure the `NoOpComputeLogManager` in your [`dagster.yaml` file](/dagster-plus/deployment/agents/customizing-configuration):

```yaml
compute_logs:
module: dagster.core.storage.noop_compute_log_manager
class: NoOpComputeLogManager
```

If you are using Helm to deploy the Kubernetes agent, use the `enabled` flag to disable compute log upload:

```yaml
computeLogs:
enabled: false
```

### Other compute log storage options

For a full list of available compute log storage options, refer to the [Compute log storage docs](/deployment/dagster-instance#compute-log-storage).

---

## Masking error messages

By default, Dagster+ surfaces error messages from your code locations in the UI, including when runs fail, sensors or schedules throw an exception, or code locations fail to load. You can mask these error messages in the case that their contents are sensitive.

To mask error messages in a Dagster+ Deployment, set the environment variable `DAGSTER_REDACT_USER_CODE_ERRORS` equal to `1` using the [**Environment variables** page](/dagster-plus/deployment/management/environment-variables/) in the UI:

<ThemedImage
alt="Environment variable UI showing DAGSTER_REDACT_USER_CODE_ERRORS set to 1"
style={{width:'100%', height: 'auto'}}
sources={{
light: '/images/dagster-cloud/logs-error-messages/configure-redact-env-var.png',
dark: '/images/dagster-cloud/logs-error-messages/configure-redact-env-var.png',
}}
/>

Once set, error messages from your code locations will be masked in the UI. A unique error ID will be generated, which you can use to look up the error message in your own logs. This error ID will appear in place of the error message in UI dialogs or in a run's event logs.

Check failure on line 94 in docs/docs-beta/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'dialogs'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'dialogs'?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md", "range": {"start": {"line": 94, "column": 241}}}, "severity": "ERROR"}

Check failure on line 94 in docs/docs-beta/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'dialogs' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'dialogs' spelled correctly?", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md", "range": {"start": {"line": 94, "column": 241}}}, "severity": "ERROR"}


<ThemedImage
alt="Error message in Dagster event logs showing unique error ID"
style={{width:'100%', height: 'auto'}}
sources={{
light: '/images/dagster-cloud/logs-error-messages/masked-err-message.png',
dark: '/images/dagster-cloud/logs-error-messages/masked-err-message.png',
}}
/>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 6d3c51e

Please sign in to comment.