Skip to content

Commit

Permalink
[docathon] Configuring assets guide (#23924)
Browse files Browse the repository at this point in the history
## Summary & Motivation

First pass at a short-but-sweet asset config guide.

## How I Tested These Changes

## Changelog [New | Bug | Docs]

NOCHANGELOG

---------

Co-authored-by: Sandy Ryza <[email protected]>
Co-authored-by: Pedram Navid <[email protected]>
  • Loading branch information
3 people authored Aug 27, 2024
1 parent 5713432 commit 0cd2bfd
Show file tree
Hide file tree
Showing 5 changed files with 79 additions and 4 deletions.
44 changes: 41 additions & 3 deletions docs/docs-beta/docs/guides/data-modeling/configuring-assets.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,43 @@
---
title: Configuring assets and ops
sidebar_label: Configuring assets
title: Configuring assets via the UI
sidebar_label: Configuring assets via the UI
sidebar_position: 50
---
---

You will commonly want to manually materialize assets using the Dagster UI to backfill historical data, debug a production issue, or some other one-off task.

Often, you will want to be able to tweak some parameters when materializing these assets. This can be accomplished through the asset configuration system.

## What you'll learn

- How to add configuration to your assets
- How to modify the configuration when launching a run
- When to use asset configuration vs. [resources](/docs/concepts/resources)

---

<details>
<summary>Prerequisites</summary>

To follow the steps in this guide, you'll need:

- A basic understanding of Dagster and assets. See the [Quick Start](/tutorial/quick-start) tutorial for an overview.
- Familiarity with [Pydantic](https://docs.pydantic.dev/latest/)
- An understanding of [Ops vs Assets](/concepts/ops-jobs/ops-vs-assets)
</details>

---

## Adding configuration to assets

For an asset to be configurable, you must first define a schema that inherits from the Dagster `Config` class. For example, let's say we want to allow users to change the lookback time window for an asset:

<CodeExample filePath="guides/data-modeling/configuring-assets/config-schema.py" language="python" title="Adding configuration" />

## Modifying the configuration when launching a run

When launching a run using Dagster's Launchpad, you can provide a run config file as YAML or JSON that overrides the default configuration for your asset:

<CodeExample filePath="guides/data-modeling/configuring-assets/run_config.yaml" language="yaml" title="Run config provided via UI" />

Run configurations reference an `op` which is the underlying compute associated with an asset. See [the Ops vs Assets](/concepts/ops-jobs/ops-vs-assets) documentation for more information.
4 changes: 3 additions & 1 deletion docs/docs-beta/src/components/CodeExample.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ const CodeExample: React.FC<CodeExampleProps> = ({filePath, language, title}) =>
// Adjust the import path to start from the docs directory
import(`!!raw-loader!/../../examples/docs_beta_snippets/docs_beta_snippets/${filePath}`)
.then((module) => {
const lines = module.default.split('\n');
const lines = module.default.split('\n').map(line => {
return line.replaceAll(/#.*?noqa.*?$/g, "").trim();
});
const mainIndex = lines.findIndex((line) => line.trim().startsWith('if __name__ == '));
const strippedContent =
mainIndex !== -1 ? lines.slice(0, mainIndex).join('\n') : module.default;
Expand Down
2 changes: 2 additions & 0 deletions docs/vale/styles/config/vocabularies/Dagster/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,5 @@ Twilio

We have
we have
lookback
Pydantic
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import dagster as dg


class ForecastModelConfig(dg.Config):
# lookback_window_days defaults to 30, but can be
# overridden by the user. If you do not provide a
# default, the user will need to provide a value.
lookback_window_days: int = 30


@dg.asset
def forecast_model(config: ForecastModelConfig):
print("Forecasting over time window:", config.lookback_window_days) # noqa: T201
# ...more code here


defs = dg.Definitions(assets=[forecast_model])

if __name__ == "__main__":
from pathlib import Path

import yaml

dg.materialize(
[forecast_model],
run_config=yaml.safe_load(
(Path(__file__).absolute().parent / "run_config.yaml").open()
),
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
ops:
forecast_model:
config:
lookback_window_days: 7

1 comment on commit 0cd2bfd

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-kzkkji7wu-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 0cd2bfd.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.