-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[docathon] Configuring assets guide (#23924)
## Summary & Motivation First pass at a short-but-sweet asset config guide. ## How I Tested These Changes ## Changelog [New | Bug | Docs] NOCHANGELOG --------- Co-authored-by: Sandy Ryza <[email protected]> Co-authored-by: Pedram Navid <[email protected]>
- Loading branch information
1 parent
5713432
commit 0cd2bfd
Showing
5 changed files
with
79 additions
and
4 deletions.
There are no files selected for viewing
44 changes: 41 additions & 3 deletions
44
docs/docs-beta/docs/guides/data-modeling/configuring-assets.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,43 @@ | ||
--- | ||
title: Configuring assets and ops | ||
sidebar_label: Configuring assets | ||
title: Configuring assets via the UI | ||
sidebar_label: Configuring assets via the UI | ||
sidebar_position: 50 | ||
--- | ||
--- | ||
|
||
You will commonly want to manually materialize assets using the Dagster UI to backfill historical data, debug a production issue, or some other one-off task. | ||
|
||
Often, you will want to be able to tweak some parameters when materializing these assets. This can be accomplished through the asset configuration system. | ||
|
||
## What you'll learn | ||
|
||
- How to add configuration to your assets | ||
- How to modify the configuration when launching a run | ||
- When to use asset configuration vs. [resources](/docs/concepts/resources) | ||
|
||
--- | ||
|
||
<details> | ||
<summary>Prerequisites</summary> | ||
|
||
To follow the steps in this guide, you'll need: | ||
|
||
- A basic understanding of Dagster and assets. See the [Quick Start](/tutorial/quick-start) tutorial for an overview. | ||
- Familiarity with [Pydantic](https://docs.pydantic.dev/latest/) | ||
- An understanding of [Ops vs Assets](/concepts/ops-jobs/ops-vs-assets) | ||
</details> | ||
|
||
--- | ||
|
||
## Adding configuration to assets | ||
|
||
For an asset to be configurable, you must first define a schema that inherits from the Dagster `Config` class. For example, let's say we want to allow users to change the lookback time window for an asset: | ||
|
||
<CodeExample filePath="guides/data-modeling/configuring-assets/config-schema.py" language="python" title="Adding configuration" /> | ||
|
||
## Modifying the configuration when launching a run | ||
|
||
When launching a run using Dagster's Launchpad, you can provide a run config file as YAML or JSON that overrides the default configuration for your asset: | ||
|
||
<CodeExample filePath="guides/data-modeling/configuring-assets/run_config.yaml" language="yaml" title="Run config provided via UI" /> | ||
|
||
Run configurations reference an `op` which is the underlying compute associated with an asset. See [the Ops vs Assets](/concepts/ops-jobs/ops-vs-assets) documentation for more information. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -38,3 +38,5 @@ Twilio | |
|
||
We have | ||
we have | ||
lookback | ||
Pydantic |
29 changes: 29 additions & 0 deletions
29
...beta_snippets/docs_beta_snippets/guides/data-modeling/configuring-assets/config-schema.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
import dagster as dg | ||
|
||
|
||
class ForecastModelConfig(dg.Config): | ||
# lookback_window_days defaults to 30, but can be | ||
# overridden by the user. If you do not provide a | ||
# default, the user will need to provide a value. | ||
lookback_window_days: int = 30 | ||
|
||
|
||
@dg.asset | ||
def forecast_model(config: ForecastModelConfig): | ||
print("Forecasting over time window:", config.lookback_window_days) # noqa: T201 | ||
# ...more code here | ||
|
||
|
||
defs = dg.Definitions(assets=[forecast_model]) | ||
|
||
if __name__ == "__main__": | ||
from pathlib import Path | ||
|
||
import yaml | ||
|
||
dg.materialize( | ||
[forecast_model], | ||
run_config=yaml.safe_load( | ||
(Path(__file__).absolute().parent / "run_config.yaml").open() | ||
), | ||
) |
4 changes: 4 additions & 0 deletions
4
..._beta_snippets/docs_beta_snippets/guides/data-modeling/configuring-assets/run_config.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
ops: | ||
forecast_model: | ||
config: | ||
lookback_window_days: 7 |
0cd2bfd
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Deploy preview for dagster-docs ready!
✅ Preview
https://dagster-docs-kzkkji7wu-elementl.vercel.app
https://master.dagster.dagster-docs.io
Built with commit 0cd2bfd.
This pull request is being automatically deployed with vercel-action