Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docathon] Configuring assets guide #23924

Merged
merged 6 commits into from
Aug 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 41 additions & 3 deletions docs/docs-beta/docs/guides/data-modeling/configuring-assets.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,43 @@
---
title: Configuring assets and ops
sidebar_label: Configuring assets
title: Configuring assets via the UI
sidebar_label: Configuring assets via the UI
sidebar_position: 50
---
---

You will commonly want to manually materialize assets using the Dagster UI to backfill historical data, debug a production issue, or some other one-off task.

Often, you will want to be able to tweak some parameters when materializing these assets. This can be accomplished through the asset configuration system.

## What you'll learn

cmpadden marked this conversation as resolved.
Show resolved Hide resolved
- How to add configuration to your assets
- How to modify the configuration when launching a run
- When to use asset configuration vs. [resources](/docs/concepts/resources)

---

<details>
<summary>Prerequisites</summary>

To follow the steps in this guide, you'll need:

petehunt marked this conversation as resolved.
Show resolved Hide resolved
- A basic understanding of Dagster and assets. See the [Quick Start](/tutorial/quick-start) tutorial for an overview.
- Familiarity with [Pydantic](https://docs.pydantic.dev/latest/)
- An understanding of [Ops vs Assets](/concepts/ops-jobs/ops-vs-assets)
</details>

---

## Adding configuration to assets

For an asset to be configurable, you must first define a schema that inherits from the Dagster `Config` class. For example, let's say we want to allow users to change the lookback time window for an asset:

<CodeExample filePath="guides/data-modeling/configuring-assets/config-schema.py" language="python" title="Adding configuration" />

## Modifying the configuration when launching a run

petehunt marked this conversation as resolved.
Show resolved Hide resolved
When launching a run using Dagster's Launchpad, you can provide a run config file as YAML or JSON that overrides the default configuration for your asset:

<CodeExample filePath="guides/data-modeling/configuring-assets/run_config.yaml" language="yaml" title="Run config provided via UI" />

Run configurations reference an `op` which is the underlying compute associated with an asset. See [the Ops vs Assets](/concepts/ops-jobs/ops-vs-assets) documentation for more information.
4 changes: 3 additions & 1 deletion docs/docs-beta/src/components/CodeExample.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ const CodeExample: React.FC<CodeExampleProps> = ({filePath, language, title}) =>
// Adjust the import path to start from the docs directory
import(`!!raw-loader!/../../examples/docs_beta_snippets/docs_beta_snippets/${filePath}`)
.then((module) => {
const lines = module.default.split('\n');
const lines = module.default.split('\n').map(line => {
return line.replaceAll(/#.*?noqa.*?$/g, "").trim();
});
const mainIndex = lines.findIndex((line) => line.trim().startsWith('if __name__ == '));
const strippedContent =
mainIndex !== -1 ? lines.slice(0, mainIndex).join('\n') : module.default;
Expand Down
2 changes: 2 additions & 0 deletions docs/vale/styles/config/vocabularies/Dagster/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,5 @@ Twilio

We have
we have
lookback
Pydantic
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import dagster as dg


class ForecastModelConfig(dg.Config):
# lookback_window_days defaults to 30, but can be
# overridden by the user. If you do not provide a
# default, the user will need to provide a value.
lookback_window_days: int = 30


@dg.asset
def forecast_model(config: ForecastModelConfig):
print("Forecasting over time window:", config.lookback_window_days) # noqa: T201
# ...more code here


defs = dg.Definitions(assets=[forecast_model])

if __name__ == "__main__":
from pathlib import Path

import yaml

dg.materialize(
[forecast_model],
run_config=yaml.safe_load(
(Path(__file__).absolute().parent / "run_config.yaml").open()
),
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
ops:
forecast_model:
config:
lookback_window_days: 7
Loading