Skip to content

Commit

Permalink
use note for prereq
Browse files Browse the repository at this point in the history
  • Loading branch information
cmpadden committed Dec 20, 2024
1 parent d6ca7c9 commit c28b0c5
Showing 1 changed file with 3 additions and 15 deletions.
18 changes: 3 additions & 15 deletions docs/docs-beta/docs/tutorial/multi-asset-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,15 @@ title: Creating a multi-asset integration
description: Create a decorator based multi-asset integration
---


When working in the Dagster ecosystem, you may have noticed that decorators are frequently used. For example, assets, jobs, and ops use decorators. If you have a service that produces many assets, it's possible to define it as a multi-asset decorator-offering a consistent and intuitive developer experience to existing Dagster APIs.

In the context of Dagster, decorators are helpful because they often wrap some form of processing. For example, when writing an asset, you define your processing code and then annotate the function with the `asset` decorator /> decorator. Then, the internal Dagster code can register the asset, assign metadata, pass in context data, or perform any other variety of operations that are required to integrate your asset code with the Dagster platform.

In this guide, you'll learn how to develop a multi-asset integration for a hypothetical replication tool.


## Prerequisites

To follow the steps in this guide, you'll need:

- Familiarity with Dagster
- An understanding of Python decorators—[Real Python's Primer on Python Decorators](https://realpython.com/primer-on-python-decorators/) is a fantastic introduction

---
:::note
This guide assumes basic familiarity with Dagster and Python decorators.
:::

## Step 1: Input

Expand Down Expand Up @@ -60,8 +53,6 @@ def replicate(replication_configuration_yaml: Path) -> Iterator[Mapping[str, Any
yield {"table": table.get("name"), "status": "success"}
```

---

## Step 2: Implementation

First, let's define a `Project` object that takes in the path of our configuration YAML file. This will allow us to encapsulate the logic that gets metadata and table information from our project configuration.
Expand Down Expand Up @@ -141,7 +132,6 @@ There are a few limitations to this approach:

For the first limitation, we can resolve this by refactoring the code in the body of our asset function into a Dagster resource.


## Step 3: Moving the replication logic into a resource

Refactoring the replication logic into a resource enables us to support better configuration and re-use of our logic.
Expand Down Expand Up @@ -179,7 +169,6 @@ def my_assets(replication_resource: ReplicationProject):
replication_resource.run(replication_project)
```


## Step 4: Using translators

At the end of [Step 2](#step-2-implementation), we mentioned that end users were unable to customize asset attributes, like the asset key, generated by our decorator. Translator classes are the recommended way of defining this logic, and they provide users with the option to override the default methods used to convert a concept from your tool (for example, a table name) to the corresponding concept in Dagster (for example, asset key).
Expand Down Expand Up @@ -269,7 +258,6 @@ class ReplicationResource(ConfigurableResource):
)
```


## Conclusion

In this guide we walked through how to define a custom multi-asset decorator, a resource for encapsulating tool logic, and a translator for defining the logic to translate a specification to Dagster concepts.
Expand Down

0 comments on commit c28b0c5

Please sign in to comment.