Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docathon] Clean up metadata docs (DOC-382) #23915

Merged
merged 4 commits into from
Aug 27, 2024
Merged

[docathon] Clean up metadata docs (DOC-382) #23915

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
60fe0dc
Clean up metadata docs
petehunt Aug 26, 2024
91ab318
one->you
petehunt Aug 26, 2024
9be2735
Finish feedback from erin
petehunt Aug 26, 2024
efca868
Fix broken link and other vale warnings
petehunt Aug 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 55 additions & 35 deletions docs/docs-beta/docs/guides/data-modeling/metadata.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
title: "Adding tags and metadata to assets"
description: "Learn how to add tags and metadata to assets to improve observability in Dagster"
title: 'Adding tags and metadata to assets'
description: 'Learn how to add tags and metadata to assets to improve observability in Dagster'
sidebar_position: 40
sidebar_label: "Enriching assets with metadata"
sidebar_label: 'Enriching assets with metadata'
---

Assets feature prominently in the Dagster UI. It is often helpful to attach information to assets to understand where they are stored, what they contain, and how they should be organized.
Assets feature prominently in the Dagster UI. It's often helpful to attach information to assets to understand where they're stored, what they contain, and how they should be organized.

In Dagster, you can attach ownership information, organize your assets with tags, attach rich, complex information with metadata, and link your assets with their source code.

Expand All @@ -30,30 +30,34 @@

## Adding owners to your assets

In a large organization, it's important to know who is responsible for a given data asset. With `owners` it's straightforward to add individuals and teams as owners for your asset:
In a large organization, it's important to know which individuals and teams are responsible for a given data asset:

<CodeExample filePath="guides/data-modeling/metadata/owners.py" language="python" title="Using owners" />

`owners` must either be an email address, or a team name prefixed by `team:`.

> With Dagster+ Pro, you can create asset-based alerts that will automatically notify an asset's owners when triggered. Refer to the [Dagster+ alert documentation](/dagster-plus/deployment/alerts) for more information.
:::tip

With Dagster+ Pro, you can create asset-based alerts that will automatically notify an asset's owners when triggered. Refer to the [Dagster+ alert documentation](/dagster-plus/deployment/alerts) for more information.

:::

## Choosing between tags or metadata for custom information

In Dagster, you can attach custom information to assets in two ways: **tags** and **metadata**.

**Tags** are a simple way to organize assets in Dagster. You can attach several tags to an asset when it is defined, and they will appear in the UI. You can also use tags to search and filter for assets in the [asset catalog](/todo). They are structured as key-value pairs of strings.
**Tags** are the primary way to organize assets in Dagster. You can attach several tags to an asset when it's defined, and they will appear in the UI. You can also use tags to search and filter for assets in the [Asset catalog](/todo). They're structured as key-value pairs of strings.
erinkcochran87 marked this conversation as resolved.
Show resolved Hide resolved

Here's an example of some tags you might apply to an asset:

Here's an example of some tags one might apply to an asset:
```python
{"domain": "marketing", "pii": "true"}
```

**Metadata** allows you to attach rich information to the asset, like a Markdown description, a table schema, or a time series. Metadata is more flexible than tags, as it can store more complex information. Metadata can be attached to an asset at definition time (i.e. when the code is first imported) or at runtime (every time an asset is materialized).
**Metadata** allows you to attach rich information to the asset, like a Markdown description, a table schema, or a time series. Metadata is more flexible than tags, as it can store more complex information. Metadata can be attached to an asset at definition time (that is, when the code is first imported) or at runtime (every time an asset is materialized).

Check warning on line 57 in docs/docs-beta/docs/guides/data-modeling/metadata.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Dagster.contractions] Use 'that's' instead of 'that is'.

Raw Output:
{"message": "[Dagster.contractions] Use 'that's' instead of 'that is'.", "location": {"path": "docs/docs-beta/docs/guides/data-modeling/metadata.md", "range": {"start": {"line": 57, "column": 265}}}, "severity": "INFO"}

Here's an example of some metadata you might apply to an asset:

Here's an example of some metadata one might apply to an asset:
```python
{
"link_to_docs": MetadataValue.url("https://..."),
Expand All @@ -70,10 +74,9 @@

Keep in mind that tags must contain only strings as keys and values. Additionally, the Dagster UI will render tags with the empty string as a "label" rather than a key-value pair.


## Attaching metadata to an asset at definition time

Attaching metadata at definition time is quite similar to how you attach tags.
Attaching metadata at definition time is similar to how you attach tags.

<CodeExample filePath="guides/data-modeling/metadata/definition-metadata.py" language="python" title="Using metadata at definition time" />

Expand All @@ -83,7 +86,7 @@

## Attaching metadata to an asset at runtime

Metadata becomes very powerful when it is attached when an asset is materialized. This allows you to update metadata when information about an asset changes and track historical metadata such as execution time and row counts as a time series.
Metadata becomes powerful when it's attached when an asset is materialized. This allows you to update metadata when information about an asset changes and track historical metadata such as execution time and row counts as a time series.

<CodeExample filePath="guides/data-modeling/metadata/runtime-metadata.py" language="python" title="Using metadata at runtime" />

Expand All @@ -93,43 +96,53 @@

Some metadata keys will be given special treatment in the Dagster UI.

| Key | Description |
|------------------------------|---------------------------------------------------------------------------------------------------------------|
| `dagster/uri` | **Type:** `str` <br/><br/> The URI for the asset, e.g. "s3://my_bucket/my_object" |
| `dagster/column_schema` | **Type:** [`TableSchema`](/todo) <br/><br/> For an asset that's a table, the schema of the columns in the table. Refer to the [Table and column metadata](#table-and-column-metadata) secton for details. |
| `dagster/column_lineage` | **Type:** [`TableColumnLineage`](/todo) <br/><br/> For an asset that's a table, the lineage of column inputs to column outputs for the table. Refer to the [Table and column metadata](#table-and-column-metadata) secton for details. |
| `dagster/row_count` | **Type:** `int` <br/><br/> For an asset that's a table, the number of rows in the table. Refer to the Table metadata documentation for details. |
| `dagster/partition_row_count` | **Type:** `int` <br/><br/> For a partition of an asset that's a table, the number of rows in the partition. |
| `dagster/relation_identifier` | **Type:** `str` <br/><br/> A unique identifier for the table/view, typically fully qualified. For example, my_database.my_schema.my_table |
| `dagster/code_references` | **Type:** [`CodeReferencesMetadataValue`](/todo) <br/><br/> A list of code references for the asset, such as file locations or references to Github URLs. Refer to the [Linking your assets with their source code](#linking-your-assets-with-their-source-code) section for details. Should only be provided in definition-level metadata, not materialization metadata. |

| Key | Description |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dagster/uri` | **Type:** `str` <br/><br/> The URI for the asset, e.g. "s3://my_bucket/my_object" |

Check warning on line 101 in docs/docs-beta/docs/guides/data-modeling/metadata.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Dagster.latin] Use 'for example' instead of 'e.g.', but consider rewriting the sentence.

Raw Output:
{"message": "[Dagster.latin] Use 'for example' instead of 'e.g.', but consider rewriting the sentence.", "location": {"path": "docs/docs-beta/docs/guides/data-modeling/metadata.md", "range": {"start": {"line": 101, "column": 85}}}, "severity": "WARNING"}

Check failure on line 101 in docs/docs-beta/docs/guides/data-modeling/metadata.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Terms] Use 'S3' instead of 's3'.

Raw Output:
{"message": "[Vale.Terms] Use 'S3' instead of 's3'.", "location": {"path": "docs/docs-beta/docs/guides/data-modeling/metadata.md", "range": {"start": {"line": 101, "column": 91}}}, "severity": "ERROR"}
| `dagster/column_schema` | **Type:** [`TableSchema`](/todo) <br/><br/> For an asset that's a table, the schema of the columns in the table. Refer to the [Table and column metadata](#table-and-column-metadata) section for details. |
| `dagster/column_lineage` | **Type:** [`TableColumnLineage`](/todo) <br/><br/> For an asset that's a table, the lineage of column inputs to column outputs for the table. Refer to the [Table and column metadata](#table-and-column-metadata) section for details. |
| `dagster/row_count` | **Type:** `int` <br/><br/> For an asset that's a table, the number of rows in the table. Refer to the Table metadata documentation for details. |
| `dagster/partition_row_count` | **Type:** `int` <br/><br/> For a partition of an asset that's a table, the number of rows in the partition. |
| `dagster/relation_identifier` | **Type:** `str` <br/><br/> A unique identifier for the table/view, typically fully qualified. For example, my_database.my_schema.my_table |
| `dagster/code_references` | **Type:** [`CodeReferencesMetadataValue`](/todo) <br/><br/> A list of code references for the asset, such as file locations or references to GitHub URLs. Refer to the [Linking assets with their source code](#linking-assets-with-their-source-code) section for details. Should only be provided in definition-level metadata, not materialization metadata. |

## Table and column metadata

Two of the most powerful metadata types are [`TableSchema`](/todo) and [`TableColumnLineage`](/todo). These metadata types allow stakeholders to view the schema of a table right within Dagster, and, in Dagster+, navigate the [asset catalog](/todo) via the column lineage.
Two of the most powerful metadata types are [`TableSchema`](/todo) and [`TableColumnLineage`](/todo). These metadata types allow stakeholders to view the schema of a table right within Dagster, and, in Dagster+, navigate the [Asset catalog](/todo) via the column lineage.

Check warning on line 111 in docs/docs-beta/docs/guides/data-modeling/metadata.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Dagster.latin] Use 'with', 'through', or 'by using' instead of 'via', but consider rewriting the sentence.

Raw Output:
{"message": "[Dagster.latin] Use 'with', 'through', or 'by using' instead of 'via', but consider rewriting the sentence.", "location": {"path": "docs/docs-beta/docs/guides/data-modeling/metadata.md", "range": {"start": {"line": 111, "column": 249}}}, "severity": "WARNING"}

### Table schema metadata

Here's a quick example of how to attach this metadata at both definition time and runtime:

<CodeExample filePath="guides/data-modeling/metadata/table-schema-metadata.py" language="python" title="Table schema metadata" />

Note that there are several data types and constraints available on [`TableColumn`](/todo) objects. Refer to the API documentation for more information.)
There are several data types and constraints available on [`TableColumn`](/todo) objects. Refer to the API documentation for more information.)

### Column lineage metadata

> Many integrations such as [dbt](https://docs.dagster.io/integrations/dbt/reference) automatically attach this metadata out-of-the-box.
:::tip

Many integrations such as [dbt](https://docs.dagster.io/integrations/dbt/reference) automatically attach this metadata out-of-the-box.

:::

Column lineage metadata is a powerful way to track how columns in a table are derived from other columns. Here is how you can manually attach this metadata:

<CodeExample filePath="guides/data-modeling/metadata/table-column-lineage-metadata.py" language="python" title="Table column lineage metadata" />

> Dagster+ provides rich visualization and navigation of column lineage in the asset catalog. Refer to the [Dagster+ documentation](/dagster-plus) for more information.
:::tip

Dagster+ provides rich visualization and navigation of column lineage in the Asset catalog. Refer to the [Dagster+ documentation](/dagster-plus) for more information.

## Linking your assets with their source code
:::

> This feature is considered experimental and is under active development. This guide will be updated as we roll out new features.
## Linking assets with their source code

:::warning

This feature is considered experimental and is under active development. This guide will be updated as we roll out new features.

:::

Attaching code reference metadata to your Dagster asset definitions allows you to easily view those assets' source code from the Dagster UI both in local development and in production.

Expand All @@ -141,22 +154,29 @@

<CodeExample filePath="guides/data-modeling/metadata/python-local-references.py" language="python" title="Local source code references" />

### Attaching custom code references for local development
### Customizing code references

You can manually add the `dagster/code_references` metadata to your asset definitions to attach custom code references. This is useful when your asset's source code is not primarily in Python (e.g. when parsing a YAML config or integrating with another language).
If you want to customize how code references are attached - such as when you are building [domain-specific languages with asset factories](/guides/data-modeling/asset-factories) - you can manually add the `dagster/code_references` metadata to your asset definitions.

<CodeExample filePath="guides/data-modeling/metadata/custom-local-references.py" language="python" title="Custom local source code references" />

### Attaching code references in production (Dagster+)
### Attaching code references in production

Dagster+ can automatically annotate your assets with code references to source control such as GitHub or GitLab.
<Tabs>
<TabItem value="dagster-plus" label="Dagster+">

Dagster+ can automatically annotate your assets with code references to source control such as GitHub or Gitlab.

<CodeExample filePath="guides/data-modeling/metadata/plus-references.py" language="python" title="Production source code references (Dagster+)" />

### Attaching code references in production (OSS)
</TabItem>
<TabItem value="dagster-open-source" label="OSS">

If you aren't using Dagster+, you can annotate your assets with code references to source control, but it requires a little more manual mapping.

<CodeExample filePath="guides/data-modeling/metadata/oss-references.py" language="python" title="Production source code references (OSS)" />

[`link_code_references_to_git`](/todo) currently supports GitHub and GitLab repositories. It also supports customization of how file paths are mapped; see the [`AnchorBasedFilePathMapping`](/todo) API docs for more information.
[`link_code_references_to_git`](/todo) currently supports GitHub and Gitlab repositories. It also supports customization of how file paths are mapped; see the [`AnchorBasedFilePathMapping`](/todo) API docs for more information.

</TabItem>
</Tabs>
Loading