[docs-revamp] Metadata docs

docs/docs-beta/.gitignore

@@ -21,7 +21,7 @@ yarn-error.log*
 *.sqlite
 
 .pnp.*
-.yarn/*
+.yarn
 !.yarn/patches
 !.yarn/plugins
 !.yarn/releases

docs/docs-beta/README.md

@@ -25,6 +25,12 @@ Install Vale with:
 brew install vale
 ```
 
+or
+
+```bash
+pip install vale
+```
+
 ---
 
 ## Overview of the docs

docs/docs-beta/docs/guides/data-modeling/metadata.md

@@ -1,6 +1,162 @@
 ---
-title: "Adding metadata and tags to assets"
-description: "Learn how to add metadata and tags to assets to improve observability in Dagster"
+title: "Adding tags and metadata to assets"
+description: "Learn how to add tags and metadata to assets to improve observability in Dagster"
 sidebar_position: 40
 sidebar_label: "Enriching assets with metadata"
 ---
+
+Assets feature prominently in the Dagster UI. It is often helpful to attach information to assets to understand where they are stored, what they contain, and how they should be organized.
+
+In Dagster, you can attach ownership information, organize your assets with tags, attach rich, complex information with metadata, and link your assets with their source code.
+
+## What you'll learn
+
+- How to add owners to your assets
+- How to use tags to organize assets
+- How to attach complex information to an asset using metadata
+- How to link your assets with their source code
+
+---
+
+<details>
+  <summary>Prerequisites</summary>
+
+To follow the steps in this guide, you'll need:
+
+- A basic understanding of Dagster and assets. See the [Quick Start](/tutorial/quick-start) tutorial for an overview.
+</details>
+
+---
+
+## Adding owners to your assets
+
+In a large organization, it's important to know who is responsible for a given data asset. With `owners` it's straightforward to add individuals and teams as owners for your asset:
+
+<CodeExample filePath="guides/data-modeling/metadata/owners.py" language="python" title="Using owners" />
+
+`owners` must either be an email address, or a team name prefixed by `team:`.
+
+> With Dagster+ Pro, you can create asset-based alerts that will automatically notify an asset's owners when triggered. Refer to the [Dagster+ alert documentation](/dagster-plus/deployment/alerts) for more information.
+
+
+
+## Choosing between tags or metadata for custom information
+
+In Dagster, you can attach custom information to assets in two ways: **tags** and **metadata**.
+
+**Tags** are a simple way to organize assets in Dagster. You can attach several tags to an asset when it is defined, and they will appear in the UI. You can also use tags to search and filter for assets in the [asset catalog](/todo). They are structured as key-value pairs of strings.
+
+Here's an example of some tags one might apply to an asset:
+```python
+{"domain": "marketing", "pii": "true"}
+```
+
+**Metadata** allows you to attach rich information to the asset, like a Markdown description, a table schema, or a time series. Metadata is more flexible than tags, as it can store more complex information. Metadata can be attached to an asset at definition time (i.e. when the code is first imported) or at runtime (every time an asset is materialized).
+
+Here's an example of some metadata one might apply to an asset:
+```python
+{
+    "link_to_docs": MetadataValue.url("https://..."),
+    "snippet": MetadataValue.md("# Embedded markdown\n..."),
+    "file_size_kb": MetadataValue.int(1024),
+}
+```
+
+## Attaching tags to an asset
+
+Like `owners`, just pass a dictionary of tags to the `tags` argument when defining an asset:
+
+<CodeExample filePath="guides/data-modeling/metadata/tags.py" language="python" title="Using tags" />
+
+Keep in mind that tags must contain only strings as keys and values. Additionally, the Dagster UI will render tags with the empty string as a "label" rather than a key-value pair.
+
+
+## Attaching metadata to an asset at definition time
+
+Attaching metadata at definition time is quite similar to how you attach tags.
+
+<CodeExample filePath="guides/data-modeling/metadata/definition-metadata.py" language="python" title="Using metadata at definition time" />
+
+To learn more about the different types of metadata you can attach, see the [`MetadataValue`](/todo) API docs.
+
+Some metadata keys will be given special treatment in the Dagster UI. See the [Standard metadata types](#standard-metadata-types) section for more information.
+
+## Attaching metadata to an asset at runtime
+
+Metadata becomes very powerful when it is attached when an asset is materialized. This allows you to update metadata when information about an asset changes and track historical metadata such as execution time and row counts as a time series.
+
+<CodeExample filePath="guides/data-modeling/metadata/runtime-metadata.py" language="python" title="Using metadata at runtime" />
+
+Any numerical metadata will be treated as a time series in the Dagster UI.
+
+## Standard metadata types
+
+Some metadata keys will be given special treatment in the Dagster UI.
+
+| Key                          | Description                                                                                                   |
+|------------------------------|---------------------------------------------------------------------------------------------------------------|
+| `dagster/uri`                  | **Type:** `str` <br/><br/> The URI for the asset, e.g. "s3://my_bucket/my_object"                                       |
+| `dagster/column_schema`        | **Type:** [`TableSchema`](/todo) <br/><br/> For an asset that's a table, the schema of the columns in the table. Refer to the [Table and column metadata](#table-and-column-metadata) secton for details. |
+| `dagster/column_lineage`       | **Type:** [`TableColumnLineage`](/todo) <br/><br/> For an asset that's a table, the lineage of column inputs to column outputs for the table. Refer to the [Table and column metadata](#table-and-column-metadata) secton for details. |
+| `dagster/row_count`            | **Type:** `int` <br/><br/> For an asset that's a table, the number of rows in the table. Refer to the Table metadata documentation for details. |
+| `dagster/partition_row_count`  | **Type:** `int` <br/><br/> For a partition of an asset that's a table, the number of rows in the partition.              |
+| `dagster/relation_identifier`  | **Type:** `str` <br/><br/> A unique identifier for the table/view, typically fully qualified. For example, my_database.my_schema.my_table |
+| `dagster/code_references`      | **Type:** [`CodeReferencesMetadataValue`](/todo) <br/><br/> A list of code references for the asset, such as file locations or references to Github URLs. Refer to the [Linking your assets with their source code](#linking-your-assets-with-their-source-code) section for details. Should only be provided in definition-level metadata, not materialization metadata. |
+
+
+## Table and column metadata
+
+Two of the most powerful metadata types are [`TableSchema`](/todo) and [`TableColumnLineage`](/todo). These metadata types allow stakeholders to view the schema of a table right within Dagster, and, in Dagster+, navigate the [asset catalog](/todo) via the column lineage.
+
+### Table schema metadata
+
+Here's a quick example of how to attach this metadata at both definition time and runtime:
+
+<CodeExample filePath="guides/data-modeling/metadata/table-schema-metadata.py" language="python" title="Table schema metadata" />
+
+Note that there are several data types and constraints available on [`TableColumn`](/todo) objects. Refer to the API documentation for more information.)
+
+### Column lineage metadata
+
+> Many integrations such as [dbt](https://docs.dagster.io/integrations/dbt/reference) automatically attach this metadata out-of-the-box.
+
+Column lineage metadata is a powerful way to track how columns in a table are derived from other columns. Here is how you can manually attach this metadata:
+
+<CodeExample filePath="guides/data-modeling/metadata/table-column-lineage-metadata.py" language="python" title="Table column lineage metadata" />
+
+> Dagster+ provides rich visualization and navigation of column lineage in the asset catalog. Refer to the [Dagster+ documentation](/dagster-plus) for more information.
+
+
+## Linking your assets with their source code
+
+> This feature is considered experimental and is under active development. This guide will be updated as we roll out new features.
+
+Attaching code reference metadata to your Dagster asset definitions allows you to easily view those assets' source code from the Dagster UI both in local development and in production.
+
+> Many integrations such as [dbt](https://docs.dagster.io/integrations/dbt/reference#attaching-code-reference-metadata) support this capability out of the box. Refer to the integration documentation for more information.
+
+### Attaching Python code references for local development
+
+Dagster can automatically attach code references to your assets during local development with one line of code.
+
+<CodeExample filePath="guides/data-modeling/metadata/python-local-references.py" language="python" title="Local source code references" />
+
+### Attaching custom code references for local development
+
+You can manually add the `dagster/code_references` metadata to your asset definitions to attach custom code references. This is useful when your asset's source code is not primarily in Python (e.g. when parsing a YAML config or integrating with another language).
+
+<CodeExample filePath="guides/data-modeling/metadata/custom-local-references.py" language="python" title="Custom local source code references" />
+
+### Attaching code references in production (Dagster+)
+
+Dagster+ can automatically annotate your assets with code references to source control such as GitHub or GitLab.
+
+<CodeExample filePath="guides/data-modeling/metadata/plus-references.py" language="python" title="Production source code references (Dagster+)" />
+
+### Attaching code references in production (OSS)
+
+If you aren't using Dagster+, you can annotate your assets with code references to source control, but it requires a little more manual mapping.
+
+<CodeExample filePath="guides/data-modeling/metadata/oss-references.py" language="python" title="Production source code references (OSS)" />
+
+[`link_code_references_to_git`](/todo) currently supports GitHub and GitLab repositories. It also supports customization of how file paths are mapped; see the [`AnchorBasedFilePathMapping`](/todo) API docs for more information.

examples/docs_beta_snippets/docs_beta_snippets/guides/data-modeling/metadata/custom-local-references.py

@@ -0,0 +1,27 @@
+from dagster import (
+    CodeReferencesMetadataValue,
+    Definitions,
+    LocalFileCodeReference,
+    asset,
+    with_source_code_references,
+)
+
+
+@asset(
+    metadata={
+        "dagster/code_references": CodeReferencesMetadataValue(
+            code_references=[
+                LocalFileCodeReference(
+                    file_path="/path/to/source.yaml",
+                    # Label and line number are optional
+                    line_number=1,
+                    label="Model YAML",
+                )
+            ]
+        )
+    }
+)
+def my_asset_modeled_in_yaml(): ...
+
+
+defs = Definitions(assets=with_source_code_references([my_asset_modeled_in_yaml]))

examples/docs_beta_snippets/docs_beta_snippets/guides/data-modeling/metadata/definition-metadata.py

@@ -0,0 +1,21 @@
+from dagster import AssetSpec, MetadataValue, asset
+
+
+# You can attach metadata via the `metadata` argument on the `@asset` decorator.
+@asset(
+    metadata={
+        "link_to_docs": MetadataValue.url("https://..."),
+        "snippet": MetadataValue.md("# Embedded markdown\n..."),
+    }
+)
+def my_asset(): ...
+
+
+# You can also use `metadata` with `AssetSpec`
+my_external_asset = AssetSpec(
+    "my_external_asset",
+    metadata={
+        "link_to_docs": MetadataValue.url("https://..."),
+        "snippet": MetadataValue.md("# Embedded markdown\n..."),
+    },
+)

examples/docs_beta_snippets/docs_beta_snippets/guides/data-modeling/metadata/oss-references.py

@@ -0,0 +1,36 @@
+import os
+from pathlib import Path
+
+from dagster import (
+    AnchorBasedFilePathMapping,
+    Definitions,
+    asset,
+    link_code_references_to_git,
+    with_source_code_references,
+)
+
+
+@asset
+def my_asset(): ...
+
+
+@asset
+def another_asset(): ...
+
+
+assets = with_source_code_references([my_asset, another_asset])
+
+defs = Definitions(
+    assets=link_code_references_to_git(
+        assets_defs=assets,
+        git_url="https://github.com/dagster-io/dagster",
+        git_branch="main",
+        file_path_mapping=AnchorBasedFilePathMapping(
+            local_file_anchor=Path(__file__),
+            file_anchor_path_in_repository="src/repo.py",
+        ),
+    )
+    # only link to github in production. link locally otherwise
+    if bool(os.getenv("IS_PRODUCTION"))
+    else assets
+)

examples/docs_beta_snippets/docs_beta_snippets/guides/data-modeling/metadata/owners.py

@@ -0,0 +1,12 @@
+from dagster import AssetSpec, asset
+
+
+# You can attach owners via the `owners` argument on the `@asset` decorator.
+@asset(owners=["[email protected]", "team:data-eng"])
+def my_asset(): ...
+
+
+# You can also use `owners` with `AssetSpec`
+my_external_asset = AssetSpec(
+    "my_external_asset", owners=["[email protected]", "team:roof", "team:corpdev"]
+)

examples/docs_beta_snippets/docs_beta_snippets/guides/data-modeling/metadata/plus-references.py

@@ -0,0 +1,20 @@
+from dagster_cloud.metadata.source_code import link_code_references_to_git_if_cloud
+
+from dagster import Definitions, asset, with_source_code_references
+
+
+@asset
+def my_asset(): ...
+
+
+@asset
+def another_asset(): ...
+
+
+defs = Definitions(
+    assets=link_code_references_to_git_if_cloud(
+        assets_defs=with_source_code_references([my_asset, another_asset]),
+        # This function also supports customizable path mapping, but usually
+        # the defaults are fine.
+    )
+)

examples/docs_beta_snippets/docs_beta_snippets/guides/data-modeling/metadata/python-local-references.py

@@ -0,0 +1,15 @@
+from dagster import Definitions, asset, with_source_code_references
+
+
+@asset
+def my_asset(): ...
+
+
+@asset
+def another_asset(): ...
+
+
+defs = Definitions(
+    # with_source_code_references() automatically attaches the proper metadata
+    assets=with_source_code_references([my_asset, another_asset])
+)

examples/docs_beta_snippets/docs_beta_snippets/guides/data-modeling/metadata/runtime-metadata.py

@@ -0,0 +1,12 @@
+from dagster import MaterializeResult, MetadataValue, asset
+
+
+@asset
+def my_asset():
+    # Asset logic goes here
+    return MaterializeResult(
+        metadata={
+            # file_size_kb will be rendered as a time series
+            "file_size_kb": MetadataValue.int(...)
+        }
+    )

examples/docs_beta_snippets/docs_beta_snippets/guides/data-modeling/metadata/table-column-lineage-metadata.py

@@ -0,0 +1,43 @@
+from dagster import (
+    AssetKey,
+    MaterializeResult,
+    TableColumnDep,
+    TableColumnLineage,
+    asset,
+)
+
+
+@asset(deps=["source_bar", "source_baz"])
+def my_asset():
+    # The following TableColumnLineage models 3 tables:
+    # source_bar, source_baz, and my_asset
+    # With the following column dependencies:
+    # - my_asset.new_column_foo depends on:
+    #   - source_bar.column_bar
+    #   - source_baz.column_baz
+    # - my_asset.new_column_qux depends on:
+    #   - source_bar.column_quuz
+    return MaterializeResult(
+        metadata={
+            "dagster/column_lineage": TableColumnLineage(
+                deps_by_column={
+                    "new_column_foo": [
+                        TableColumnDep(
+                            asset_key=AssetKey("source_bar"),
+                            column_name="column_bar",
+                        ),
+                        TableColumnDep(
+                            asset_key=AssetKey("source_baz"),
+                            column_name="column_baz",
+                        ),
+                    ],
+                    "new_column_qux": [
+                        TableColumnDep(
+                            asset_key=AssetKey("source_bar"),
+                            column_name="column_quuz",
+                        ),
+                    ],
+                }
+            )
+        }
+    )