Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] Add more explanation for FilePathMapping for code references #23385

Merged
merged 4 commits into from
Aug 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified docs/content/api/modules.json.gz
Binary file not shown.
Binary file modified docs/content/api/searchindex.json.gz
Binary file not shown.
Binary file modified docs/content/api/sections.json.gz
Binary file not shown.
4 changes: 2 additions & 2 deletions docs/content/guides/dagster/code-references.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Dagster's dbt integration can automatically attach references to the SQL files b

In some cases, you may want to manually attach code references to your asset definitions. Some assets may have a more complex source structure, such as an asset whose definition is spread across multiple Python source files or an asset which is partially defined with a `.sql` model file.

To manually attach code references to an asset definition, use `CodeReferencesMetadataValue`. You can then choose to augment these manual references with `with_source_code_references`:
To manually attach code references to an asset definition, use <PyObject module="dagster" object="CodeReferencesMetadataValue"/>. You can then choose to augment these manual references with <PyObject module="dagster" object="with_source_code_references"/>:

```python file=/guides/dagster/code_references/manual_references.py
import os
Expand Down Expand Up @@ -165,7 +165,7 @@ defs = Definitions(

### In any Dagster environment

The <PyObject module="dagster" object="link_code_references_to_git"/> utility allows you to convert local file code references to source control links. You'll need to provide the base URL of your git repository, the branch or commit hash, and a path to the repository root locally.
The <PyObject module="dagster" object="link_code_references_to_git"/> utility allows you to convert local file code references to source control links. You'll need to provide the base URL of your git repository, the branch or commit hash, and a <PyObject module="dagster" object="FilePathMapping"/> which tells Dagster how to convert local file paths to paths in the repository. The simplest way to do so is with an <PyObject module="dagster" object="AnchorBasedFilePathMapping"/>, which uses a local file path and the corresponding path in the repository to infer the mapping for other files.

```python file=/guides/dagster/code_references/link_to_source_control.py
from pathlib import Path
Expand Down
Binary file modified docs/next/public/objects.inv
Binary file not shown.
6 changes: 6 additions & 0 deletions docs/sphinx/sections/api/apidocs/metadata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ All metadata types inherit from `MetadataValue`. The following types are defined

.. autoclass:: UrlMetadataValue

.. autoclass:: CodeReferencesMetadataValue

Tables
^^^^^^

Expand Down Expand Up @@ -76,3 +78,7 @@ For more information, refer to the `Linking to asset definition code with code r
.. autofunction:: with_source_code_references

.. autofunction:: link_code_references_to_git

.. autoclass:: FilePathMapping

.. autoclass:: AnchorBasedFilePathMapping
38 changes: 37 additions & 1 deletion examples/quickstart_etl/quickstart_etl/definitions.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,52 @@
from pathlib import Path

from dagster import (
Definitions,
ScheduleDefinition,
define_asset_job,
graph_asset,
link_code_references_to_git,
load_assets_from_package_module,
op,
with_source_code_references,
)
from dagster._core.definitions.metadata.source_code import AnchorBasedFilePathMapping

from . import assets

daily_refresh_schedule = ScheduleDefinition(
job=define_asset_job(name="all_assets_job"), cron_schedule="0 0 * * *"
)


@op
def foo_op():
return 5


@graph_asset
def my_asset():
return foo_op()


my_assets = with_source_code_references(
[
my_asset,
*load_assets_from_package_module(assets),
]
)

my_assets = link_code_references_to_git(
assets_defs=my_assets,
git_url="https://github.com/dagster-io/dagster/",
git_branch="master",
file_path_mapping=AnchorBasedFilePathMapping(
local_file_anchor=Path(__file__).parent,
file_anchor_path_in_repository="examples/quickstart_etl/quickstart_etl/",
),
)

defs = Definitions(
assets=load_assets_from_package_module(assets), schedules=[daily_refresh_schedule]
assets=my_assets,
schedules=[daily_refresh_schedule],
)
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from typing import TYPE_CHECKING, Any, Callable, List, Optional, Sequence, Union

import dagster._check as check
from dagster._annotations import experimental
from dagster._annotations import experimental, public
from dagster._model import DagsterModel
from dagster._serdes import whitelist_for_serdes

Expand Down Expand Up @@ -160,8 +160,17 @@ class FilePathMapping(ABC):
mapping function can be provided to handle these cases.
"""

@public
@abstractmethod
def convert_to_source_control_path(self, local_path: Path) -> str: ...
def convert_to_source_control_path(self, local_path: Path) -> str:
"""Maps a local file path to the corresponding path in a source control repository.

Args:
local_path (Path): The local file path to map.

Returns:
str: The corresponding path in the hosted source control repository, relative to the repository root.
"""


@experimental
Expand Down Expand Up @@ -192,7 +201,17 @@ class AnchorBasedFilePathMapping(FilePathMapping):
local_file_anchor: Path
file_anchor_path_in_repository: str

@public
def convert_to_source_control_path(self, local_path: Path) -> str:
"""Maps a local file path to the corresponding path in a source control repository
based on the anchor file and its corresponding path in the repository.

Args:
local_path (Path): The local file path to map.

Returns:
str: The corresponding path in the hosted source control repository, relative to the repository root.
"""
path_from_anchor_to_target = os.path.relpath(
local_path,
self.local_file_anchor,
Expand Down