Skip to content

Commit

Permalink
[dagster-fivetran] Add fetch_column_metadata docs for Fivetran (#26813)
Browse files Browse the repository at this point in the history
## Summary & Motivation

As title. Added for both new and old docs site.

## How I Tested These Changes

BK and docs preview
  • Loading branch information
maximearmstrong authored Jan 3, 2025
1 parent f0cf1f1 commit 18c3c18
Show file tree
Hide file tree
Showing 10 changed files with 81 additions and 0 deletions.
Binary file modified docs/content/api/modules.json.gz
Binary file not shown.
Binary file modified docs/content/api/searchindex.json.gz
Binary file not shown.
Binary file modified docs/content/api/sections.json.gz
Binary file not shown.
30 changes: 30 additions & 0 deletions docs/content/integrations/fivetran/fivetran.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,36 @@ Note that `super()` is called in each of the overridden methods to generate the

You can pass an instance of the custom <PyObject module="dagster_fivetran" object="DagsterFivetranTranslator" /> to the <PyObject module="dagster_fivetran" method="fivetran_assets" /> decorator or the <PyObject module="dagster_fivetran" method="build_fivetran_assets_definitions" /> factory.

### Fetching column-level metadata for Fivetran assets

Dagster allows you to emit column-level metadata, like [column schema](/concepts/metadata-tags/asset-metadata/table-metadata#attaching-column-schema) and [column lineage](/concepts/metadata-tags/asset-metadata/column-level-lineage), as [materialization metadata](/concepts/metadata-tags/asset-metadata).

With this metadata, you can view documentation in Dagster for all columns in your Fivetran connector tables.

To enable this feature, call <PyObject object="fetch_column_metadata" module="dagster_fivetran.fivetran_event_iterator.FivetranEventIterator" displayText="fetch_column_metadata()" /> on the <PyObject object="FivetranEventIterator" module="dagster_fivetran.fivetran_event_iterator" /> returned by the `sync_and_poll()` call on the <PyObject module="dagster_fivetran" object="FivetranWorkspace" /> resource.

```python file=/integrations/fivetran/fetch_column_metadata_fivetran_assets.py
from dagster_fivetran import FivetranWorkspace, fivetran_assets

import dagster as dg

fivetran_workspace = FivetranWorkspace(
account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"),
api_key=dg.EnvVar("FIVETRAN_API_KEY"),
api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),
)


@fivetran_assets(
connector_id="fivetran_connector_id",
workspace=fivetran_workspace,
)
def fivetran_connector_assets(
context: dg.AssetExecutionContext, fivetran: FivetranWorkspace
):
yield from fivetran.sync_and_poll(context=context).fetch_column_metadata()
```

### Load Fivetran assets from multiple workspaces

Definitions from multiple Fivetran workspaces can be combined by instantiating multiple <PyObject module="dagster_fivetran" object="FivetranWorkspace" /> resources and merging their specs. This lets you view all your Fivetran assets in a single asset graph:
Expand Down
10 changes: 10 additions & 0 deletions docs/docs-beta/docs/integrations/libraries/fivetran.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,16 @@ Note that `super()` is called in each of the overridden methods to generate the

You can pass an instance of the custom <PyObject module="dagster_fivetran" object="DagsterFivetranTranslator" /> to the <PyObject module="dagster_fivetran" method="fivetran_assets" /> decorator or the <PyObject module="dagster_fivetran" method="build_fivetran_assets_definitions" /> factory.

### Fetching column-level metadata for Fivetran assets

Dagster allows you to emit column-level metadata, like [column schema](/guides/build/create-asset-pipelines/metadata#standard-metadata-types) and [column lineage](/guides/build/create-asset-pipelines/metadata#column-lineage), as [materialization metadata](/guides/build/create-asset-pipelines/metadata#runtime-metadata).

With this metadata, you can view documentation in Dagster for all columns in your Fivetran connector tables.

To enable this feature, call <PyObject object="fetch_column_metadata" module="dagster_fivetran.fivetran_event_iterator.FivetranEventIterator" displayText="fetch_column_metadata()" /> on the <PyObject object="FivetranEventIterator" module="dagster_fivetran.fivetran_event_iterator" /> returned by the `sync_and_poll()` call on the <PyObject module="dagster_fivetran" object="FivetranWorkspace" /> resource.

<CodeExample filePath="integrations/fivetran/fetch_column_metadata_fivetran_assets.py" language="python" />

### Load Fivetran assets from multiple workspaces

Definitions from multiple Fivetran workspaces can be combined by instantiating multiple <PyObject module="dagster_fivetran" object="FivetranWorkspace" /> resources and merging their specs. This lets you view all your Fivetran assets in a single asset graph:
Expand Down
Binary file modified docs/next/public/objects.inv
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ Assets (Fivetran API)

.. autofunction:: build_fivetran_assets_definitions

.. autoclass:: dagster_fivetran.fivetran_event_iterator.FivetranEventIterator


Legacy
======
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
from dagster_fivetran import FivetranWorkspace, fivetran_assets

import dagster as dg

fivetran_workspace = FivetranWorkspace(
account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"),
api_key=dg.EnvVar("FIVETRAN_API_KEY"),
api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),
)


@fivetran_assets(
connector_id="fivetran_connector_id",
workspace=fivetran_workspace,
)
def fivetran_connector_assets(
context: dg.AssetExecutionContext, fivetran: FivetranWorkspace
):
yield from fivetran.sync_and_poll(context=context).fetch_column_metadata()
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
f"{snippets_folder}/fivetran/multiple_fivetran_workspaces.py",
f"{snippets_folder}/fivetran/representing_fivetran_assets.py",
f"{snippets_folder}/fivetran/sync_and_materialize_fivetran_assets.py",
f"{snippets_folder}/fivetran/fetch_column_metadata_fivetran_assets.py",
f"{snippets_folder}/airbyte_cloud/customize_airbyte_cloud_asset_defs.py",
f"{snippets_folder}/airbyte_cloud/customize_airbyte_cloud_translator_asset_spec.py",
f"{snippets_folder}/airbyte_cloud/multiple_airbyte_cloud_workspaces.py",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
from dagster_fivetran import FivetranWorkspace, fivetran_assets

import dagster as dg

fivetran_workspace = FivetranWorkspace(
account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"),
api_key=dg.EnvVar("FIVETRAN_API_KEY"),
api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),
)


@fivetran_assets(
connector_id="fivetran_connector_id",
workspace=fivetran_workspace,
)
def fivetran_connector_assets(
context: dg.AssetExecutionContext, fivetran: FivetranWorkspace
):
yield from fivetran.sync_and_poll(context=context).fetch_column_metadata()

2 comments on commit 18c3c18

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-j1mpt0j1f-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 18c3c18.
This pull request is being automatically deployed with vercel-action

@github-actions
Copy link

@github-actions github-actions bot commented on 18c3c18 Jan 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs-beta ready!

✅ Preview
https://dagster-docs-beta-8yh9djtk8-elementl.vercel.app

Built with commit 18c3c18.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.