[11/n][dagster-fivetran] Implement materialization method in FivetranWorkspace #25961

maximearmstrong · 2024-11-15T20:42:25Z

Summary & Motivation

This PR implements FivetranWorkspace.sync_and_poll, the materialization method for Fivetran assets. This method:

calls FivetranClient.sync_and_poll
takes the FivetranOutput returned by FivetranClient.sync_and_poll and generates the asset materializations
yields MaterializeResult for each expected asset and AssetMaterialization for each unexpected asset
- a connector table that was not in the connector at definitions loading time can be in the FivetranOutput. Eg. the table was added after definitions loading time and before sync.
logs a warning for each unmaterialized table
- a connector table can be created at definitions loading time, but can be missing in the FivetranOutput. Eg. the table was deleted after definitions loading time and before sync.

Can be leveraged like:

from dagster_fivetran import FivetranWorkspace, fivetran_assets

import dagster as dg

fivetran_workspace = FivetranWorkspace(
    account_id=dg.EnvVar("FIVETRAN_API_KEY"),
    api_key=dg.EnvVar("FIVETRAN_API_KEY"),
    api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),
)


@fivetran_assets(
    connector_id="connector_id",
    name="connector_id",
    group_name="connector_id",
    workspace=fivetran_workspace,
)
def fivetran_connector_assets(context: dg.AssetExecutionContext, fivetran: FivetranWorkspace):
    yield from fivetran.sync_and_poll(context=context)


defs = dg.Definitions(
    assets=[fivetran_connector_assets],
    resources={"fivetran": fivetran_workspace},
)

How I Tested These Changes

Additional tests with BK

Tested with a live Fivetran instance:

Changelog

[dagster-fivetran]

maximearmstrong · 2024-11-15T20:42:55Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py

dpeng817 · 2024-12-03T21:00:26Z

python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py

+                materialized_asset_keys.add(materialization.asset_key)
+            else:
+                context.log.warning(
+                    f"An unexpected asset was materialized: {materialization.asset_key}"


maybe note that we're still going to yield a materialization event.

python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py

dpeng817 · 2024-12-03T21:24:23Z

python_modules/libraries/dagster-fivetran/dagster_fivetran_tests/experimental/conftest.py

@@ -501,3 +515,25 @@ def all_api_mocks_fixture(
        status=200,
    )
    yield fetch_workspace_data_api_mocks
+
+
+@pytest.fixture(name="sync_and_poll")


poll_and_sync vs sync_and_poll

dpeng817 · 2024-12-03T21:25:12Z

python_modules/libraries/dagster-fivetran/dagster_fivetran_tests/experimental/test_resources.py

+    connector_id: str,
+    fetch_workspace_data_api_mocks: responses.RequestsMock,
+    sync_and_poll: MagicMock,
+) -> None:


nit: mention that this is actually testing sync_and_poll. Calling it the materialization method feels a bit overly specified to me.

dpeng817 · 2024-12-03T21:27:07Z

python_modules/libraries/dagster-fivetran/dagster_fivetran_tests/experimental/test_resources.py

+        assert len(materialized_asset_keys) == 4
+        assert my_fivetran_assets.keys == materialized_asset_keys
+
+        # Mocked FivetranClient.sync_and_poll returns API response


would be nice to have some way of determining that the expected log messages actually happened - this just kinda makes sure it doesn't error. Could potentially directly invoke the fivetran asset and assert against the materializeresult objects + AssetMaterializations when appropriate

Done with capsys in 36e9da9

…Workspace

…Workspace (dagster-io#25961) ## Summary & Motivation This PR implements `FivetranWorkspace.sync_and_poll`, the materialization method for Fivetran assets. This method: - calls `FivetranClient.sync_and_poll` - takes the FivetranOutput returned by `FivetranClient.sync_and_poll` and generates the asset materializations - yields `MaterializeResult` for each expected asset and `AssetMaterialization` for each unexpected asset - a connector table that was not in the connector at definitions loading time can be in the FivetranOutput. Eg. the table was added after definitions loading time and before sync. - logs a warning for each unmaterialized table - a connector table can be created at definitions loading time, but can be missing in the FivetranOutput. Eg. the table was deleted after definitions loading time and before sync. Can be leveraged like: ```python from dagster_fivetran import FivetranWorkspace, fivetran_assets import dagster as dg fivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_API_KEY"), api_key=dg.EnvVar("FIVETRAN_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_API_SECRET"), ) @fivetran_assets( connector_id="connector_id", name="connector_id", group_name="connector_id", workspace=fivetran_workspace, ) def fivetran_connector_assets(context: dg.AssetExecutionContext, fivetran: FivetranWorkspace): yield from fivetran.sync_and_poll(context=context) defs = dg.Definitions( assets=[fivetran_connector_assets], resources={"fivetran": fivetran_workspace}, ) ``` ## How I Tested These Changes Additional tests with BK Tested with a live Fivetran instance: <img width="1251" alt="Screenshot 2024-11-26 at 5 46 31 PM" src="https://github.com/user-attachments/assets/e4253119-045f-4ac7-8b98-eb805e24a843"> ## Changelog [dagster-fivetran]

This was referenced Nov 15, 2024

[9/n][dagster-fivetran] Implement base sync methods in FivetranClient #25911

Merged

[10/n][dagster-fivetran] Implement fivetran_assets and build_fivetran_assets_definitions #25944

Merged

graphite-app bot reviewed Nov 15, 2024

View reviewed changes

python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py Outdated Show resolved Hide resolved

python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py Outdated Show resolved Hide resolved

maximearmstrong force-pushed the maxime/rework-fivetran-10 branch from 0521c7f to 6e115f1 Compare November 19, 2024 18:13

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch from 0e0b1e2 to 78fe129 Compare November 19, 2024 18:13

maximearmstrong mentioned this pull request Nov 19, 2024

[dagster-fivetran] Mark Fivetran legacy code as deprecated #26009

Open

maximearmstrong force-pushed the maxime/rework-fivetran-10 branch from 6e115f1 to e07232e Compare November 20, 2024 00:48

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch from 78fe129 to d3f1bf6 Compare November 20, 2024 00:48

This was referenced Nov 20, 2024

[docs] Move legacy Fivetran docs #26025

Merged

[docs] Add docs for reworked Fivetran API #26026

Merged

maximearmstrong force-pushed the maxime/rework-fivetran-10 branch from e07232e to 3be662b Compare November 21, 2024 23:55

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch from d3f1bf6 to b9e54d6 Compare November 21, 2024 23:55

maximearmstrong force-pushed the maxime/rework-fivetran-10 branch from 3be662b to 5a87eb7 Compare November 22, 2024 13:54

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch 4 times, most recently from b417e4e to 3150e3e Compare November 22, 2024 14:22

maximearmstrong force-pushed the maxime/rework-fivetran-10 branch from ebe8457 to 748298f Compare November 27, 2024 19:28

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch from c16e280 to 2df7622 Compare November 27, 2024 19:28

maximearmstrong changed the base branch from maxime/rework-fivetran-10 to maxime/use-translator-instance-in-load-fivetran-asset-specs November 27, 2024 19:28

maximearmstrong force-pushed the maxime/use-translator-instance-in-load-fivetran-asset-specs branch from d2fa78b to 67d0fcd Compare November 27, 2024 20:15

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch from 2df7622 to 3317a7d Compare November 27, 2024 20:15

maximearmstrong force-pushed the maxime/use-translator-instance-in-load-fivetran-asset-specs branch from 67d0fcd to 6cf70a0 Compare November 27, 2024 21:18

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch from 3317a7d to b0e8961 Compare November 27, 2024 21:18

maximearmstrong mentioned this pull request Nov 27, 2024

[dagster-fivetran] Implement get_columns_config_for_table in FivetranClient #26181

Merged

dpeng817 reviewed Dec 3, 2024

View reviewed changes

python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py Show resolved Hide resolved

dpeng817 reviewed Dec 3, 2024

View reviewed changes

dpeng817 approved these changes Dec 3, 2024

View reviewed changes

maximearmstrong force-pushed the maxime/use-translator-instance-in-load-fivetran-asset-specs branch from 6cf70a0 to 47cb47a Compare December 5, 2024 18:40

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch from b0e8961 to c94dc6b Compare December 5, 2024 18:40

maximearmstrong force-pushed the maxime/use-translator-instance-in-load-fivetran-asset-specs branch from 47cb47a to 79f0f78 Compare December 5, 2024 20:13

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch from c94dc6b to 36e9da9 Compare December 5, 2024 20:13

Base automatically changed from maxime/use-translator-instance-in-load-fivetran-asset-specs to master December 5, 2024 20:46

maximearmstrong added 7 commits December 5, 2024 15:46

[11/n][dagster-fivetran] Implement materialization method in Fivetran…

9a58787

…Workspace

Remove fetch_column_metadata and infer_missing_tables

63f49ee

Lint

fdc139e

Update materialization logic

6a0d6f9

Move materialization logic to method

14c7d68

Update materialization logic; add tests

0824a5c

Update post review

99e73bd

maximearmstrong force-pushed the maxime/rework-fivetran-11 branch from 36e9da9 to 99e73bd Compare December 5, 2024 20:46

maximearmstrong merged commit f6a914b into master Dec 5, 2024
1 check failed

maximearmstrong deleted the maxime/rework-fivetran-11 branch December 5, 2024 21:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[11/n][dagster-fivetran] Implement materialization method in FivetranWorkspace #25961

[11/n][dagster-fivetran] Implement materialization method in FivetranWorkspace #25961

maximearmstrong commented Nov 15, 2024 •

edited

Loading

maximearmstrong commented Nov 15, 2024 •

edited

Loading

dpeng817 Dec 3, 2024

dpeng817 Dec 3, 2024

dpeng817 Dec 3, 2024

dpeng817 Dec 3, 2024

maximearmstrong Dec 5, 2024

[11/n][dagster-fivetran] Implement materialization method in FivetranWorkspace #25961

[11/n][dagster-fivetran] Implement materialization method in FivetranWorkspace #25961

Conversation

maximearmstrong commented Nov 15, 2024 • edited Loading

Summary & Motivation

How I Tested These Changes

Changelog

maximearmstrong commented Nov 15, 2024 • edited Loading

dpeng817 Dec 3, 2024

Choose a reason for hiding this comment

dpeng817 Dec 3, 2024

Choose a reason for hiding this comment

dpeng817 Dec 3, 2024

Choose a reason for hiding this comment

dpeng817 Dec 3, 2024

Choose a reason for hiding this comment

maximearmstrong Dec 5, 2024

Choose a reason for hiding this comment

maximearmstrong commented Nov 15, 2024 •

edited

Loading

maximearmstrong commented Nov 15, 2024 •

edited

Loading