[3/n][dagster-fivetran] Implement FivetranClient for rework

[maximearmstrong]

Summary & Motivation

This PR implements FivetranClient, which is based on the legacy FivetranResource code. Basically reusing how make_request was implemented and adding more methods to leverage in a subsequent PR to fetch the Fivetran workspace data.

How I Tested These Changes

Additional unit test

[maximearmstrong]

• [11/n][dagster-fivetran] Implement materialization method in FivetranWorkspace #25961 : 3 dependent PRs (#26009 , #26025 , #26181 )
• [dagster-fivetran] Use Fivetran translator instance in load specs fn and state-backed defs #26133
• [10/n][dagster-fivetran] Implement fivetran_assets and build_fivetran_assets_definitions #25944
• [dagster-fivetran] Implement resync_and_poll method in FivetranClient #26062
• [dagster-fivetran] Implement sync_and_poll method in FivetranClient #26061
• [dagster-fivetran] Implement base poll method in FivetranClient #26060
• [dagster-fivetran] Implement base resync method in FivetranClient #26059
• [9/n][dagster-fivetran] Implement base sync methods in FivetranClient #25911
• [8/n][dagster-fivetran] Implement FivetranConnector and FivetranDestination #25889
• [7/n][dagster-fivetran] Implement load_fivetran_asset_specs #25808
• [6/n][dagster-fivetran] Implement FivetranWorkspaceDefsLoader #25807
• [5/n][dagster-fivetran] Implement FivetranWorkspaceData to FivetranConnectorTableProps method #25797
• [4/n][dagster-fivetran] Implement fetch_fivetran_workspace_data #25788
• [3/n][dagster-fivetran] Implement FivetranClient for rework #25756 👈 (View in Graphite)
• [2/n][dagster-fivetran] Update DagsterFivetranTranslator and related classes for rework #25751
• [1/n][dagster-fivetran] Scaffold FivetranWorkspace for rework #25750
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[benpankow]

Nice, looks good to me

[dpeng817]

I won't block review on this, but I'd like to get your thoughts regarding testing strategy here.

One thing that we also started to do with dlift + airlift was have some test battery running live against the integration. IMO when a new person onboards to the integration, something like that is pretty invaluable in terms of getting started with the tooling, I will always trust it better than just pure unit tests especially when running against an external system which can change at any point. The _lack_of these I think is a major reason why we don't feel confidence in integrations which haven't been touched in a while. Doesn't have to be super complicated, but having a "kitchen sink" testbed for the integration similar to what we did for dlift/airlift: https://github.com/dagster-io/dagster/blob/master/examples/experimental/dagster-dlift/kitchen-sink/Makefile could be very nice. Then, you can leverage dagster-buildkite to make it only run in the nightly + on PRs that touch integration code.

Thoughts?

[dpeng817]

In general I find API mocks like this to be relatively inflexible and hard to parse. How would you feel about instead implementing a "client fake" similar to how we did for airlift and dlift? Something like this:
https://github.com/dagster-io/dagster/blob/master/examples/experimental/dagster-dlift/dagster_dlift/test/client_fake.py

I've found it to be more composable and human-readable than mocking. I won't necessarily block review on this, but every time I've based a testing strategy around mocks I've regretted it. I assume this is also somewhat inherited from the old testing strategy, but I think if we're spending time in this code we might as well improve the situation.

[maximearmstrong]

See comment in next PR

[dpeng817]

comment on how these were generated? What is our plan if the schema changes, etc?

[maximearmstrong]

Updated in e4892e8

[dpeng817]

Oops meant to approve.

[maximearmstrong]

One thing that we also started to do with dlift + airlift was have some test battery running live against the integration. IMO when a new person onboards to the integration, something like that is pretty invaluable in terms of getting started with the tooling, I will always trust it better than just pure unit tests especially when running against an external system which can change at any point. The _lack_of these I think is a major reason why we don't feel confidence in integrations which haven't been touched in a while. Doesn't have to be super complicated, but having a "kitchen sink" testbed for the integration similar to what we did for dlift/airlift: https://github.com/dagster-io/dagster/blob/master/examples/experimental/dagster-dlift/kitchen-sink/Makefile could be very nice. Then, you can leverage dagster-buildkite to make it only run in the nightly + on PRs that touch integration code.

Thoughts?

I'm very much in favor of this strategy. I agree that unit tests mocking external services are not enough to make sure that the integration works properly. I'll open a ticket for that.