[dagster-fivetran] Implement base poll method in FivetranClient #26060
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Nov 20, 2024
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
python_modules/libraries/dagster-fivetran/dagster_fivetran/resources.py
Outdated
Show resolved
Hide resolved
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
cadbe37 to
90123af
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
469b919 to
6fff27d
Compare
dpeng817
approved these changes
Nov 20, 2024
| if connector.last_sync_completed_at > previous_sync_completed_at: | ||
| break | ||
|
|
||
| if poll_timeout and datetime.now() > poll_start + timedelta(seconds=poll_timeout): |
| # Sleep for the configured time interval before polling again. | ||
| time.sleep(poll_interval) | ||
|
|
||
| if not connector.is_last_sync_successful: |
| @@ -59,3 +61,10 @@ def test_basic_resource_request( | |||
| client.start_resync(connector_id=connector_id, resync_parameters={"property1": ["string"]}) | |||
| assert len(all_api_mocks.calls) == 3 | |||
| assert f"{connector_id}/schemas/tables/resync" in all_api_mocks.calls[2].request.url | |||
|
|
|||
| # poll calls | |||
| all_api_mocks.calls.reset() | |||
There was a problem hiding this comment.
I know that this is just re-originating a method that exists prior to this PR, but I'm just looking at all the unexercised code paths [1] and [2]; would be nice to test failure conditions here if it's not hard. Won't block review on that, especially since the problem exists beyond just this PR
There was a problem hiding this comment.
That's totally fair - done in a55e620
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
90123af to
b81d516
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
6fff27d to
77163b2
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
b81d516 to
004f258
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
77163b2 to
a55e620
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
004f258 to
03b4f71
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
a55e620 to
7ff0145
Compare
This was referenced Nov 21, 2024
Merged
Merged
[5/n][dagster-fivetran] Implement
FivetranWorkspaceData to FivetranConnectorTableProps method
#25797
Merged
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
03b4f71 to
6855c14
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
3bdb351 to
0d4c5c3
Compare
This was referenced Nov 23, 2024
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
6855c14 to
2021ba0
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
0d4c5c3 to
919e38e
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
2021ba0 to
3df2067
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
919e38e to
b85bbe8
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
3df2067 to
22bfe6c
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
b85bbe8 to
c7250c1
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
22bfe6c to
ffab308
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
c7250c1 to
9161976
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
ffab308 to
16121e5
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
9161976 to
13b8e6d
Compare
maximearmstrong
force-pushed
the
maxime/implement-resync-base-method-fivetran-client
branch
from
16121e5 to
8a48241
Compare
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
13b8e6d to
f039336
Compare
Base automatically changed from
maxime/implement-resync-base-method-fivetran-client
to
master
November 27, 2024 13:44
maximearmstrong
force-pushed
the
maxime/implement-base-poll-method-fivetran-client
branch
from
f039336 to
4b0d41a
Compare
maximearmstrong
deleted the
maxime/implement-base-poll-method-fivetran-client
branch
cmpadden
pushed a commit
that referenced
this pull request
Dec 5, 2024
## Summary & Motivation This PR reworks legacy oll method and implements it in the `FivetranClient`: - `poll_sync` is added based on legacy `poll_sync` - the way of handling the connector sync status has been updated - Logic has been reworked and moved to `FivetranConnector` properties Tests mock the request API calls and make sure that all calls are made. ## How I Tested These Changes Additional unit tests with BK
pskinnerthyme
pushed a commit
to pskinnerthyme/dagster
that referenced
this pull request
Dec 16, 2024
…ter-io#26060) ## Summary & Motivation This PR reworks legacy oll method and implements it in the `FivetranClient`: - `poll_sync` is added based on legacy `poll_sync` - the way of handling the connector sync status has been updated - Logic has been reworked and moved to `FivetranConnector` properties Tests mock the request API calls and make sure that all calls are made. ## How I Tested These Changes Additional unit tests with BK
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This PR reworks legacy oll method and implements it in the
FivetranClient:poll_syncis added based on legacypoll_syncFivetranConnectorpropertiesTests mock the request API calls and make sure that all calls are made.
How I Tested These Changes
Additional unit tests with BK