[5/n][dagster-airbyte] Implement fetch_airbyte_workspace_data #26253
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Dec 3, 2024
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-4
branch
from
af60e39 to
bfade38
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
4dc0619 to
e8ae9d0
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
python_modules/libraries/dagster-airbyte/dagster_airbyte_tests/experimental/test_asset_specs.py
Outdated
Show resolved
Hide resolved
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-4
branch
from
bfade38 to
1520501
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
4be30a4 to
61380f0
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-4
branch
from
1520501 to
f295f42
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
61380f0 to
726f64a
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-4
branch
from
f295f42 to
c3daf88
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
2 times, most recently
from
3a8a668 to
7b1189b
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-4
branch
from
7eb0fde to
224d89c
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
2 times, most recently
from
f9cf32c to
d231cae
Compare
maximearmstrong
commented
Dec 4, 2024
|
|
||
| @whitelist_for_serdes | ||
| @record | ||
| class AirbyteStream: |
There was a problem hiding this comment.
A stream in Airbyte corresponds to a table. This could be also called AirbyteTable, but I kept Airbyte's ontology here.
There was a problem hiding this comment.
Maybe note that in docstring but makes sense.
dpeng817
reviewed
Dec 5, 2024
Comment on lines
+1043
to
+1044
| connection_id=partial_connection_details["connectionId"] | ||
| ) | ||
| connection = AirbyteConnection.from_connection_details( | ||
| connection_details=full_connection_details | ||
| ) | ||
| connections_by_id[connection.id] = connection |
There was a problem hiding this comment.
is there any casse where get_connection_details shouldn't just return AirbyteConnection? Similar for destinations. Just feels like it would simplify this loop quite a bit.
There was a problem hiding this comment.
We keep only what is needed for the translation of data to AssetSpec in AirbyteConnection and AirbyteDestination, but users could use the client and call get_connections with it something like:
from dagster_airbyte import AirbyteCloudWorkspace, load_airbyte_cloud_asset_specs
import dagster as dg
airbyte_cloud_workspace = AirbyteCloudWorkspace(
workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"),
client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"),
client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),
)
@airbyte_assets(
connection_id="connection_id",
workspace=airbyte_cloud_workspace,
)
def airbyte_connection_assets(context: dg.AssetExecutionContext, airbyte: AirbyteCloudWorkspace):
client = airbyte.get_client()
connections = client.get_connections()
# do something with connections
...
yield from airbyte.sync_and_poll(context=context)In this case, they may want something in the raw API response.
dpeng817
approved these changes
Dec 5, 2024
dpeng817
reviewed
Dec 5, 2024
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-4
branch
from
414214d to
d19da64
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
d231cae to
4054ff8
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-4
branch
from
d19da64 to
59f1db9
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
4054ff8 to
da101d9
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-4
branch
from
59f1db9 to
2ffd65b
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
da101d9 to
874b666
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-4
branch
from
2ffd65b to
35bc6f4
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
874b666 to
3ca1897
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
3ca1897 to
977b2d4
Compare
pskinnerthyme
pushed a commit
to pskinnerthyme/dagster
that referenced
this pull request
Dec 16, 2024
…r-io#26253) ## Summary & Motivation This PR implements `AirbyteCloudWorkspace.fetch_airbyte_workspace_data`, that fetches the connections and destinations included in a given workspace. ## How I Tested These Changes Additional unit tests with BK.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This PR implements
AirbyteCloudWorkspace.fetch_airbyte_workspace_data, that fetches the connections and destinations included in a given workspace.How I Tested These Changes
Additional unit tests with BK.