[8/n][dagster-airbyte] Implement load_airbyte_cloud_asset_specs #26269
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Dec 4, 2024
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
661cd0c to
02a9165
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
d81b894 to
bd93c69
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
02a9165 to
f5d6a1f
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
bd93c69 to
1e2fea5
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
f5d6a1f to
b849b85
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
1e2fea5 to
9154fc9
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
b849b85 to
b49c6ef
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
9154fc9 to
97d2b5a
Compare
dpeng817
reviewed
Dec 5, 2024
|
|
||
|
|
||
| airbyte_cloud_specs = load_airbyte_cloud_asset_specs(airbyte_cloud_workspace) | ||
| defs = dg.Definitions(assets=airbyte_cloud_specs, resources={"airbyte": airbyte_cloud_workspace} |
There was a problem hiding this comment.
you don't need the resource just to load asset specs right
There was a problem hiding this comment.
Right, good catch
dpeng817
reviewed
Dec 5, 2024
Comment on lines
42
to
44
| # 4 tables for the connector | ||
| assert len(all_assets) == 1 | ||
| assert len(all_assets_keys) == 1 |
There was a problem hiding this comment.
this comment seems completely unrelated?
There was a problem hiding this comment.
Indeed - will fix
dpeng817
reviewed
Dec 5, 2024
Comment on lines
46
to
48
| # Sanity check outputs, translator tests cover details here | ||
| first_asset_key = next(key for key in all_assets_keys) | ||
| assert first_asset_key.path == ["test_prefix_test_stream"] |
There was a problem hiding this comment.
name the actual tests in question please
dpeng817
reviewed
Dec 5, 2024
| assert len(all_assets_keys) == 1 | ||
|
|
||
| # Sanity check outputs, translator tests cover details here | ||
| first_asset_key = next(key for key in all_assets_keys) |
There was a problem hiding this comment.
should just call this the_asset_key since we expect one
dpeng817
approved these changes
Dec 5, 2024
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
b49c6ef to
b3fd504
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
97d2b5a to
929f117
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
b3fd504 to
ca76f95
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
929f117 to
0a494a8
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
ca76f95 to
9831074
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
0a494a8 to
8c0d00a
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
9831074 to
5f467c2
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
8c0d00a to
045ac16
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
5f467c2 to
2007ef8
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
045ac16 to
cdb5323
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
2007ef8 to
3630f44
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
2 times, most recently
from
6f56cce to
f578813
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-7
branch
from
d48ba20 to
4648e5c
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
f578813 to
cd95ca5
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-8
branch
from
cd95ca5 to
72c6a47
Compare
pskinnerthyme
pushed a commit
to pskinnerthyme/dagster
that referenced
this pull request
Dec 16, 2024
…ter-io#26269) ## Summary & Motivation This PR implements the `load_airbyte_cloud_asset_specs` function: ```python from dagster_airbyte import AirbyteCloudWorkspace, load_airbyte_cloud_asset_specs import dagster as dg airbyte_cloud_workspace = AirbyteCloudWorkspace( workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"), client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"), client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"), ) airbyte_cloud_specs = load_airbyte_cloud_asset_specs(airbyte_cloud_workspace) defs = dg.Definitions(assets=airbyte_cloud_specs, resources={"airbyte": airbyte_cloud_workspace} ``` ## How I Tested These Changes Additional unit test ## Changelog [dagster-airbyte] The `load_airbyte_cloud_asset_specs` function is added. It can be used with the `AirbyteCloudWorkspace` resource and `DagsterAirbyteTranslator` translator to load your Airbyte Cloud connection streams as external assets in Dagster.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This PR implements the
load_airbyte_cloud_asset_specsfunction:How I Tested These Changes
Additional unit test
Changelog
[dagster-airbyte] The
load_airbyte_cloud_asset_specsfunction is added. It can be used with theAirbyteCloudWorkspaceresource andDagsterAirbyteTranslatortranslator to load your Airbyte Cloud connection streams as external assets in Dagster.