[6/n][dagster-airbyte] Implement AirbyteWorkspaceData to AirbyteConnectionTableProps method #26257
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Dec 3, 2024
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
9606d1d to
904c458
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
4be30a4 to
61380f0
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
904c458 to
30495b8
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
61380f0 to
726f64a
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
30495b8 to
41211ed
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
726f64a to
3a8a668
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
41211ed to
bfed07f
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
3a8a668 to
7b1189b
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
bfed07f to
a99e648
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
7b1189b to
f9cf32c
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
a99e648 to
ab61c91
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
f9cf32c to
d231cae
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
ab61c91 to
c986086
Compare
maximearmstrong
commented
Dec 4, 2024
|
|
||
| class AirbyteConnectionTableProps(NamedTuple): ... | ||
|
|
||
| class AirbyteConnectionTableProps(NamedTuple): |
There was a problem hiding this comment.
We could also name this AirbyteConnectionStreamProps if we want to keep as close as possible to Airbyte's ontology.
dpeng817
reviewed
Dec 5, 2024
Comment on lines
30
to
39
| assert len(table_props_data) == 1 | ||
| first_table_props = next(iter(table_props_data)) | ||
| assert first_table_props.table_name == f"{TEST_STREAM_PREFIX}{TEST_STREAM_NAME}" | ||
| assert first_table_props.stream_prefix == TEST_STREAM_PREFIX | ||
| assert first_table_props.stream_name == TEST_STREAM_NAME | ||
| assert first_table_props.connection_id == TEST_CONNECTION_ID | ||
| assert first_table_props.connection_name == TEST_CONNECTION_NAME | ||
| assert first_table_props.json_schema == TEST_JSON_SCHEMA | ||
| assert first_table_props.database == TEST_DESTINATION_DATABASE | ||
| assert first_table_props.schema == TEST_DESTINATION_SCHEMA |
There was a problem hiding this comment.
nit; could just create an "expected table props data" and have a single assertion instead of many (might require making this a record)
There was a problem hiding this comment.
Done in 0d1570d
dpeng817
approved these changes
Dec 5, 2024
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
d231cae to
4054ff8
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
c986086 to
03f3e0b
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
4054ff8 to
da101d9
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
03f3e0b to
0137ae8
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
da101d9 to
874b666
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
0137ae8 to
ef486b7
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
874b666 to
3ca1897
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
ef486b7 to
c575675
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-5
branch
from
3ca1897 to
977b2d4
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
c575675 to
0d1570d
Compare
maximearmstrong
force-pushed
the
maxime/rework-airbyte-cloud-6
branch
from
0d1570d to
b2af23f
Compare
pskinnerthyme
pushed a commit
to pskinnerthyme/dagster
that referenced
this pull request
Dec 16, 2024
…ctionTableProps method (dagster-io#26257) ## Summary & Motivation This PR implement AirbyteWorkspaceData.to_airbyte_connection_table_props, which creates the props that will be passed to the translator. These props include every detail required to create the asset spec for a given table (Airbyte stream) ## How I Tested These Changes Additional unit tests with BK
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This PR implement AirbyteWorkspaceData.to_airbyte_connection_table_props, which creates the props that will be passed to the translator. These props include every detail required to create the asset spec for a given table (Airbyte stream)
How I Tested These Changes
Additional unit tests with BK