Replace dagster_airbyte load_assets_from_airbyte_project #21230
adam-bloom
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
https://docs.dagster.io/integrations/airbyte#loading-airbyte-asset-definitions-from-yaml-config
This worked really well for a large number of airbyte connections. It is significantly more performant than
load_assets_from_airbyte_instance
and shortens dagster start times. (I had a use case where dagster startup just timed out on me when I tried to useload_assets_from_airbyte_instance
.)However, it relies on the
octavia
file structure. Airbyte has removedoctavia
completely, in favor of their new (if a little buggy) terraform provider. As such, we need a new way to define local airbyte connections. I was able to build a local implementation that wrapsbuild_airbyte_assets
easily enough, but I'm wondering if we should build an official way to extract information from terraform state files (I just duplicated). Additionally, we should probably update docs/deprecate the asset loader that depends onoctavia
Semi related to #19051
Beta Was this translation helpful? Give feedback.
All reactions