Update Dagster Airbyte to use the new Airbyte API

[spazylbekova]

What's the use case?

Airbyte transitions from the configuration API to the new API. Airbyte does not have active commitments to support configuration API long-term. I was wondering if there are any plans to support the new Airbyte API. We are struggling to make configuration API work for us since we use Okta for webapp.

API documentation
https://docs.airbyte.com/api-documentation

Ideas of implementation

Update dagster-airbyte library to support the new Airbyte API

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[tangos974]

Hi, I'd like to follow up on this issue, as per the airbyte docs:

"To increase security, Airbyte is migrating to use in-app Applications to secure access to the Airbyte API. As a result, we're removing the portal.airbyte.com site, which has previously been used for API key management.

API keys generated from the previous Portal will stop working on August 15, 2024."

So that means two things:

• People (like me) who haven't generated an airbyte api key the old way (before portal.airbyte.com was removed) cannot use the dagster_airbyte library to create an airbyte asset in dagster - i.e. we can't pilot airbyte cloud from dagster at all without sacrificing a lot of the functionalities that made us choose dagster in the first place.

• In 2 weeks from now, all keys generated that way will stop working. That means that the amount of people in my case will grow substantially overnight.

Is there an ETA for updating the dagster_airbyte library to support the new Airbyte API? Should my team and I begin working on a workaround, or is there an update planned for the next release?

Thanks in advance.

[garethbrickman]

@tangos974 #23451