Community-supported integration libraries

[yuhan]

If you're hosting a Dagster integration library on your own, let us know on this discussion.

We'll help surface it in the documentation.

If you are looking for a place to put your integration, we recommend the community-integrations repository. In this repo we'll manage the builds and releases for you:

• https://github.com/dagster-io/community-integrations

[PedramNavid]

Here's one: https://github.com/hightouchio/dagster-hightouch :)

[PedramNavid]

Here's two https://github.com/hex-inc/dagster-hex

[msnidal]

Hello! I made this integration for Stitch (a managed batch extractor service similar to Fivetran or Airbyte's cloud offering) recently - I tried to see if there was an existing integration since it got referenced in this user story but there was nothing out there so I decided to make one! I based it off the Fivetran and Airbyte libraries though the actual extraction/load mechanics are quite different by the way Stitch works, and likewise licensed it as Apache 2.0 which seems to be the norm in the ecosystem.

There are still some features I intend to add over the next couple weeks (mainly a proper reconciliation system) but as-is it supports manually triggering extracts/loads and materializing assets for each of the streams (tables) built that can be referenced downstream. I've published it on PyPI and done my best to test it though its still in development so feedback is always welcome!

https://github.com/msnidal/dagster-stitch

[acarter24]

I found a great Meltano integration by @JulesHuisman

https://github.com/quantile-development/dagster-meltano

[ThomAub]

Hey we open sourced some time ago this integration https://github.com/PayLead/dagster-nomad

[AbdallahMGomaa]

Our company Perian has developed this dagster integration with the Perian sky platform for serverless GPUs
https://github.com/Perian-io/dagster-perian

[sqkkyzx]

I've developed an integration with Dingtalk for my project to facilitate the convenient use of DingTalk API across multiple codebases.

dagster-dingtalk

I'm not affiliated with the Dingtalk official team, and I anticipate maintaining it for at least two years. Since Dingtalk is predominantly used by companies in China, I haven't written internationalized documentation. I'm unsure whether I should request to have it showcased in the documentation.

[cmpadden]

Very neat! I will add an entry to the integrations list at the very least, so that others can find it if they specifically search for Dingtalk.

Thanks @sqkkyzx !

Edit: PR has been created here #25878

[sqkkyzx]

I see that A guide for writing a resource based integration is coming soon!, and I am looking forward to it. I hope the documentation will guide me to write integrations more effectively.

[JasperHG90]

I'm developing an integration of dagster with pyiceberg! Currently in alpha as the library depends on unreleased changes in the PyIceberg library. End-to-end example here. I'm not affiliated with PyIceberg.

[cmpadden]

This is really cool, thanks for sharing @JasperHG90. I'm going to set aside some time to read through the demo to better understand things!

If you end up refactoring this into a standalone integration, we'd love to highlight it. We have the https://github.com/dagster-io/community-integrations repo if you're looking for a place to throw it with us managing builds/releases.

[cody-scott]

I maintain an integration for ingesting to MSSQL via their Bulk Copy Tool (BCP).

Works with pandas and polars, with polars using LazyFrames.

https://github.com/cody-scott/dagster-mssql-bcp

[cmpadden]

Very cool - I've included it in our integration listing here: #25986