Design considerations for Meltano plugin to Dagster

[blarghmatey]

I am in the early stage of building an integration between Dagster and Meltano. To make that possible I am working on adding a lib module to the core Meltano functionality that will be published as a standalone python library that I will then be wrapping with a Dagster plugin (https://gitlab.com/meltano/meltano/-/issues/2459).

I have some thoughts and questions about how best to approach the interface for the Dagster plugin which will also inform the design of the Meltano lib.

• Meltano leverages a backing database through SQLAlchemy and Alembic for tracking the state of extract and load operations. As part of building the plugin it would be useful to allow for hooking into those existing models and migrations to create the necessary tables as part of the default Dagster schema. Is this something that has been done by other plugins? Is it a viable option?
• The default way to manage a Meltano project is via their CLI and a meltano.yml file. There are some CLI elements that might be useful to use in the context of a Dagster project, such as their ability to 'discover' Singer taps and targets. Would it be viable to mount a subset of Meltano commands within the namespace of the Dagster CLI?
• What user experience elements should I be considering while building this integration? (e.g. exposing elements in Dagit, default metadata to expose, etc.)

[mgasner]

I can speak to the first two of these bullets and would reserve comment on the third for folks who use Meltano regularly.

• It's surprisingly difficult to "compose" Alembic migration trees, and you can see some of the consequences/pitfalls of trying to do so in the migrations associated with Dagster's SQL run and event log storages, where we support the case of these two storages targeting either different schemas/databases or the same schema/database. It isn't ideal and it's hard for me to imagine making it easily pluggable without a bunch of custom tooling (which I think would amount to writing something more flexible on top of some of the alembic core). I think that it will be much simpler to target a separate schema.

• One possible model for this is the dagster-celery CLI.