Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to reload code definitions without interrupting jobs? #51

Open
acarter24 opened this issue Dec 20, 2023 · 0 comments
Open

How to reload code definitions without interrupting jobs? #51

acarter24 opened this issue Dec 20, 2023 · 0 comments

Comments

@acarter24
Copy link

I have been very happily using the dagster extension with meltano integration now for at least 6 months, very stable. I am running it in a single docker instance on Azure ACS. Dagster also orchestrates my downstream DBT models, I use a dbt group with the same name as my meltano job and everything flows very nicely.

One issue I have discovered is that I cannot reload my code definitions without restarting the entire docker container. This interrupts any ongoing runs. So a single line change in a dbt model means a full tear down of the container,

I know with dagster it is possible to have your code locations separate to your main container, and just update your code, without interrupting running jobs. But is this workflow even possible using this extension, given that dagster sits 'under' the main meltano install?

I could definitely live with always having to do a full redeploy if my meltano.yml changes, as this mostly stays the same. But being able to do the 'reload definitions' for minor dbt tweaks would be a game changer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant