Recovering from Dagster-Daemon or Host machine crash - What is the expected behavior?

[nbuck1234]

My primary question: Is there a way for the dagster daemon to recover from being restarted during a job run? Is the description below the expected behavior?

My setup:

• Dedicated AWS Windows 10 host for Dagster-Webserver, Daemon, and Postgresql Database
• Ops are generally remote execution via subprocess - > Powershell or other python library. ( I verified the same observed behavior with a local python process)

My observations relating to dagster services and in-progress runs:

• Webserver: Restarting the webserver at any time does not affect the outcome of the run.

• Database: If the PostgreSQL database service is restarted before or within a short time of an in process run completes, dagster picks it up and the run will show up complete in the web ui. If the database comes back up after a few minutes of the run actually completed, you will end up seeing an error in ui that the run result couldn't be written to the database.

• Daemon: This is the big one. If the daemon goes down during a run, regardless of whether it is restarted before the run should have completed, all sense of state for the run is gone. You can do whatever you want, but the run will still show in progress in ui, and it must be force terminated. I would love to find a way to restart the daemon and have some amount of recovery and/or graceful exits on in process jobs.

I hope this post is helpful to the community, cheers.
Nick

[nbuck1234]

Just adding a little bump here to try to get some eyeballs.

[gibsondan]

Hi @nbuck1234 - our recommendation for how to make runs more resilient to daemon failures is to use one of our run launchers that launches runs in an isolated container.

The one that works in a single host is the DockerRunLauncher: https://docs.dagster.io/deployment/guides/docker#launching-runs-in-containers

and we also have integrestions with Kubernetes: https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm

and Amazon ECS: https://docs.dagster.io/deployment/guides/aws#launching-runs-in-ecs.

These take a bit more work to deploy, but give you much better isolation guarantees since runs happen in an entirely different container, rather than in a subprocess of the daemon process. Once the run has launched, the daemon can go down entirely and the run will continue unimpeded.

[garethbrickman]

"A deployment can have multiple simultaneous instances of dagster-webserver, but should include only a single dagster-daemon process." From Running the Dagster daemon documentation.

[stevenmurphy12]

Cool thanks Gareth. Do you know how we should handle cases like I described above, in which the single dagster-daemon process gets interrupted?

[garethbrickman]

Unfortunately I don't know of a redundancy for such occurrences

[nbuck1234]

I've encountered a similar issue:

• Submitted a backfill with 110 partitions, so 110 runs
• Approx 70 of those got submitted
• Dagster Daemon K8s pod got killed and restarted
• The 70 submitted jobs completed successfully
• The remaining 40 runs never get submitted.

I am correct in saying that we should only have one instance of the Dagster Daemon running per deployment?

@stevenmurphy12 When you brought dagster daemon back online - how about the partitions that were in-progress at the time the daemon went down? Did it keep track of what happened with those for you?

[stevenmurphy12]

It seemed to keep track of those yeah, it showed those partitions as complete. Whether that was fortunate timing, or if the backfill daemon catches up with the result of job runs I'm not sure. Would need to test it further.