Within the same job, I would like to be able to run the next op after I receive a webhook response or a message in my queue

[biancarosa]

This is the rough sketch of what we do today.

The thing is, when running in cloud-based environment, this architecture is leading to us spending more money than necessary on compute that is basically idle waiting for external services responses.

I was wonder if we could do something like this:

For this approach to work, we'd likely need:

• The ability to stop/pause the job execution after Op 1.
• Dagsterdaemon listening through this callback queue through a sensor or something & re-starting the job, passing the data needed for Op 2 that needs to execute next.

Does this make sense? Does it make sense to analyze if this is the kind of feature that Dagster wants/needs support in Dagster in the future? Or has anyone solved that problem differently?

[garethbrickman]

Hey @biancarosa, at this time there is no way to pause/stop a job while awaiting the outcome of an external process and then resume it. A path of least resistance would require splitting each op into 2 jobs: this way Op1 could finish and then you could use a sensor to poll for when the external process has completed via an External Asset and only then trigger Op2 to run in its own job.

If it's not feasible to use 2 jobs, then you may be able to fine tune the compute resources, depending on your deployment environment, so as to compartmentalize the pod that runs each op.

If using Kubernetes this would be achievable now using the k8s_job_executor (more information in the documentation here).

If you're using ECS this capability is unfortunately not yet available but has been requested here: #9671