Is there a Celery-specific RunLauncher (without k8s)?

[skeller88]

I'm not seeing a Celery RunLauncher, but I'm seeing a celery executor, which I guess you can launch with any of the run launchers? I'm confused as to why there's a CeleryK8sRunLauncher in the first place, because it seems like that run launcher uses the celery broker as the queue, and then k8s as the worker/executor. It seems more accurate to think of the Celery broker/backend as the RunCoordinator?

[gibsondan]

Using the CeleryK8sRunLauncher is required if you're using the celery_k8s_job_executor, which is discussed here and lets you use celery to maintain global concurrency limits across multiple runs: https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm-advanced#per-op-limits-in-kubernetes

I think if we were redoing the celery+k8s integration from scratch in 2023 we probably would not have a separate run launcher for this, and would instead just use the K8sRunLauncher there as well, as in practice they do the same thing (launch the run worker for a given run in a k8s pod)

[XBeg9]

Can CeleryK8sRunLauncher and celery_executor be used together? celery_k8s_job_executor with CeleryK8sRunLauncher works, but it spins up steps for each op... that's too granular.

Reported this as an issue #22027

[Radiergummi]

This is really confusing; I'm specifically not using K8s, and the docs just don't really say anything on that. Should I use the default run launcher? How do I do this? Does it need configuration? Nobody knows.