Worker restart

[spapas]

Hello, I've deployed a procrastinate worker in prod using a systemd unit. It works fine. However I'd like to know if there's a way for me to reload the worker so it picks any changes to the tasks, for exampe be sending it a unix signal.

Right now I can restart it by restarting the systemd unix but I'd prefer a better solution. For example, we can sent it a HUP signal to gunicorn to reload: https://docs.gunicorn.org/en/stable/signals.html#reload-the-configuration

[ewjoachim]

but I'd prefer a better solution

Can you spare a few words on why this is not a good solution to you? What problem are you trying to solve?

[spapas]

Two main reasons:

1. systemd needs root access to restart, sending a signal does not. Yes, it is possible to deploy user-systemd units (i.e systemctl --user) but it's not common and I avoid it since other sysadmins may be confused.
2. running systemctl restart will delay the deploy process until the worker is restarted. If a long-running task is run then the deploy may take a lot of time (until the task is finished). Sending the singal would be immediate.

It's not a problem per-se, I was just asking if it was possible to reload the worker without the need to restart it.

[ewjoachim]

SIGHUP in Gunicorn may reload the config but only reloads the python modules in certain cases (when preload_app is not active). This is because the main process in Gunicorn (to my knowledge) is not the one that answers requests. Upon SIGHUP, it may stop the worker process(es) and start a new one.

Procrastinate (currently) runs everything in its own process, so if we want to restart, we can't stop a subprocess and start a new one: there's no subprocess to restart.

Reloading a Python module in place may be a bit complex, especially around the fact that old objects continue existing while new ones are added. In other works, if you use importlib.reload on a module that has a class following the Singleton design pattern, you'll now have 2 distinct singleton instances. To my knowledge, there is no proper way to reload a Python module inside a given process.

Then, the only solution I'd see would be to run os.execl(). I don't know if this is a good idea.

[spapas]

Hello @ewjoachim makes sense. I'll keep restarting the systemd unit :)

[ewjoachim]

Well, maybe the os.exec strategy would be worth exploring.

[ewjoachim]

Note: this is something you could do yourself by making your own script that calls app.run_worker() and have the script gracefully interrupt the worker on SIGHUP + run os.exec