How can I reload definitions within the Dagster UI without needing to redeploy my code?

[gibsondan]

Sometimes, Dagster definitions are sourced from some place other than just code (for example, a mounted volume with a YAML file) and it's natural to want to be able to reload your definitions when that external datasource changes even though the code hasn't changed. How can you set up your Dagster deployment so that you can press "Reload" in the Dagster UI and reload your definitions?

[gibsondan]

Dagster uses code servers to isolate user code from Dagster system code. When running dagster dev, Dagster typically manages your code servers for you as subprocesses, and pressing the "Reload" button on the Code Locations tab in the Dagster UI will start a new code server and reload your code.

However, many production deployments of Dagster (e.g. deploying in Docker, ECS, or Kubernetes using our Helm chart) recommend running code servers in separate containers for more isolation, using the dagster api grpc command. The only way to reload your definitions when runningdagster api grpc separately is to restart the process, which typically requires restarting the container or pod on which it is running. This means that pressing the "Reload" button in the Dagster UI does not actually reload your code if you are using the dagster api grpc command.

However, in the dagster 1.3.6 release we added an experimental dagster code-server start command that can be used instead of dagster api grpc and has identical arguments. This new command runs user code in a subprocess and proxies API calls from Dagster system components to that subprocess. When running this command, pressing "Reload" from the Code Locations tab in the Dagster UI will restart the subprocess, allowing you to reload your definitions without needing to restart the whole server.

If you're using the Dagster Helm chart, you can use this command by replacing dagsterApiGrpcArgs with codeServerArgs in the user code deployments section of the Helm chart. No other changes are required.

Note: Reloading the code from the Dagster UI makes a synchronous request to the server that waits until the code has reloaded. If your code takes a long time to load, this query can time out. The default timeout is set at 3 minutes, but can be increased by setting this value in your dagster.yaml file:

code_servers:
  reload_timeout: 600 # Value in seconds to wait before timing out

or in your Helm chart:

additionalInstanceConfig:
  code_servers:
    reload_timeout: 600 # Value in seconds to wait

If the request is timing out at the network level (e.g. returning a 502 error) even after increasing the Dagster time out, you may need to increase the timeout being used by the ingress into your dagster webserver (or refresh the page in the browser once the request has finished on the Dagster webserver)

[skwon615]

you may need to increase the (or refresh the page in the browser once the request has finished on the Dagster webserver)

Hi @gibsondan - sorry to bump an old thread! But in the above quote, what were you suggesting that we increase? (Looks like that last key word before the parentheses is missing 😓)

[gibsondan]

sorry about that, updated - I meant to say that if the timeout is happening at a layer above dagster (e.g. a load balancer like nginx) you might need to increase timeout settings there

[skwon615]

Awesome, thank you! :)

[v1gnesh]

@gibsondan -
1 - What happens if I run the parts (dagster-daemon, dagster-web, dagster-web-ro, dagster-code-server) as systemd services. Will the 'reload' button be able to restart the systemd service for dagster-code-server?
2 - Do I need define code_server in workspace.yaml, in place of python_module or python_file. If so, what are the allowed parameters?

[C-Loftus]

Is there any way to get hot reloading with Dagster inside a container so you don't need to rebuild? Or is that a bad practice to even want?

Either code-server start is not working for me or perhaps I am misunderstanding its use case.

• if I mount my code as a volume in the user code container, it doesn't get updated if I reload definitions
  • It seems like the metadata about the assets used to create the DAG does get updates, but if I materialize an asset, that code remains old.
• My entrypoint is dagster code-server start -h 0.0.0.0 -p 4000 --python-file /opt/dagster/app/code/main.py -d /opt/dagster/app/code
• All my code shows up, I am just unclear if there is a better way improve dev iterations.
  • It is a bit slow to make changes and then need to rebuild the user code container every time.

If this is intended behavior, I was curious if anyone has written help scripts to watch the container for updates and auto rebuild.