Passing large number of secrets/env variables to a docker executor run

[cody-scott]

Hi, looking for guidance on how to pass a growing number of variables to a run launched with:

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      ...

Running Dagster 1.7 in OSS using docker.

We have a growing number of environment variables to pull from for a variety of items, but we're running up to a problem where whenever we change the dagster.yaml to add these new variables to the run_launcher, we need to re-launch the daemon and ui container to capture these changes. If i recall these secrets are passed from the daemon to the container that is launched, so the environment variables need to be accessible on that container, and up to date, which leans better towards an external secrets provider.

We have also tried swapping out the multiple variables with a single variable which connects to KeyVault, where the other ones sit. Our problem in this case is if we try to use the EnvVar approach, where it reads from the environment variables, we need to load all the secrets into the os.environ before we proceed, which is causing some timeouts if it takes a long time to read all the secrets. It also slows down the launch of containers as each launch needs to go and pull all the secrets, load it in, then start the run, even if it only needs 1 secret.

Am I missing the right way to inject secrets from an external place (keyvault, aws, etc.) using EnvVar, or is this approach to use the env var not valid in this case?

We discussed moving the logic to pull the secret into the resources, but it feels like its creating a really tight link between the secrets provider and the resource, instead of the EnvVar approach where the resource pulls in the credentials as needed and is a more generalized.

I see a few options:

1. Define every single environment variable when launching the container from dagster. When we update the variables we would need to relaunch all the containers, instead of just the user code container
2. Incorporate external secret provider and pass only that variable in, pulling the others on container launch (slows as noted above).
3. Have a single JSON object stored on the secrets provider, effectively mirroring a .env file, to speed up the load time

Any suggestions on a preferred way forward?

[cody-scott]

Is the simple answer here that i should be defining every single variable as environment variables, then restarting all the containers whenever it changes?

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - VAR1
      - VAR2
      - VAR3

certainly isn't as nice as just defining a single KeyVault token and connecting/pulling with that.

[cody-scott]

For closing this out we ended up doing the easy way of creating our own run launcher, subclassing the DockerRunLauncher. It basically just passes all but a few of the daemons environment variables over to the launched container. Has tradeoffs, but works great for us.

As a suggestion, if there was a config flag to pass_all_env_vars this is basically what we did, so an option to do this would help.

[cody-scott]

I would add, if there is a better method (for those running in a simple compose + single machine) please share!

[lfagliano]

Hi Cody! By any chance, do you have a sharable example from your custom launcher? Considering doing the same in a similar project.

[mjf-89]

Even though this discussion is marked as closed, I wanted to share my experience because I frequently struggle with environment variable (EnvVar) resolution when using the DockerRunLauncher. I believe this topic isn't well-documented, and it could also be useful to @lfagliano, who raised a related question last week.

In my view, placing all user-code-related variables in the dagster.yaml file is not the ideal solution. Instead, the environment variables defined in the dagster.yaml launcher configuration should be limited to instance-wide settings that rarely change, such as DAGSTER_POSTGRES_USER and DAGSTER_POSTGRES_PASSWORD. All other variables should be handled in the user-code GRPC server.

The undocumented part is that you can specify a container-context within the user-code GRPC server, which configures any containers that the server creates to run the code.

You can set this up using either the --container-context CLI argument or the equivalent DAGSTER_CLI_API_GRPC_CONTAINER_CONTEXT environment variable. The value for this option is a JSON string. While the schema isn't documented, you can view it here. In particular, you can inject environment variables into the created containers by specifying a JSON structure like {"env_vars": ["VAR_NAME1=VAR_VALUE1","VAR_NAME2=VAR_VALUE2",...]}.

A method that has worked well for me is to standardize the names of environment variables relevant to Dagster pipelines with a generic prefix like USER_CODE_*. Then, in the GRPC container, I use an entry point like this:

env_vars=$(env | grep 'USER_CODE_' | xargs -I{} echo \"{}\" | paste -sd, - )
dagster api grpc -p 4000 -h 0.0.0.0 -m $USER_CODE_MODULE --container-context="{\"env_vars\": [$env_vars]}"

This approach ensures that all USER_CODE_* environment variables exported in the GRPC server are also available in the containers created by the run launcher. The advantage is that you only need to restart the user-code GRPC server whenever you need to modify or add environment variables to the configurable objects of a pipeline, simplifying the process considerably.

I would also suggest to take a look at the helm chart of the k8s deployment option since also the k8s run launcher has the same issue and within the helm chart is resolved setting the container-context via the env variable DAGSTER_CLI_API_GRPC_CONTAINER_CONTEXT.