Secrets in user code deployment pods should be available in run coordinator + step jobs

[pietercusters]

Very similar for env vars (see #3102) I think it makes sense to make the secrets in the user code deployments (set through envSecrets) available in the pods of the jobs.

Currently we are working around this by passing all the secrets to k8sRunLauncher.envSecrets which means all these secrets will be set as env vars of the pods of every single job. Furthermore, if we add a secret we need to do a helm upgrade of the dagster deployment. Both are obviously not desirable.

Maybe I'm missing something here, let me know! If not it's a feature request ;)

[johannkm]

Yep, that's something we want to improve! Currently have a diff here #7203

[pietercusters]

Great!

[gibsondan]

This will be possible in 0.14.10 later this week and the default behavior starting in 0.15.0 - there's an includeConfigInLaunchedRuns param that can be set on the user code deployment that will cause the secrets to be passed through automatically to the launched run pods.

[geoHeil]

Is something similar possible for the docker executor?

[gibsondan]

@geoHeil Yes this is possible in docker, but it's a bit more verbose because there's no helm to template the config into both places for us - so you have to specify the config in two different places. For example, to set the env var FOO=bar on both the user code server and any runs that it launches you can do this (note the DAGSTER_CONTAINER_CONTEXT env var):

  docker_example_user_code:
    build:
      context: .
      dockerfile: ./Dockerfile_user_code
    container_name: docker_example_user_code
    image: docker_example_user_code_image
    restart: always
    environment:
      FOO: "bar"
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
      DAGSTER_CURRENT_IMAGE: "docker_example_user_code_image"
      DAGSTER_CONTAINER_CONTEXT: '{"docker": {"env_vars": ["FOO=bar"]}}'
    networks:
      - docker_example_network

[geoHeil]

ah - yes I am already using this method. But the previously mentioned one sounded nicer/ less verbose.