Missing DAGSTER_PG_PASSWORD in scheduled task

[mausch]

Hi, I have Dagster running with a separate code deployment on kubernetes, using the k8sRunLauncher. I can materialise assets just fine.
However when I try to schedule a job I get:

dagster._core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'postgres_url': Field(<dagster._config.source.StringSourceType object at 0x7ffff560be90>, default=@, is_required=False), 'postgres_db': Field(<dagster._config.field_utils.Shape object at 0x7fffd8be63d0>, default=@, is_required=False), 'should_autocreate_tables': Field(<dagster._config.config_type.Bool object at 0x7ffff55c8c10>, default=True, is_required=False)}.
    Error 1: Post processing at path root:postgres_db:password of original value {'env': 'DAGSTER_PG_PASSWORD'} failed:
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_PG_PASSWORD" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_config/post_process.py", line 79, in _post_process
    new_value = context.config_type.post_process(config_value)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_config/source.py", line 40, in post_process
    return str(_ensure_env_variable(cfg))
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_config/source.py", line 16, in _ensure_env_variable
    raise PostProcessingError(

  File "/myn/.venv/lib/python3.11/site-packages/dagster/_grpc/impl.py", line 350, in get_external_schedule_execution
    return schedule_def.evaluate_tick(schedule_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_core/definitions/schedule_definition.py", line 903, in evaluate_tick
    CachingDynamicPartitionsLoader(context.instance) if context.instance_ref else None
                                   ^^^^^^^^^^^^^^^^
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_core/definitions/schedule_definition.py", line 310, in instance
    DagsterInstance.from_ref(self._instance_ref)
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_core/instance/__init__.py", line 635, in from_ref
    unified_storage = instance_ref.storage
                      ^^^^^^^^^^^^^^^^^^^^
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_core/instance/ref.py", line 499, in storage
    return self.storage_data.rehydrate(as_type=DagsterStorage) if self.storage_data else None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_serdes/config_class.py", line 129, in rehydrate
    return klass.from_config_value(self, check.not_none(result.value))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_core/storage/legacy_storage.py", line 120, in from_config_value
    ).rehydrate(as_type=RunStorage)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/myn/.venv/lib/python3.11/site-packages/dagster/_serdes/config_class.py", line 124, in rehydrate
    raise DagsterInvalidConfigError(

No idea where DAGSTER_PG_PASSWORD is missing? I can see that's already set for the webserver , the daemon, and the k8sRunLauncher sets it on k8s jobs too. Does it also have to be set on my code deployment too for some reason? Why?

Many thanks.

[garethbrickman]

Are you setting DAGSTER_PG_PASSWORD in your Kubernetes deployment itself? You can check the environment variables in your pod by running:

kubectl exec -it <your-pod-name> -- env | grep DAGSTER_PG_PASSWORD

[mausch]

I'm not using DAGSTER_PG_PASSWORD anywhere in my user code, which is why this is so puzzling

[garethbrickman]

Yeah I'm unsure as well. Are you setting that in values.yml, like in this line?

[mausch]

I'm setting postgresql.* in the helm chart, yes. I'm not explicitly setting or consuming DAGSTER_PG_PASSWORD anywhere in my code though.

[gibsondan]

Hi @mausch - the env var needs to be set on your "user code deployment" pod as well. The helm chart should be setting it for you automatically in the environment variables for the pod - here's the Helm YAML where it does that: https://github.com/dagster-io/dagster/blob/master/helm/dagster/charts/dagster-user-deployments/templates/deployment-user.yaml#L57-L61

If you describe the pod is that section missing?

[mausch]

Thanks, does that mean I need to use that helm chart to deploy all my user code?
I thought I could deploy my code any way I wanted (we're really trying to stay away from helm as much as possible)?