migrate dagster-ssh ssh_resource to Pydantic config

[dagsir[bot]]

How do I migrate the dagster-ssh integration's SSHResource to use Pydantic config?

I can't get past these errors when configureing the SSHResource with ssh_resource.configured({"username": ..., "password": ...})
Using os.getenv() to retrieve SSH connection secrets:

dagster._core.errors.DagsterInvalidConfigError: Error in config for resource compliance_reports_ssh
    Error 1: Value at path root:config:password must not be None. Expected "(String | { env: String })"
    Error 2: Value at path root:config:username must not be None. Expected "(String | { env: String })"
  File "/usr/local/lib/python3.9/site-packages/dagster/_grpc/impl.py", line 138, in core_execute_run
    yield from execute_run_iterator(
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/execution/api.py", line 861, in __iter__
    yield from self.execution_context_manager.prepare_context()
  File "/usr/local/lib/python3.9/site-packages/dagster/_utils/__init__.py", line 489, in generate_setup_events
    obj = next(self.generator)
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/execution/context_creation_job.py", line 291, in orchestration_context_event_generator
    context_creation_data = create_context_creation_data(
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/execution/context_creation_job.py", line 106, in create_context_creation_data
    resolved_run_config = ResolvedRunConfig.build(job_def, run_config)
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/system_config/objects.py", line 178, in build
    config_mapped_resource_configs = config_map_resources(resource_defs, resource_configs)
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/system_config/objects.py", line 251, in config_map_resources
    raise DagsterInvalidConfigError(

Using EnvVar() to retrieve SSH connection secrets:

dagster._core.errors.DagsterInvalidConfigError: Error in config for resource compliance_reports_ssh
    Error 1: Invalid use of environment variable wrapper. Value "COMPLIANCE_REPORTS_SSH_PASSWORD" is wrapped with EnvVar(), which is reserved for passing to structured pydantic config objects only. To provide an environment variable to a run config dictionary, replace EnvVar("COMPLIANCE_REPORTS_SSH_PASSWORD") with {'env': 'COMPLIANCE_REPORTS_SSH_PASSWORD'}, or pass a structured RunConfig object.
    Error 2: Invalid use of environment variable wrapper. Value "COMPLIANCE_REPORTS_SSH_USERNAME" is wrapped with EnvVar(), which is reserved for passing to structured pydantic config objects only. To provide an environment variable to a run config dictionary, replace EnvVar("COMPLIANCE_REPORTS_SSH_USERNAME") with {'env': 'COMPLIANCE_REPORTS_SSH_USERNAME'}, or pass a structured RunConfig object.
  File "/usr/local/lib/python3.9/site-packages/dagster/_grpc/impl.py", line 511, in get_external_execution_plan_snapshot
    create_execution_plan(
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/execution/api.py", line 723, in create_execution_plan
    resolved_run_config = ResolvedRunConfig.build(job_def, run_config)
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/system_config/objects.py", line 178, in build
    config_mapped_resource_configs = config_map_resources(resource_defs, resource_configs)
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/system_config/objects.py", line 251, in config_map_resources
    raise DagsterInvalidConfigError(

---

The question was originally asked in Dagster Slack.

[ldnicolasmay]

Using {"env": "PASSWORD_ENV_VAR"} results in a similar-but-different error:

dagster._core.errors.DagsterInvalidConfigError: Error in config for resource compliance_reports_ssh
    Error 1: Post processing at path root:config:username of original value {'env': 'COMPLIANCE_REPORTS_SSH_USERNAME'} failed:
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "COMPLIANCE_REPORTS_SSH_USERNAME" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/usr/local/lib/python3.9/site-packages/dagster/_config/post_process.py", line 79, in _post_process
    new_value = context.config_type.post_process(config_value)
  File "/usr/local/lib/python3.9/site-packages/dagster/_config/source.py", line 40, in post_process
    return str(_ensure_env_variable(cfg))
  File "/usr/local/lib/python3.9/site-packages/dagster/_config/source.py", line 16, in _ensure_env_variable
    raise PostProcessingError(

    Error 2: Post processing at path root:config:password of original value {'env': 'COMPLIANCE_REPORTS_SSH_PASSWORD'} failed:
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "COMPLIANCE_REPORTS_SSH_PASSWORD" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/usr/local/lib/python3.9/site-packages/dagster/_config/post_process.py", line 79, in _post_process
    new_value = context.config_type.post_process(config_value)
  File "/usr/local/lib/python3.9/site-packages/dagster/_config/source.py", line 40, in post_process
    return str(_ensure_env_variable(cfg))
  File "/usr/local/lib/python3.9/site-packages/dagster/_config/source.py", line 16, in _ensure_env_variable
    raise PostProcessingError(

  File "/usr/local/lib/python3.9/site-packages/dagster/_grpc/impl.py", line 138, in core_execute_run
    yield from execute_run_iterator(
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/execution/api.py", line 861, in __iter__
    yield from self.execution_context_manager.prepare_context()
  File "/usr/local/lib/python3.9/site-packages/dagster/_utils/__init__.py", line 489, in generate_setup_events
    obj = next(self.generator)
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/execution/context_creation_job.py", line 291, in orchestration_context_event_generator
    context_creation_data = create_context_creation_data(
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/execution/context_creation_job.py", line 106, in create_context_creation_data
    resolved_run_config = ResolvedRunConfig.build(job_def, run_config)
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/system_config/objects.py", line 178, in build
    config_mapped_resource_configs = config_map_resources(resource_defs, resource_configs)
  File "/usr/local/lib/python3.9/site-packages/dagster/_core/system_config/objects.py", line 251, in config_map_resources
    raise DagsterInvalidConfigError(

[jamiedemaria]

We have a PR open for this that got stuck in review limbo #15180 I'll try to get it across the line today

[ldnicolasmay]

That's great... Thx @jamiedemaria !

[jamiedemaria]

merged #15180 in - it'll go out in the next release