DbtProject doesn't take profiles_dir and dbt_project.prepare_if_dev fails if DBT_PROFILES_DIR set in environment

[themightychris]

What's the issue?

If DBT_PROFILES_DIR is set in your environment to a relative path in order to use a shared versioned one within your repository that uses env_var to avoid sharing secrets when you run dbt commands directly, then dbt_project.prepare_if_dev will fail.

DbtCliResource takes profiles_dir which you can set to a resolved absolute path when you declare the DbtCliResource resource that everything else uses, but dbt_project.prepare_if_dev instantiates its own DbtCliResource and only passes project_dir to it. This lets you override DBT_PROJECT_DIR in the Dagster context but not DBT_PROFILES_DIR, so the raw DBT_PROFILES_DIR from your environment gets picked up and can't be overridden for DbtProject's usage of DbtCliResource like it can for the rest of your project using your own configured DbtCliResource resource.

DbtCliResource.run changes directory into the dbt project directory before running with all other paths converted to absolute, but then DBT_PROFILES_DIR being set to something relative breaks dbt parse

What did you expect to happen?

DbtProject should be configurable with a resolved profiles_dir and passed to DbtCliResource when invoked

Maybe that's not the intended use of DbtProject as it's documented as only designed to capture information about the project itself and profiles could be considered not inherent to the project. But DbtProject making use of DbtCliResource and not being able to fully configure those uses like you can elsewhere seems problematic. I'm not sure what the best fix would be but I'm stuck with no path to making DbtProject's CLI calls work when DBT_PROFILES_DIR is set to something relative in my environment that works for running dbt directly

How to reproduce?

No response

Dagster version

1.9.5

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
By submitting this issue, you agree to follow Dagster's Code of Conduct.