DbtProject doesn't take profiles_dir and dbt_project.prepare_if_dev fails if DBT_PROFILES_DIR set in environment

[themightychris]

What's the issue?

If DBT_PROFILES_DIR is set in your environment to a relative path in order to use a shared versioned one within your repository that uses env_var to avoid sharing secrets when you run dbt commands directly, then dbt_project.prepare_if_dev will fail.

DbtCliResource takes profiles_dir which you can set to a resolved absolute path when you declare the DbtCliResource resource that everything else uses, but dbt_project.prepare_if_dev instantiates its own DbtCliResource and only passes project_dir to it. This lets you override DBT_PROJECT_DIR in the Dagster context but not DBT_PROFILES_DIR, so the raw DBT_PROFILES_DIR from your environment gets picked up and can't be overridden for DbtProject's usage of DbtCliResource like it can for the rest of your project using your own configured DbtCliResource resource.

DbtCliResource.run changes directory into the dbt project directory before running with all other paths converted to absolute, but then DBT_PROFILES_DIR being set to something relative breaks dbt parse

What did you expect to happen?

DbtProject should be configurable with a resolved profiles_dir and passed to DbtCliResource when invoked

Maybe that's not the intended use of DbtProject as it's documented as only designed to capture information about the project itself and profiles could be considered not inherent to the project. But DbtProject making use of DbtCliResource and not being able to fully configure those uses like you can elsewhere seems problematic. I'm not sure what the best fix would be but I'm stuck with no path to making DbtProject's CLI calls work when DBT_PROFILES_DIR is set to something relative in my environment that works for running dbt directly

How to reproduce?

No response

Dagster version

1.9.5

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
By submitting this issue, you agree to follow Dagster's Code of Conduct.

[menzenski]

I believe I've run into the same issue when trying to add the dagster-dbt integration to an existing dbt project, which defines the profiles.yml in a nonstandard place.

The dagster-dbt project scaffold command can't find the profiles file, even though the DBT_PROFILES_DIR environment variable is set (the dbt project can find it fine).

$ DBT_PROFILES_DIR=/Users/matt/dev/my_dbt_project/profiles/ poetry run dagster-dbt project scaffold --project-name my_dbt_project --dbt-project-dir ./src
Running with dagster-dbt version: 0.25.6.
Initializing Dagster project my_dbt_project in the current working directory for dbt project
directory /Users/matt/dev/my_dbt_project/src.
A profiles.yml file was not found in either /Users/matt/dev/my_dbt_project/src or
/Users/matt/.dbt. Please ensure that a valid profiles.yml exists in your environment.

If I copy the profiles.yml file into the dbt project directory src/, then it works (but I don't want profiles.yml to live here):

$ poetry run dagster-dbt project scaffold --project-name my_dbt_project --dbt-project-dir ./src
Running with dagster-dbt version: 0.25.6.
Initializing Dagster project my_dbt_project in the current working directory for dbt project
directory /Users/matt/dev/my_dbt_project/src.
Using profiles.yml found in /Users/matt/dev/my_dbt_project/src/profiles.yml.
Your Dagster project has been initialized. To view your dbt project in Dagster, run the
following commands:

 cd '/Users/matt/dev/my_dbt_project/my_dbt_project'
 dagster dev

[maximearmstrong]

A fix has been merged and will be included in the next release, see PR #27416.