dbt run and dbt test with different number of dbt threads

[psimm]

Is it possible to change how dbt assets are built? By default the command executed is dbt build which uses the DBT_THREADS specified in the environment variable.

I would like to change that so assets are built with dbt run with a single thread and tested with dbt test using a higher number of threads. This is a workaround to limitations of the MotherDuck data warehouse, which supports parallel reads but only sequential writes.

I looked at DagsterDbtTranslator but none of the methods seem to change how assets are materialized.

DbtManifestAssetSelection has a method build which uses the DagsterDbtTranslator but I couldn't figure out if it can be modified.

Relevant docs and code:

• https://docs.getdbt.com/docs/running-a-dbt-project/using-threads
• https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator

• dagster/python_modules/libraries/dagster-dbt/dagster_dbt/dbt_manifest_asset_selection.py

  Line 21 in 9b9b968

  class DbtManifestAssetSelection(AssetSelection, arbitrary_types_allowed=True):

[nixent]

dbt build is combination of run, seed and test. You can invoke those commands separately instead of single build and specify number of threads you want, it will overwrite number of threads defined in the dbt profiles.yml. Alternatively you can create separate profiles and pass them to dbt.cli

@dbt_assets(manifest=dbt_project.manifest_path)
def jaffle_shop_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["run", "--threads", "1"], context=context).stream()
    yield from dbt.cli(["test", "--threads", "8"], context=context).stream()

[psimm]

Thank you! That works well and is a lot clearer than what I was looking at.