Help needed for each dbt_asset to use different dynamic partitions

[dagsir[bot]]

Hi folks,

I need to load csv files from different providers (sources) into an SQL database. I receive the csv files on a bucket on Azure, one bucket per source.

For each source I have 2 SQL tables, on source_import that is the raw data from the csv, and one source (where I compute some columns).
To do it, I have a dbt project that I would like to orchestrate with dagster.

I would like to use dagster dynamic partitioning, with one seperate DynamicPartitionsDefinition for each source and one partition for each csv file, so I can reprocess a particular csv file of a particular source whenever I need.

I have a sensor per source, each of them listening to a particular bucket on my lake. Each sensor fills a seperate DynamicPartitionsDefinition with the filename that was just pushed to the bucket they are each checking.

So I end up up with source1_partition that contains all the csv received from source1, and source2_partition for the csv received from my source2, and on. All good so far.

My issue is when I bring dbt in process, as I need each dbt_asset to use a different dynamic partitions. dbt_asset_source1 needs to check source1_partition, dbt_asset_source2 needs to check source2_parition.

So far I have been using @dbt_assets decorator to load all the dbt_assets at once, but it regroups all the dbt models, and I can not set the dynamic partitions at dbt_asset level, I can only set if for all the dbt models.

I ended up trying #13053 and #17722 but without success. My dbt_assets are not showing the right dynamic partition in the user interface.

---

The question was originally asked in Dagster Slack.