allow upstream dependencies in DBT assets that aren't sources #21077
mathew-mwi
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
If I create a dbt asset the only way I can link to an upstream process is via the sources meta reference but this doesn't work in all situations, for example I am using DBT athena and one of the tasks my model is doing is loading external sources via DBT and not another asset, this means that I don't have an upstream dependency. However, I still want my processes to run in a specific sequence, for example, if I re-run a partition I need to clear out old data from S3 as part of AWS Athena doesn't support overwriting data for unloads, so I need to have an asset that clears the location before my DBT model runs.
Another example is database set up processes, I may need to run scripts in my database in parallel, to set up a tonne of schemas, stages, objects, permissions, etc. this process is run sequentially in DBT but is far more efficient in an asset as I can run in parallel.
Please add in a feature to define upstream dependencies for DBT assets that are just upstream processes of the dbt model but not sources.
Beta Was this translation helpful? Give feedback.
All reactions