Auto-materialization for dbt views using code version

[cbini]

I'm wondering if it would be possible to build auto-materialize rules so that a a dbt view asset will only update if it's code version has changed.

As it stands, with an "eager" policy, every view will update anytime an upstream asset updates. However, if the view code has not changed, it's actually unnecessary to run a materialization, as it's not producing any changes to the data model.

This results in Dagster doing a bunch of extra work and a busier server, which has real costs if your running it in a kubernetes cluster. It would be great if we could specify that view models should be passed over if there has been no code change when evaluated.

[sryza]

This is not currently possible, but here's the issue where we're tracking the ability to auto-materialize when there's a code change: #15686.

Once we add that auto-materialize rule, then you'll be able to get the behavior you're looking for by creating an auto-materialize policy without the parent_updated rule, but with the code_version_changed rule.

[cbini]

thanks @sryza! I tried this by applying an eager policy to dbt table models and eager without parent_updated to views. I'm noticing that this setup blocks table models that are downstream from view models from auto-materializing. Can you think of a way to make these assets skippable while allowing downstream assets to proceed?

[cbini]

does this look right btw? the asset says there's upstream data, which is accurate, but the auto-materializer says no to "Upstream data has changed since latest materialization"

[sryza]

I'm noticing that this setup blocks table models that are downstream from view models from auto-materializing. Can you think of a way to make these assets skippable while allowing downstream assets to proceed?

What's causing this is probably the skip_on_parent_outdated rule. Dagster thinks the view model is outdated, so won't materialize the downstream assets. You could remove that rule as well?

[OwenKephart]

This is now possible using the Declarative Automation. You can set the automation_condition for your dbt views to AutomationCondition.code_version_changed() to have them only materialize when a code version change is detected.

You can accomplish this by updating the get_automation_condition() method on your DagsterDbtTranslator to return this condition for dbt models which are views.

[ThomasRolfsnes-EP]

@OwenKephart Hi! I much appreciate the work that's been done here!

How does the this work with downstream assets?

For example with a DAG A -> B -> C, where A and C are regular dagster assets with eager AutomationCondition, and B is a dbt-view with code_version_changed.

We expect B and C to materialize on code-changes to B. And C (and not B) to materialize when A materializes. Is this the current behaviour?

edit: I found that this is work in progress, no worries (#23420)