Automation Condition and AssetSpec weird behaviour

[ssillaots-boku]

Hi!
I have some AssetSpec assets which have downstream assets.

I'm trying to migrate from AutomaterializePolicy to AutomationCondition and have run into an issue.

Simplified example of the flow:

some_external_table = AssetSpec(key=AssetKey(['some_external_table']))


@asset(partitions_def=DailyPartitionsDefinition(datetime(2024, 9, 19)),
       deps=[some_external_table],
       automation_condition=AutomationCondition.eager())
def downstream() -> None:
    return

For some reason AutomationCondition expects AssetSpec to be materialized and thus downstream doesn't start whereas
AutomaterializePolicy starts a downstream run.

Downstream has 6 partitions and in the evaluation you can see that 6 partitions are missing which is really weird since some_external_table isn't partitioned. It seems like a bug

Picture of the evaluation:

Is there a workaround?

[ssillaots-boku]

As a workaround I could change AssetSpec assets to regular assets but that wouldn't fix dbt source assets by default. I would need to create additional assets to accommodate dbt source assets. A lot of changes...
Hopefully there's an easier way.

[alangenfeld]

Since you can report materializations for external assets
https://docs.dagster.io/concepts/assets/external-assets#updating-external-asset-metadata
the automation condition treats them like any other asset by default.

AutomaterializePolicy predated this evolution of external assets being able to have events reported for them, which is why it behaves as it does.

Modifying the automation condition is one way to deal with this problem, such as by removing the any_deps_missing condition

AutomationCondition.eager().without(~AutomationCondition.any_deps_missing())

or replacing it with a variant that ignores certain keys

AutomationCondition.eager().without(~AutomationCondition.any_deps_missing()) & ~(AutomationCondition.any_deps_missing().ignore(AssetSelection.keys(...)))

The 6 True is an effect of how we map between partition and non-partitioned assets while evaluating conditions.

[ssillaots-boku]

Thanks for answering. I've been off this topic for quite a while.

With Dagster assets I can easily solve this with number of ways. But perhaps you've got ideas how to solve it with dbt assets.

More specifically I'm talking about assets that are "source" dbt assets and thus converted to AssetSpecs in Dagster. Existing ones I could "materialize". But is there a way to automatically "materialize" new "source" assets that will be created in the future.
So an example flow is this:

1. someone creates new dbt model
2. dbt model uses/creates new source asset
3. source asset (which is an AssetSpec in Dagster) is in ideal flow automatically materialized when deployed

Basically I want to automate materialization of source assets/AssetSpec because right now it would be a manual process.
Or another way would be to make AutomationCondition .ignore somehow dynamic which ignores dbt source assets based on a tag. But then again I'm not sure how it's possible to add tags to dbt source assets.

Perhaps there's a whole another way to approach it.

It might be that the easiest way is to change all current AssetSpecs to actual assets and create future dbt source assets as regular assets as well.

[ssillaots-boku]

An additional attribute for AssetSpec would solve it quite easily but obviously I'm not sure how complicated would it be to implement in source code. Something where AutomationCondition will not take AssetSpec into consideration when it's marked so (a boolean value for example).

[ssillaots-boku]

Might have come up with an AutomationCondition that ignores Source Assets

AutomationCondition.eager().without(~AutomationCondition.any_deps_missing()) 
& ~AutomationCondition.any_deps_missing().allow(AssetSelection.assets().all()

Let me know if there's a case where this doesn't work "out of the box"