Upstream dependencies for Automaterialization

[matthias-Q]

Hi,
I have a question regarding AssetDeps. Currently, we are using them extensively, but we explicitly do not set some upstream dependencies because they are not as important as others. Imagine a mapping table that rarely changes. A downstream asset needs this table, but it is not required to be updated everyday.
How do I set the Automaterialization rules for that? Is there a way to model the dependency and explicitly ignore certain upstream tables?

This is similary to 'Auto-materialize even if some parents are missing' from the docs.

Cheers,
Matt

[OwenKephart]

Hi @matthias-Q , this is now possible using Declarative Automation: https://docs.dagster.io/concepts/automation/declarative-automation/customizing-automation-conditions#targeting-dependencies

The simplest approach at the moment would be to start with copying the implementation of AutomationCondition.on_cron(), as it seems like you want a daily cadence for your downstream asset. The behavior of this condition results in a run being launched for the downstream asset once per day, after all of its parents have been updated during that day. This is problematic, as you don't care about updates for one of those upstreams.

To avoid this, you can tack a .ignore(AssetSelection.keys(<tables you want to ignore>)) to the all_deps_updated_since_cron bit. This will allow the condition to avoid waiting for those tables to refresh. This would look like:

            ...
            all_deps_updated_since_cron = AutomationCondition.all_deps_match(
                AutomationCondition.newly_updated().since(cron_tick_passed)
                | AutomationCondition.will_be_requested()
            ).ignore(AssetSelection.keys(...))
            ...

[matthias-Q]

Sounds good. Will take a look next week.

[matthias-Q]

@OwenKephart

Thanks this change is awesome. However I have some questions:

1. Do I understand correctly that the AutomationConditions and evaluated upstream? So my first Asset is lazy I have some lazy assets in the between and the last assets (depending on all lazy upstream assets) has the AutomationCondition.on_cron('0 1 * * *'). Does that mean that the first asset in that graph will start at 1 ?

2. I have an Asset that requires some special tags (e.g. ecs/memory: 16384). If I use AutomationCondition.on_cron does the automatically triggered job use the tags defined in the AssetDefinition (tags_by_output_name`) Otherwise I still have to specify a job+schedule to set these configurations

3. We have a large legacy job (Java Code that creates multiple files and takes forever). To represents these files as assets, the job yields AssetMaterialization events and the assets are defined as dummy assets. In the past this was a suggested workaround because there would not be an AssetDefinition otherwise. If point 1 is true, how can I Automate assets that are downstream from these? These dummy assets do not have a on_cron tick, they are just refreshed.

[OwenKephart]

Hi!

(1) That's correct yep
(2) The tags defined on an AssetsDefinition are independent of the tags that get applied to runs targeting that asset. One solution would be to isolate that asset into its own AutomationConditionSensorDefinition and set the run_tags there.
(3) on_cron will wait for updates of parent assets, regardless of how that update is accomplished. This means that if you have some alternative process which generates AssetMaterialization events, this will be handled the same way by downstream assets as if these were traditional AssetsDefinitions.

[matthias-Q]

Hi,
1 and 3 are clear (the process that triggers the java job also emit AssetMaterialization events).

For 2. I hope that there will be some improvement there. I think the ecs tags should be added on the asset level. I will most likely end up with a lot of AutomationConditionSensorDefinitions. That does not sound very convenient.

[OwenKephart]

Very reasonable re: 2 -- In some ways, I view this as more of an improvement required in the land of executor definitions (i.e. executors that respect the tags on assets) rather than a Declarative Automation-specific feature