Add executed_with_root_target condition to handle partial runs / failures

[OwenKephart]

Summary & Motivation

Resolves: #24389

To put it simply, this ensures the any_deps_updated condition does not "count" an update from a parent if the run that materialized that parent also planned to materialize the child.

This guards against a variety of situations where a run can be launched with the intention of updating many assets, but fail to do so for whatever reason (intentional skip, failure event). In those cases, the assumption is that the child should NOT be attempted again (even though the parent did successfully materialize).

This is a fairly niche implementation detail that users should not have to think about when crafting their own policies.

How I Tested These Changes

I added a parameterized test that failed before this change (for the skip and fail cases) and passes now.

Changelog

Fixed an issue with AutomationCondition.eager() that could cause it to attempt to launch a second attempt of an asset in cases where it was skipped or failed during a run where one of its parents successfully materialized.

[OwenKephart]

• [1.9] Replace Auto-materialize references in the UI with Automation condition #25414
• [1.9] Update evaluation id logic to be in sync with tick ids #25269
• [1.9] Add automation_condition to observable_source_asset #25344
• [1.9] Create default_automation_condition_sensor in Definitions object #25201
• [1.9] Update deprecation warnings for automation_condition, auto_materialize_policy, and freshness_policy #25208
• [1.9] Remove experimental warnings from AutomationCondition static constructors #25207
• [1.9] Promote hidden params to top-level args in AutomationConditionSensorDefinition #25206
• [1.9] Update default value of allow_backfills to True #25204
• Add executed_with_root_target condition to handle partial runs / failures #25253 👈
• Add error for user code sensors with too many entities targeted #25215
• Add test demonstrating that we do not hit the RESOURCE_EXHAUSTED error above a given threshold #25214
• Propagate automation condition information through the UI for user code conditions #25211
• Export AutomationContext and AutomationResult #25210
• Store allow_backfills in the sensor metadata #25203
• Create backfill_in_progress AutomationCondition #25190
• [refactor] Move all operands into a single file #25187
• Remove description property, make names more descriptive #25186
• Simplify return types of static constructors for AutomationCondition #25184
• Add all_deps_blocking_checks_passed condition #25171
• Make existing conditions work with Asset Checks #25170
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @OwenKephart and the rest of your teammates on Graphite

[schrockn]

Are there performance implications of this given that it is in the eager condition and ends up invoking expensively_compute_partition_keys? Are we saved by latest_time_window here?

[OwenKephart]

We're actually saved by the fact that this is and'd together with the newly_updated condition. So we only check this on the exact tick that a parent updates, which pretty severely caps the number of total partitions we might be dealing with here

responded

[OwenKephart]

Merge activity

• Oct 22, 12:50 PM EDT: A user started a stack merge that includes this pull request via Graphite.
• Oct 22, 1:01 PM EDT: Graphite rebased this pull request as part of a merge.
• Oct 22, 1:02 PM EDT: A user merged this pull request with Graphite.