Why is auto-materialisation launching occasional duplicated runs

[bolinzzz]

Hi all! My team deployed Dagster to K8s and is using auto-materialisation. Lately we noticed that there are occasional duplicated runs. See the scrrenshot below of an asset from our production environment:

On 29 and 30 Nov, two auto-materialisations or runs happened right after one another. For other days, only one run was submitted.
Here's the auto-materialisation history

Just wondering what might be causing this? Could it be K8s evicting and moving dagster-run pods from one node to another when they just started? Please provide some insight and guidance!

[bolinzzz]

I think I know the reason. Looking at 30 Nov. The auto-materialisation first launched the run at 13:00, but the run did not start until 13:09. Attached a screenshot of the run events

I guess the auto-materialisation evaluation happens every 10 minutes, so at 13:10, the asset freshness policy is still violated thus another run is launched. For the second run, it is able to start right away

[OwenKephart]

Hi @bolinzzz -- in theory, the evaluation should take into account runs that have not started / are in the queue when deciding what to kick off, so this seems like a bug to me.

Do you have some more information on your setup? What freshness policies you have, the shape of your graph, etc.?

[bolinzzz]

Hi Owen, we are on Dagster 1.5.9. The auto-materialisation policy of the asset is
Lazy with the following freshness policy By 01:30 UTC, this asset should incorporate all data up to 1 day before that time.. We will be addingAutoMaterializeRule.materialize_on_cron.

A screenshot of the graph:

[tyler-jenkins]

Hello @OwenKephart and @bolinzzz !!

We are seeing the same thing with our freshness policies on 1.5.10.

We have attempted to customize our policies slightly to allow for missing parents (where we first saw the issue of multiple runs) and then changed it back to a straightforward eager and proceeded to also see the issue. The duplicated run in our case is usually launched a couple minutes after it was first seen. We have seen it with dynamically partitioned and normal assets

[cyberjar09]

we are on 1.7.8 and still seeing the issue

[garethbrickman]

As of 1.8.0, Auto-Materialization is replaced by Declarative Automation. More info in #22811

[bolinzzz]

AutoMaterializeRule.skip_on_run_in_progress() which has been added as part of 1.6.12 release, it is not part of AutoMaterializePolicy.eager() by default.

See this: #19214