Backfill retries

[jamiedemaria]

Summary & Motivation

Enables reexecuting a backfill with either all partitions retried or only the failed partitions retried

Re-uses some of the graphene types for run re-execution. I could create different types that are backfill specific instead.

If reexecuting from failure:
For asset backfills, it will create a new backfill that will backfill the set of assets that were not successfully materialized in the first backfill. For job backfills, uses the existing fromFailure attribute that will retry a job backfill.

Constraints:

• the first backfill must be in a completed state before it can be retried
• for asset backfills, if reexecuting from failure some assets must have not been materialized in the first backfill. This differs from another action.

When a retried backfill is created we add the parent backfill id and the root backfill id as tags like we do for run retries.

How I Tested These Changes

new tests

[jamiedemaria]

• Backfill retries #23679 : 3 dependent PRs (#25137 , #25165 , #25167 ) 👈
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @jamiedemaria and the rest of your teammates on Graphite

[github-actions]

Deploy preview for dagit-core-storybook ready!

✅ Preview
https://dagit-core-storybook-n05gsv92q-elementl.vercel.app
https://jamie-backfill-retries.core-storybook.dagster-docs.io

Built with commit 58456f5.
This pull request is being automatically deployed with vercel-action

[jamiedemaria]

moving this back to draft since i'm going to shift focus to status and filtering (see discussion in planning doc about retries not being high priority for mvp)

[jamiedemaria]

i think it could make sense to make a backfill-specific version of GrapheneReexecutionParams that would take parentBackfillId and strategy where strategy can be FROM_FAILURE or ALL. i will make that update

[jamiedemaria]

might need to add a GrapheneBackfillNotFound output type. will look into

[clairelin135]

If backfills can only be retried from the UI hitting this seems unlikely?

[jamiedemaria]

yeah, the other backfill actions have this too, and i figured it doesn't hurt to have here

[jamiedemaria]

it'd be to make it more in line w the run re-execution types which have GrapheneRunNotFound as a potential return type. i dont think it's really that necessary here though

[jamiedemaria]

@prha @sryza @clairelin135 pinging for review for this one!

[clairelin135]

Implementation looks good to me!

Seems like this implementation will retry canceled partitions and failed partitions. Not sure if this is expected behavior, maybe we should communicate this in the UI somehow?

[clairelin135]

If backfills can only be retried from the UI hitting this seems unlikely?

[jamiedemaria]

Seems like this implementation will retry canceled partitions and failed partitions.

Yeah the idea is that you can take a canceled or failed backfill and retry anything that didn't work the first time. one example of an issue that has come up is k8s pods being evicted and causing a run in a backfill to fail. we've had users request the ability to retry the just partitions that failed/didn't run so that they dont have to manually make the backfill that targets just those partitions themselves