Backfills: Automatic Recalculation and maximum window for single-run backfills

[raayu83]

Hello,

just looked at Dagster again after more than a year absence and like the direction it is going.

Single-Run Backfills is one of the most important features that is now available.

But I wonder if the support for Backfill policies for Time based partitions could be extended. Especially for the following two use cases which are probably pretty common:

• When querying reporting data from Third Party APIs, often the data changes during a certain time range. So there is a need to reprocess data that could have changed again at every scheduled run. Depending on the source this could be just the previous day or as much as a year. Would it be possible to add support to not only process new partitions, but also reprocess any partition up to n partitions in the past? (while still respecting backfill policies such as single time range policy)

• Additionally, sometimes APIs and other sources deliver a lot of data or take a lot of time and only a limited time range can be processed at once (due to API timeouts and memory limits). But processing every single day would also be overkill. Would it be possible add something that is a middle ground between processing every single day and processing the whole range? something like "process up to n partitions at a single run"

[raayu83]

Just noticed the second request is already implemented as max_partitions_per_run for the multi_run BackfillPolicy.

[raayu83]

Regarding the first request, it would be important that this is the default behaviour for a job, but when a manual is started only days selected should be materialized.

[sryza]

Would it be possible to add support to not only process new partitions, but also reprocess any partition up to n partitions in the past? (while still respecting backfill policies such as single time range policy)

When you launch a backfill, you can choose any range of partitions to target. It doesn't need to be all the partitions. Dos that answer your question?

[raayu83]

Hi Sandy,

sorry I, wasn’t clear enough.
What I want to have is a schedule that does not only materialize the asset up to the current partition, but also rematerializes the n previous partitions.

[sryza]

Ah, got it. I think this example might be what you're looking for: #14622.

[raayu83]

Exactly what I was looking for, thanks!
I wonder why this isn’t in the docs, would probably be useful for many.

[raayu83]

I have a few questions though:

• does this really work with dynamic date ranges like „the past 14 days“? Wouldn’t the schedule compute the values passed to the runrequest and only store the static result?
• what happens if the asset selection also includes unpartitioned assets?

[sryza]

does this really work with dynamic date ranges like „the past 14 days“? Wouldn’t the schedule compute the values passed to the runrequest and only store the static result?

You would need to compute the date range that you want to cover in the decorated schedule function. You can do that by starting with context.scheduled_execution_time or datetime.now().

what happens if the asset selection also includes unpartitioned assets?

That should work fine. Dagster will ignore the partitions when generating materialization events for them.

[raayu83]

Thank you very much for the clarification!