You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Issue:
Not able to back-fill downstream asset, using multi-run BackfillPolicy when Automaterialize is ON.
Details:
I am using dagster 1.7.9 with Auto-materialize.lazy rule.
I have a use case, there are two assets A and B. B is dependent on A. I have defined Multi-partition definition(which is same for asset A and B) such as ["x|21-10-2024", "b|20-10-2024....] where x,b are Dynamic partition and second one is Daily-partition.
I want to run back-fill such that each run will execute for 5 partitions. I used BackfillPolicy.multi_run(max_partitions_per_run=5) on @asset decorator.
It is working when I run the back-fill only for Asset A or Asset B.
But it is not working when I turn on the Auto-materialize feature.
When I trigger back-fill for Asset A with Auto-materialize feature turned ON, multi-run works for Asset A and after it completes, only single partition of Asset B is back-filled. I want multi-run policy to be applied to Asset B as well. So when Asset A gets completed, back-fill should be triggered for Asset B for the same partition range as of Asset A.
Could you please let me know how it can be achieved.
In short, I want downstream asset to be triggered for multiple partition once upstream asset completes its backfill.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Issue:
Not able to back-fill downstream asset, using multi-run BackfillPolicy when Automaterialize is ON.
Details:
I am using dagster 1.7.9 with Auto-materialize.lazy rule.
I have a use case, there are two assets A and B. B is dependent on A. I have defined Multi-partition definition(which is same for asset A and B) such as ["x|21-10-2024", "b|20-10-2024....] where x,b are Dynamic partition and second one is Daily-partition.
I want to run back-fill such that each run will execute for 5 partitions. I used BackfillPolicy.multi_run(max_partitions_per_run=5) on @asset decorator.
It is working when I run the back-fill only for Asset A or Asset B.
But it is not working when I turn on the Auto-materialize feature.
When I trigger back-fill for Asset A with Auto-materialize feature turned ON, multi-run works for Asset A and after it completes, only single partition of Asset B is back-filled. I want multi-run policy to be applied to Asset B as well. So when Asset A gets completed, back-fill should be triggered for Asset B for the same partition range as of Asset A.
Could you please let me know how it can be achieved.
In short, I want downstream asset to be triggered for multiple partition once upstream asset completes its backfill.
Beta Was this translation helpful? Give feedback.
All reactions