Support multi-partitioning in AssetSlice #20353
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
schrockn
force-pushed
the
asset-graph-view-2
branch
from
90a0736 to
aa23d4e
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
986f0ed to
a530f5d
Compare
schrockn
force-pushed
the
asset-graph-view-2
branch
from
aa23d4e to
0c64516
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
a530f5d to
02f35a4
Compare
This was referenced Mar 8, 2024
schrockn
force-pushed
the
asset-graph-view-2
branch
from
0c64516 to
fb2c810
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
d3ecbbf to
14e6b97
Compare
smackesey
reviewed
Mar 8, 2024
python_modules/dagster/dagster/_core/asset_graph_view/asset_graph_view.py
Outdated
Show resolved
Hide resolved
|
|
||
| def _get_multi_dim_info(self) -> "MultiDimInfo": | ||
| check.inst(self._partitions_def, MultiPartitionsDefinition) | ||
| assert isinstance(self._partitions_def, MultiPartitionsDefinition) # appease pyright |
There was a problem hiding this comment.
| multi_dim_info = self._get_multi_dim_info() | ||
| last_tw = multi_dim_info.tw_partition_def.get_last_partition_window( | ||
| self._asset_graph_view.effective_dt | ||
| ) |
There was a problem hiding this comment.
Since this is going to become a core part of the system, IMO we should put the necessary time window dimension accessors on MultiPartitionsDefinition instead of having this one-off MultiDimInfo class.
EDIT: Looks like you did actually add to MultiPartitionsDefinition?
There was a problem hiding this comment.
I just added some helper methods to MultiPartitionsDefinition. I think we should add to the core layer in a follow up, alongside an effort to represent multi-partioning in the underlying PartitionsSubset more efficiently.
schrockn
force-pushed
the
asset-graph-view-2
branch
from
fb2c810 to
aa748ae
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
14e6b97 to
d73cb65
Compare
schrockn
force-pushed
the
asset-graph-view-2
branch
from
aa748ae to
5bfd3c4
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
d73cb65 to
109372a
Compare
This was referenced Mar 10, 2024
schrockn
force-pushed
the
asset-graph-view-2
branch
from
5bfd3c4 to
8e151ba
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
109372a to
c8e57dd
Compare
schrockn
force-pushed
the
asset-graph-view-2
branch
from
8e151ba to
1d2e500
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
c8e57dd to
38c8b37
Compare
schrockn
force-pushed
the
asset-graph-view-2
branch
from
1d2e500 to
c2d6e90
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
38c8b37 to
2cc13a9
Compare
schrockn
force-pushed
the
asset-graph-view-2
branch
from
5a89498 to
fdc715f
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
e23ea96 to
52bf7bd
Compare
schrockn
force-pushed
the
asset-graph-view-2
branch
from
fdc715f to
1077b0d
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
2 times, most recently
from
7e8c5cb to
905ef8a
Compare
schrockn
force-pushed
the
asset-graph-view-2
branch
from
9e70a03 to
b0423d4
Compare
schrockn
force-pushed
the
asset-graph-view-3
branch
from
905ef8a to
39ac36c
Compare
smackesey
approved these changes
Mar 13, 2024
schrockn
force-pushed
the
asset-graph-view-3
branch
from
39ac36c to
bbaf547
Compare
PedramNavid
pushed a commit
that referenced
this pull request
Mar 28, 2024
## Summary & Motivation Support multi-partioning in `AssetSlice`. The most notable business logic here is the treatment of the "latest" time window and time windows. Notably this code is aware of whether or not the underlying multi partition definition has a time-windowed partition. If so, methods like latest time window return the full set of partions resident in the _other_ partition dimension affiliated with with the time. ## How I Tested These Changes BK
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Support multi-partioning in
AssetSlice.The most notable business logic here is the treatment of the "latest" time window and time windows. Notably this code is aware of whether or not the underlying multi partition definition has a time-windowed partition. If so, methods like latest time window return the full set of partions resident in the other partition dimension affiliated with with the time.
How I Tested These Changes
BK