You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Today, if I have a multi partitioned asset, the folder levels in storage reflect the alphabetical ordering of the partition names. I would like to suggest it to reflect the ordering that is passed to the MultiPartitionsDefinition.
Example:
tickers_partitions_def=StaticPartitionsDefinition(["tic1", "tic2")
run_day_partitions_def=StaticPartitionsDefinition("2023-04-01", "2023-05-01")
# Partitions are ordered by name, when directories are createdmulti_partitions_def=MultiPartitionsDefinition(
partitions_defs={"ticker": tickers_partitions_def, "run_day": run_day_partitions_def}
)
@asset(partitions_def=multi_partitions_def, )defquotes(context) ->List[Dict]:
ticker=context.partition_key.keys_by_dimension["ticker"]
run_day=context.partition_key.keys_by_dimension["run_day"]
return {"ticker": ticker, "run_day": run_day}
will create the following directory structure in $dagster_home/storage:
2023-04-01/tic1
2023-04-01/tic2
2023-05-01/tic1
2023-05-01/tic2
If I rename the "run_day" partitions definition to "today"
the order changes to
tic1/2023-04-01
tic1/2023-05-01
tic2/2023-04-01
tic2/2023-05-01
But it is weird to play with names (and find meaningful ones) just to keep an ordering of files that makes more sense (for other systems, for example).
I would like to suggest that the order of partitions_defs is respected. So:
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Today, if I have a multi partitioned asset, the folder levels in storage reflect the alphabetical ordering of the partition names. I would like to suggest it to reflect the ordering that is passed to the MultiPartitionsDefinition.
Example:
will create the following directory structure in $dagster_home/storage:
2023-04-01/tic1
2023-04-01/tic2
2023-05-01/tic1
2023-05-01/tic2
If I rename the "run_day" partitions definition to "today"
the order changes to
tic1/2023-04-01
tic1/2023-05-01
tic2/2023-04-01
tic2/2023-05-01
But it is weird to play with names (and find meaningful ones) just to keep an ordering of files that makes more sense (for other systems, for example).
I would like to suggest that the order of
partitions_defs
is respected. So:would have
ticker
as first level, whilewould have
run_day
as first level.Beta Was this translation helpful? Give feedback.
All reactions