Recommended way to create a partitioned asset from a non-partitioned upstream.

[easontm]

tl;dr I would like my non-partitioned materialization command to result in a partitioned asset, or for partition specification in the launchpad of partitioned assets to be optional.

Here's my contrived example

from dagster import(
    observable_source_asset,
    asset,
    DynamicPartitionsDefinition,
    AssetMaterialization,
    AssetKey,
)

@observable_source_asset
def my_source_asset():
    ...

dynamic_partition_def = DynamicPartitionsDefinition(name="my_partitions")

@asset(partitions_def=dynamic_partition_def)
def my_partitioned_asset(context: AssetExecutionContext, my_source_asset):
    # deleting previously existing partitions which don't exist after observation
    for partition in context.instance.get_dynamic_partitions(dynamic_partition_def.name):
        if partition not in my_source_asset:
            context.instance.delete_dynamic_partition(dynamic_partition_def.name, partition)
    # adding new partitions
    for partition in my_source_asset:
        if partition not in context.instance.get_dynamic_partitions(dynamic_partition_def.name):
            context.instance.add_dynamic_partition(dynamic_partition_def.name, partition)
            yield AssetMaterialization(
                asset_key=AssetKey("my_partitioned_asset"),
                metadata={"key": "value"},
                partition=partition,
            )

Context:

I want the asset in the UI to have partitions because logically it does have partitions, and I would like to be able to report metadata on them individually. Further, the data is spatially partitioned (not temporally), so it's reasonable for partitions to both appear and disappear based on the nature of the source.

Issue I'm having

I can't actually "materialize" my_partitioned_asset. Because it's partitioned, at the launchpad I'm asked to provide a partition. But I don't have one in mind. I just want it go. So far I've gotten by with doing "add partition" and putting in a dummy, which gets immediately deleted by the asset function so it works but it is not pretty.

Addendum

It seems the only way this might be possible is to have an op do this work, and hide it behind a job. Kind of a bummer that I can no longer use the asset graph for initiating asset runs. I definitely think this is a gap in the capabilities of the UI (a partitioned asset being able to be triggered with "Materialize" allowing the asset code itself to determine what partitions will become available).

[JensGM]

Hey, any news here?

[sryza]

Here's an issue where we're tracking related functionality: #9559. I.e. the ability to launch a run for a dynamically partitioned asset without specifying the partitions ahead of time.