Scheduling a job for a multi-partitioned asset - "Cannot access partition_key for a non-partitioned run"

[thomaschristopherking]

Hi, I am trying to schedule a job for a multi-partitioned asset. The multi-partition comprises a dynamic partition that contains identifiers (for different companies), and a daily partition.

Is there any documentation for this scenario? I have tried to follow when the scheduled job runs I get an error "Cannot access partition_key for a non-partitioned run", so it seems like the job is not supplying a partition.

This is how I have setup the job

elastic_search_ingestion = define_asset_job(name="elastic_search_ingestion",
                                            selection=[
                                                daily_pii_s3_source_data,
                                                daily_index_pii_elastic_search_commands,
                                                daily_executed_index_pii_elastic_search_commands
                                            ],
                                            partitions_def=daily_company_partition)

This is the schedule:
elastic_search_ingestion_schedule = ScheduleDefinition(job=elastic_search_ingestion, cron_schedule="30 11 * * *")

And this is the definition:

assets = (list(load_assets_from_package_module(elastic_search)))
definition = Definitions(
    assets=assets,
    schedules=[elastic_search_ingestion_schedule],
    jobs=[elastic_search_ingestion],
    sensors=[company_id_sensor],
    resources={
        "opensearch": OpenSearchResource(),
        "person_ingestion_config": PersonIngestionConfig()
    },
)

Any ideas?

Alternatively, is there any documentatiuon on how to materialize a partitioned asset from a function? I.e., is there a function-call that I can make? Or an API I can call? If I can write a function that will materialize assets for a specific partition then I might be able to schedule a job for multi-partitioned assets (edited)

[sryza]

Hey @thomaschristopherking - your code snippet has a schedule definition that says "run at 11:30 every day", but what partitions should run? If you want to launch a run for every partition, here's an example that demonstrates how to do this: #15532.