partition methods on AssetExecutionContext

[jamiedemaria]

Summary & Motivation

Adds partition_key_range_for_asset_key to AssetExecutionContext. Also does some minor reorganizing of partition methods on StepExecutionContext to support passing AssetKeys rather than input names

To Do:

• update tests to use new methods, instead of deprecated ones
• determine if we need to add a partition_keys_for_asset_key method to support materializing multiple static partitions

How I Tested These Changes

[jamiedemaria]

Current dependencies on/for this PR:

• master
  • PR interface for ExtContext and AssetExecutionContext #16480
    • PR Make AssetExecutionContext a subclass of OpExecutionContext #16596
      • PR partition methods on AssetExecutionContext #16625 👈
      • PR provide AssetExecutionContext class to context #16493
        • PR update build_op_context to build_asset_context where relevant #16673
          • PR support direct invocation with AssetExecutionContext #16635
            • PR test AssetExecutionContext subclass deprecations #16598
    • PR OpExecutionContext metaclass #16486
      • PR Simplified AssetExecutionContext #16487

This comment was auto-generated by Graphite.

[jamiedemaria]

still not super happy with the is_dependency solution to the self dependent asset problem. here's my comment from the first PR exploring this:

---

the is_dependency bool param here is to handle the self-dependent partition case. Basically when you have a self-dependent partition, if you said context.partition_key_for_asset_key("my_asset") it's unclear if you want the partition key of my_asset as a dependency, or as is being currently materialized. I'm handling this (for now at least) by allowing users to specify is_dependency to indicate they want it loaded as a dependency.

In all other cases, the function should be able to figure out the correct partition key to load given the current asset

I'm not set on this solution, so if you have any concerns or other ideas let me know

[jamiedemaria]

I'd also like to have a discussion of what partition methods should be on the AssetExecutionContext. Currently we have:

• partition_key property
• partition_key_range property
• partition_time_window property
• partition_key_range_for_asset_key method

This leaves out the ability to:

• Directly get a TimeWindow for a partition of an upstream asset, or any asset in a multi asset, bc there's no partition_time_window_for_asset_key method. The user can construct this info from the key range and the partitions def, so we may just need to document this process. It is pretty convoluted though, so we may be better off adding a key_range -> time_window helper method to the partition definitions
• Directly get a list of all partition keys being materialized for a static partitioned asset. The user can construct this info by using key range and the partitions def, so we may just need to document this process
• Get the PartitionsDefinition for an asset from the context. This seems pretty important to add, especially if we want the users to use the PartitionsDefinition to get time windows/key lists/etc

I just want to make sure this is the direction we want to go. We can always add more methods in at a later date, but if we decide we need them now, i'll go ahead and add them in

[jamiedemaria]

Discussing and aligning on partition method APIs here https://github.com/dagster-io/internal/discussions/6704. then will implement in this PR

[alangenfeld]

whats the overlap with this and #16480?

back to your queue

[jamiedemaria]

closing for now. I think determining the partition method APIs will be best done in a discussion.