You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This adds unnecessary friction for the user because:
They have to redundantly specify the asset key. (Python's implementation uses optional arguments with defaults, and doesn't have this issue).
If an invalid asset key is specified, we currently report back that invalid key to Dagster 🐛
Proposal
Ideally, users won't have to specify this key. This can be achieve in two ways:
Approach 1: Use Option<&str>
This is similar to Python's implementation. However, Rust doesn't allow default values in function arguments. So this will mean that users still have to write None.
Nothing's prevents the user from including an asset key in a single-asset use case, which means we still need validation internally.
Approach 2: Different PipesContext APIs for single vs multi-assets
Different APIs can be created for the single vs multi-asset case:
implPipesContext{// Note: No `asset_key` in API belowfnreport_asset_materialization(metadata: ...){ ...}// Same signature for reporting asset checks}implMultiAssetPipesContext{// Mandatory `asset_key` argumentfnreport_asset_materialization(metadata: ...,asset_key:&str){ ...}// Similar signature for reporting asset checks}
The downside of this approach is the creation of a separate class. This may be okay, if we do not reach for this technique again — we do not expect similar redundant arguments on the user's journey / hot path. Also, the two structs can be hidden (like open_dagster_pipes hiding PipesContext) behind a builder pattern, which is quite common in Rust. (See examples below)
Single Asset Case
let context = PipesContext::builder().with_params_loader(...)// Users can optionally customize their components.build();let metadata = ...;context.report_asset_materialization(metadata)?;// Internally uses `context.data` for asset key
Multi-Asset Case
let context = PipesContext::builder().with_context_loader(...)// Users can customize their components.build_multi_asset();// A multi-asset context is builtlet metadata = ...;context.report_asset_materialization(metadata,"asset1")?;// Validates "asset1" against list of assets in `context.data`
I'm leaning towards Approach 2.
The text was updated successfully, but these errors were encountered:
Issue
Currently, users have to specify the
asset_key
when reporting data back to Dagster.community-integrations/libraries/pipes/implementations/rust/example-dagster-pipes-rust-project/rust_processing_jobs/src/main.rs
Line 13 in 3ee24de
community-integrations/libraries/pipes/implementations/rust/example-dagster-pipes-rust-project/rust_processing_jobs/src/main.rs
Lines 19 to 20 in 3ee24de
This adds unnecessary friction for the user because:
Proposal
Ideally, users won't have to specify this key. This can be achieve in two ways:
Approach 1: Use Option<&str>
This is similar to Python's implementation. However, Rust doesn't allow default values in function arguments. So this will mean that users still have to write
None
.Nothing's prevents the user from including an asset key in a single-asset use case, which means we still need validation internally.
Approach 2: Different
PipesContext
APIs for single vs multi-assetsDifferent APIs can be created for the single vs multi-asset case:
The downside of this approach is the creation of a separate class. This may be okay, if we do not reach for this technique again — we do not expect similar redundant arguments on the user's journey / hot path. Also, the two structs can be hidden (like
open_dagster_pipes
hidingPipesContext
) behind a builder pattern, which is quite common in Rust. (See examples below)Single Asset Case
Multi-Asset Case
I'm leaning towards Approach 2.
The text was updated successfully, but these errors were encountered: