RFC: @observable_source_asset adapter to generate AssetsDefinition and SensorDefinition #16712
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Sep 22, 2023
schrockn
commented
Sep 22, 2023
Comment on lines
+409
to
+433
| @observable_source_asset | ||
| def an_asset() -> DataVersion: | ||
| return DataVersion("data_version") | ||
|
|
||
| # calling these helpers could be in the Definitions object itself | ||
| defs = Definitions( | ||
| assets=[create_assets_def_from_source_asset(an_asset)], | ||
| sensors=[sensor_def_from_observable_source_asset(an_asset)], | ||
| ) | ||
|
|
||
| instance = DagsterInstance.ephemeral() | ||
|
|
||
| result = defs.get_implicit_global_asset_job_def().execute_in_process(instance=instance) | ||
| assert result.success | ||
|
|
||
| assert get_latest_asset_observation(instance, an_asset.key).data_version == "data_version" | ||
|
|
||
| sensor_def = defs.get_sensor_def(_get_auto_sensor_name(an_asset.key)) | ||
|
|
||
| sensor_result = sensor_def.evaluate_tick(build_sensor_context(instance=instance)) | ||
|
|
||
| assert len(sensor_result.asset_events) == 1 | ||
| asset_observation = sensor_result.asset_events[0] | ||
| assert isinstance(asset_observation, AssetObservation) | ||
| assert asset_observation.asset_key == an_asset.key |
There was a problem hiding this comment.
@sryza this is the relevant test case to your interests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This PR demonstrates how we could write an adapter to achieve backwards compatibility with existing
observable_source_assetdeclarations.In a test case we have the following code:
If we so chose, we could push down calling those functions against the source asset to within the Definitions
__init__method machinery to make this completely automatic. Whether or not that is a good idea is a different discussion.How I Tested These Changes
BK