Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Source Asset -> Unexecutable Asset Adapter #16617

Merged
merged 2 commits into from
Sep 20, 2023
Merged

Source Asset -> Unexecutable Asset Adapter #16617

merged 2 commits into from
Sep 20, 2023

Conversation

schrockn
Copy link
Member

@schrockn schrockn commented Sep 19, 2023 •
edited
Loading

Summary & Motivation

Allow us to wrap an unexecutable source asset in a vanilla assets definition.

How I Tested These Changes

BK

@schrockn
Copy link
Member Author

schrockn commented Sep 19, 2023 •
edited
Loading

Current dependencies on/for this PR:

This comment was auto-generated by Graphite.

This was referenced Sep 19, 2023
Merged
Closed
@schrockn schrockn force-pushed the source-asset-wrapping-1 branch 3 times, most recently from 808fc37 to 2e3b4a1 Compare September 19, 2023 13:20
@schrockn schrockn mentioned this pull request Sep 19, 2023
Merged
@schrockn schrockn force-pushed the source-asset-wrapping-1 branch from f47b698 to 5aabb85 Compare September 19, 2023 13:25
@schrockn schrockn marked this pull request as ready for review September 19, 2023 13:29
This was referenced Sep 19, 2023
Merged
Closed
@schrockn schrockn force-pushed the source-asset-wrapping-1 branch from 5aabb85 to 4e8484e Compare September 20, 2023 03:42
@github-actions
Copy link

github-actions bot commented Sep 20, 2023 •
edited
Loading

Deploy preview for dagit-core-storybook ready!

✅ Preview
https://dagit-core-storybook-3mbypstg2-elementl.vercel.app
https://source-asset-wrapping-1.core-storybook.dagster-docs.io

Built with commit 5f71dbf.
This pull request is being automatically deployed with vercel-action

@github-actions
Copy link

github-actions bot commented Sep 20, 2023 •
edited
Loading

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-h6aefk5i0-elementl.vercel.app
https://source-asset-wrapping-1.dagster.dagster-docs.io

Direct link to changed pages:

@github-actions
Copy link

github-actions bot commented Sep 20, 2023 •
edited
Loading

Deploy preview for dagit-storybook ready!

✅ Preview
https://dagit-storybook-qlq7vqwkj-elementl.vercel.app
https://source-asset-wrapping-1.components-storybook.dagster-docs.io

Built with commit 5f71dbf.
This pull request is being automatically deployed with vercel-action

alangenfeld
alangenfeld reviewed Sep 20, 2023
View reviewed changes
Copy link
Member

@alangenfeld alangenfeld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What exactly is the goal checking this in? Just check pointing that in process APIs work with the wrapper approach?

We'll need something along the lines of #16637 to validate compatibility through the host process / UI experience. Does that just come later? Should we block this PR on covering that as well?

python_modules/dagster/dagster/_core/definitions/observable_asset.py
Comment on lines +68 to +71
"metadata": {
**source_asset.metadata,
**{SYSTEM_METADATA_KEY_ASSET_EXECUTION_TYPE: AssetExecutionType.UNEXECUTABLE.value},
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well need to ensure is_source get toggled on the ExternalAssetNode somehow, maybe other metadata or another update to the execution type enum

@schrockn
Copy link
Member Author

Goal is to essentially enable writing that test case and making incremental progress on an implementation. I'm more than happy to move it to the test file itself or to park it.

alangenfeld
alangenfeld approved these changes Sep 20, 2023
View reviewed changes
Copy link
Member

@alangenfeld alangenfeld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its fine to land the incremental progress so we can iterate.

Can you update the summary accordingly. Just want to make it easy for someone in the future who bumps in to any loose ends left from this to be able to discern what was going on.

schrockn and others added 2 commits September 20, 2023 16:36
@schrockn
Source Asset -> Unexecutable Asset Adapter
62ff96a 
lint

fix rebasE

support partitions

cleanup

cleanup
@schrockn
minor test cahnge
5f71dbf
@schrockn schrockn force-pushed the source-asset-wrapping-1 branch from 4e8484e to 5f71dbf Compare September 20, 2023 20:36
@schrockn schrockn merged commit f58f789 into master Sep 20, 2023
3 checks passed
@schrockn schrockn deleted the source-asset-wrapping-1 branch September 20, 2023 21:19
@alangenfeld alangenfeld mentioned this pull request Sep 21, 2023
Open
This was referenced Sep 22, 2023
Closed
Closed
Closed
@schrockn schrockn mentioned this pull request Oct 6, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alangenfeld alangenfeld alangenfeld approved these changes

@smackesey smackesey Awaiting requested review from smackesey

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@schrockn @alangenfeld