[ui] Use single hidden asset job for all single-run asset executions #23747
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Deploy preview for dagit-core-storybook ready! ✅ Preview Built with commit 63de5da. |
sryza
force-pushed
the
partition-set-to-job/graphql
branch
from
c094220 to
9bfad11
Compare
bengotow
force-pushed
the
partition-set-to-job/graphql-ui
branch
from
c3255b1 to
1a237c0
Compare
bengotow
force-pushed
the
partition-set-to-job/graphql-ui
branch
from
1a237c0 to
6b35aa4
Compare
salazarm
reviewed
Aug 20, 2024
Comment on lines
34
to
47
| | { | ||
| presetName: string; | ||
| tags: PipelineRunTag[] | null; | ||
| } | ||
| | { | ||
| isAssetJob: true; // partition set name not required | ||
| partitionName: string | null; | ||
| tags: PipelineRunTag[] | null; | ||
| } | ||
| | { | ||
| partitionsSetName: string; // required for op jobs | ||
| partitionName: string | null; | ||
| tags: PipelineRunTag[] | null; | ||
| }; |
There was a problem hiding this comment.
Might be nice to give these types individual names just to clarify their purposes.
There was a problem hiding this comment.
Ahh yeah I like that, will add that instead of the isAssetJob key
salazarm
reviewed
Aug 21, 2024
Comment on lines
+17
to
+23
| const {data} = await client.query< | ||
| ConfigPartitionSelectionQuery, | ||
| ConfigPartitionSelectionQueryVariables | ||
| >({ | ||
| query: CONFIG_PARTITION_SELECTION_QUERY, | ||
| variables, | ||
| }); |
There was a problem hiding this comment.
hmm, this should really be considered as part of the page load imo if it's part of the launchpad page. Not a blocker for this PR but I'm thinking maybe we just have some kind of wrapper around async functions
There was a problem hiding this comment.
Ahh this one might be tricky... I think for op jobs the launchpad loads and then you have to choose a partition before this config is fetched -- in some cases I think it does autopopulate (for presets maybe?) but not all the time.
sryza
added a commit
that referenced
this pull request
Aug 21, 2024
This reverts commit 35f00aa. ## Summary & Motivation We had reverted the single implicit asset job stack after discovering that it caused problems in the UI. Those problems are fixed by the stack ending in [this GraphQL change](#23494) and [this frontend change](#23747). So this unreverts. It shouldn't be merged until that frontend change makes it in. ## How I Tested These Changes
jmsanders
pushed a commit
that referenced
this pull request
Aug 21, 2024
…23747) ## Summary & Motivation Related: https://linear.app/dagster-labs/issue/FE-509/update-the-materialize-button-to-pull-jobpartitiondefinition-using-the This PR stacks on top of #23494: - On the backend, there is now just a single hidden asset job which contains all assets. - This asset job has no partition_set, but given a set of asset keys, you can still retrieve the list of available partitions and config + tags for a specific partition using new resolvers on the job. (Added in @sryza's PR) This requires some changes to expectations made on the front-end: - We no longer check both the partition definitions AND the presence of a partition set on the job the assets have in common. If you select partitioned assets you will get the partitions dialog. - When we launch a job targeting a single partition, OR open the launchpad and allow you to choose a partition, we now use a new resolver that is asset-specific to retrieve the config YAML + tags. (The "partition => config YAML" resolver previously required the partition set definition) The last point means that there is a different config-loader query for the hidden asset job and standard user-defined asset/ op jobs (which still use partition set definitions). I co-located both approaches in the same file to try to make things a bit easier to follow, but there's a bit of code duplication. Sidenote: This code is super old and still supports pipelines with multiple modes. If modes are fully gone I would love to rip that stuff out... ## How I Tested These Changes I updated the tests and also ran through a bunch of manual tests listed that I listed out in https://www.notion.so/dagster/Hidden-asset-job-change-test-cases-7a96046234da46f3a0e6b2a44e349474?pvs=4. --------- Co-authored-by: Sandy Ryza <[email protected]> Co-authored-by: bengotow <[email protected]> (cherry picked from commit 82e5040)
jmsanders
pushed a commit
that referenced
this pull request
Aug 21, 2024
This reverts commit 35f00aa. ## Summary & Motivation We had reverted the single implicit asset job stack after discovering that it caused problems in the UI. Those problems are fixed by the stack ending in [this GraphQL change](#23494) and [this frontend change](#23747). So this unreverts. It shouldn't be merged until that frontend change makes it in. ## How I Tested These Changes (cherry picked from commit 2616f6d)
cmpadden
pushed a commit
that referenced
this pull request
Aug 21, 2024
This reverts commit 35f00aa. ## Summary & Motivation We had reverted the single implicit asset job stack after discovering that it caused problems in the UI. Those problems are fixed by the stack ending in [this GraphQL change](#23494) and [this frontend change](#23747). So this unreverts. It shouldn't be merged until that frontend change makes it in. ## How I Tested These Changes
jmsanders
pushed a commit
that referenced
this pull request
Aug 22, 2024
…23747) ## Summary & Motivation Related: https://linear.app/dagster-labs/issue/FE-509/update-the-materialize-button-to-pull-jobpartitiondefinition-using-the This PR stacks on top of #23494: - On the backend, there is now just a single hidden asset job which contains all assets. - This asset job has no partition_set, but given a set of asset keys, you can still retrieve the list of available partitions and config + tags for a specific partition using new resolvers on the job. (Added in @sryza's PR) This requires some changes to expectations made on the front-end: - We no longer check both the partition definitions AND the presence of a partition set on the job the assets have in common. If you select partitioned assets you will get the partitions dialog. - When we launch a job targeting a single partition, OR open the launchpad and allow you to choose a partition, we now use a new resolver that is asset-specific to retrieve the config YAML + tags. (The "partition => config YAML" resolver previously required the partition set definition) The last point means that there is a different config-loader query for the hidden asset job and standard user-defined asset/ op jobs (which still use partition set definitions). I co-located both approaches in the same file to try to make things a bit easier to follow, but there's a bit of code duplication. Sidenote: This code is super old and still supports pipelines with multiple modes. If modes are fully gone I would love to rip that stuff out... ## How I Tested These Changes I updated the tests and also ran through a bunch of manual tests listed that I listed out in https://www.notion.so/dagster/Hidden-asset-job-change-test-cases-7a96046234da46f3a0e6b2a44e349474?pvs=4. --------- Co-authored-by: Sandy Ryza <[email protected]> Co-authored-by: bengotow <[email protected]> (cherry picked from commit 82e5040)
jmsanders
pushed a commit
that referenced
this pull request
Aug 22, 2024
This reverts commit 35f00aa. ## Summary & Motivation We had reverted the single implicit asset job stack after discovering that it caused problems in the UI. Those problems are fixed by the stack ending in [this GraphQL change](#23494) and [this frontend change](#23747). So this unreverts. It shouldn't be merged until that frontend change makes it in. ## How I Tested These Changes (cherry picked from commit 2616f6d)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Related: https://linear.app/dagster-labs/issue/FE-509/update-the-materialize-button-to-pull-jobpartitiondefinition-using-the
This PR stacks on top of #23494:
On the backend, there is now just a single hidden asset job which contains all assets.
This asset job has no partition_set, but given a set of asset keys, you can still retrieve the list of available partitions and config + tags for a specific partition using new resolvers on the job. (Added in @sryza's PR)
This requires some changes to expectations made on the front-end:
We no longer check both the partition definitions AND the presence of a partition set on the job the assets have in common. If you select partitioned assets you will get the partitions dialog.
When we launch a job targeting a single partition, OR open the launchpad and allow you to choose a partition, we now use a new resolver that is asset-specific to retrieve the config YAML + tags. (The "partition => config YAML" resolver previously required the partition set definition)
The last point means that there is a different config-loader query for the hidden asset job and standard user-defined asset/ op jobs (which still use partition set definitions). I co-located both approaches in the same file to try to make things a bit easier to follow, but there's a bit of code duplication.
Sidenote:
This code is super old and still supports pipelines with multiple modes. If modes are fully gone I would love to rip that stuff out...
How I Tested These Changes
I updated the tests and also ran through a bunch of manual tests listed that I listed out in https://www.notion.so/dagster/Hidden-asset-job-change-test-cases-7a96046234da46f3a0e6b2a44e349474?pvs=4.