[single-implicit-asset-job] add partitionKeysOrError and partition to GrapheneJob
#23494
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sryza
force-pushed
the
partition-set-to-job/grpc
branch
8 times, most recently
from
8500be5 to
28f571b
Compare
sryza
force-pushed
the
partition-set-to-job/graphql
branch
from
f2f3f53 to
930b119
Compare
sryza
changed the title
partitionKeysOrError and partition to GrapheneJobpartitionKeysOrError and partition to GrapheneJob
sryza
changed the title
partitionKeysOrError and partition to GrapheneJobpartitionKeysOrError and partition to GrapheneJob
sryza
force-pushed
the
partition-set-to-job/grpc
branch
from
28f571b to
fc88adb
Compare
sryza
force-pushed
the
partition-set-to-job/graphql
branch
from
930b119 to
fac980d
Compare
|
Deploy preview for dagit-core-storybook ready! ✅ Preview Built with commit d8a9aa5. |
sryza
force-pushed
the
partition-set-to-job/graphql
branch
from
fac980d to
91e220c
Compare
sryza
force-pushed
the
partition-set-to-job/grpc
branch
2 times, most recently
from
1f24881 to
9be36bd
Compare
sryza
commented
Aug 20, 2024
| from .util import non_null_list | ||
|
|
||
| GrapheneAutoMaterializeDecisionType = graphene.Enum.from_enum(AutoMaterializeDecisionType) | ||
|
|
||
|
|
||
| class GraphenePartitionKeys(graphene.ObjectType): |
There was a problem hiding this comment.
Just moved this to a different file
sryza
force-pushed
the
partition-set-to-job/graphql
branch
from
91e220c to
c094220
Compare
|
Will put up a PR for the |
sryza
force-pushed
the
partition-set-to-job/graphql
branch
from
c094220 to
9bfad11
Compare
sryza
commented
Aug 20, 2024
| @@ -290,5 +290,29 @@ def to_graphql_input(self) -> Mapping[str, Any]: | |||
| } | |||
|
|
|||
|
|
|||
| def apply_cursor_limit_reverse( | |||
There was a problem hiding this comment.
just moved this from a different file
sryza
commented
Aug 20, 2024
| @@ -0,0 +1,26 @@ | |||
| import graphene | |||
There was a problem hiding this comment.
Just moved this from a different file
OwenKephart
approved these changes
Aug 20, 2024
bengotow
added a commit
that referenced
this pull request
Aug 21, 2024
…23747) ## Summary & Motivation Related: https://linear.app/dagster-labs/issue/FE-509/update-the-materialize-button-to-pull-jobpartitiondefinition-using-the This PR stacks on top of #23494: - On the backend, there is now just a single hidden asset job which contains all assets. - This asset job has no partition_set, but given a set of asset keys, you can still retrieve the list of available partitions and config + tags for a specific partition using new resolvers on the job. (Added in @sryza's PR) This requires some changes to expectations made on the front-end: - We no longer check both the partition definitions AND the presence of a partition set on the job the assets have in common. If you select partitioned assets you will get the partitions dialog. - When we launch a job targeting a single partition, OR open the launchpad and allow you to choose a partition, we now use a new resolver that is asset-specific to retrieve the config YAML + tags. (The "partition => config YAML" resolver previously required the partition set definition) The last point means that there is a different config-loader query for the hidden asset job and standard user-defined asset/ op jobs (which still use partition set definitions). I co-located both approaches in the same file to try to make things a bit easier to follow, but there's a bit of code duplication. Sidenote: This code is super old and still supports pipelines with multiple modes. If modes are fully gone I would love to rip that stuff out... ## How I Tested These Changes I updated the tests and also ran through a bunch of manual tests listed that I listed out in https://www.notion.so/dagster/Hidden-asset-job-change-test-cases-7a96046234da46f3a0e6b2a44e349474?pvs=4. --------- Co-authored-by: Sandy Ryza <[email protected]> Co-authored-by: bengotow <[email protected]>
sryza
added a commit
that referenced
this pull request
Aug 21, 2024
This reverts commit 35f00aa. ## Summary & Motivation We had reverted the single implicit asset job stack after discovering that it caused problems in the UI. Those problems are fixed by the stack ending in [this GraphQL change](#23494) and [this frontend change](#23747). So this unreverts. It shouldn't be merged until that frontend change makes it in. ## How I Tested These Changes
jmsanders
pushed a commit
that referenced
this pull request
Aug 21, 2024
…` to `GrapheneJob` (#23494) ## Summary & Motivation This enables the frontend to query for partition information with respect to an asset selection within a job. This enables a world where jobs can contain assets with different `PartitionsDefinition`s. It's used by @bengotow 's `partition-set-to-job/graphql-ui` branch. Sits atop #23491. ## How I Tested These Changes (cherry picked from commit 3bd53c0)
jmsanders
pushed a commit
that referenced
this pull request
Aug 21, 2024
…23747) ## Summary & Motivation Related: https://linear.app/dagster-labs/issue/FE-509/update-the-materialize-button-to-pull-jobpartitiondefinition-using-the This PR stacks on top of #23494: - On the backend, there is now just a single hidden asset job which contains all assets. - This asset job has no partition_set, but given a set of asset keys, you can still retrieve the list of available partitions and config + tags for a specific partition using new resolvers on the job. (Added in @sryza's PR) This requires some changes to expectations made on the front-end: - We no longer check both the partition definitions AND the presence of a partition set on the job the assets have in common. If you select partitioned assets you will get the partitions dialog. - When we launch a job targeting a single partition, OR open the launchpad and allow you to choose a partition, we now use a new resolver that is asset-specific to retrieve the config YAML + tags. (The "partition => config YAML" resolver previously required the partition set definition) The last point means that there is a different config-loader query for the hidden asset job and standard user-defined asset/ op jobs (which still use partition set definitions). I co-located both approaches in the same file to try to make things a bit easier to follow, but there's a bit of code duplication. Sidenote: This code is super old and still supports pipelines with multiple modes. If modes are fully gone I would love to rip that stuff out... ## How I Tested These Changes I updated the tests and also ran through a bunch of manual tests listed that I listed out in https://www.notion.so/dagster/Hidden-asset-job-change-test-cases-7a96046234da46f3a0e6b2a44e349474?pvs=4. --------- Co-authored-by: Sandy Ryza <[email protected]> Co-authored-by: bengotow <[email protected]> (cherry picked from commit 82e5040)
jmsanders
pushed a commit
that referenced
this pull request
Aug 21, 2024
This reverts commit 35f00aa. ## Summary & Motivation We had reverted the single implicit asset job stack after discovering that it caused problems in the UI. Those problems are fixed by the stack ending in [this GraphQL change](#23494) and [this frontend change](#23747). So this unreverts. It shouldn't be merged until that frontend change makes it in. ## How I Tested These Changes (cherry picked from commit 2616f6d)
cmpadden
pushed a commit
that referenced
this pull request
Aug 21, 2024
This reverts commit 35f00aa. ## Summary & Motivation We had reverted the single implicit asset job stack after discovering that it caused problems in the UI. Those problems are fixed by the stack ending in [this GraphQL change](#23494) and [this frontend change](#23747). So this unreverts. It shouldn't be merged until that frontend change makes it in. ## How I Tested These Changes
jmsanders
pushed a commit
that referenced
this pull request
Aug 22, 2024
…` to `GrapheneJob` (#23494) ## Summary & Motivation This enables the frontend to query for partition information with respect to an asset selection within a job. This enables a world where jobs can contain assets with different `PartitionsDefinition`s. It's used by @bengotow 's `partition-set-to-job/graphql-ui` branch. Sits atop #23491. ## How I Tested These Changes (cherry picked from commit 3bd53c0)
jmsanders
pushed a commit
that referenced
this pull request
Aug 22, 2024
…23747) ## Summary & Motivation Related: https://linear.app/dagster-labs/issue/FE-509/update-the-materialize-button-to-pull-jobpartitiondefinition-using-the This PR stacks on top of #23494: - On the backend, there is now just a single hidden asset job which contains all assets. - This asset job has no partition_set, but given a set of asset keys, you can still retrieve the list of available partitions and config + tags for a specific partition using new resolvers on the job. (Added in @sryza's PR) This requires some changes to expectations made on the front-end: - We no longer check both the partition definitions AND the presence of a partition set on the job the assets have in common. If you select partitioned assets you will get the partitions dialog. - When we launch a job targeting a single partition, OR open the launchpad and allow you to choose a partition, we now use a new resolver that is asset-specific to retrieve the config YAML + tags. (The "partition => config YAML" resolver previously required the partition set definition) The last point means that there is a different config-loader query for the hidden asset job and standard user-defined asset/ op jobs (which still use partition set definitions). I co-located both approaches in the same file to try to make things a bit easier to follow, but there's a bit of code duplication. Sidenote: This code is super old and still supports pipelines with multiple modes. If modes are fully gone I would love to rip that stuff out... ## How I Tested These Changes I updated the tests and also ran through a bunch of manual tests listed that I listed out in https://www.notion.so/dagster/Hidden-asset-job-change-test-cases-7a96046234da46f3a0e6b2a44e349474?pvs=4. --------- Co-authored-by: Sandy Ryza <[email protected]> Co-authored-by: bengotow <[email protected]> (cherry picked from commit 82e5040)
jmsanders
pushed a commit
that referenced
this pull request
Aug 22, 2024
This reverts commit 35f00aa. ## Summary & Motivation We had reverted the single implicit asset job stack after discovering that it caused problems in the UI. Those problems are fixed by the stack ending in [this GraphQL change](#23494) and [this frontend change](#23747). So this unreverts. It shouldn't be merged until that frontend change makes it in. ## How I Tested These Changes (cherry picked from commit 2616f6d)
sryza
added a commit
that referenced
this pull request
Aug 22, 2024
…3807) ## Summary & Motivation While working on #23491 and #23494, I wrote a set of tests that validate that operations like `get_tags_for_partition` over GraphQL and gRPC, when an asset job includes assets with different `PartitionsDefinition`s and an asset selection is provided. However, I had to take them out due to the way the stack was sequenced. This PR adds them back in. ## How I Tested These Changes ## Changelog NOCHANGELOG
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This enables the frontend to query for partition information with respect to an asset selection within a job. This enables a world where jobs can contain assets with different
PartitionsDefinitions.It's used by @bengotow 's
partition-set-to-job/graphql-uibranch.Sits atop #23491.
How I Tested These Changes