Propagate automation condition information through the UI for user code conditions #25211

OwenKephart · 2024-10-11T00:29:32Z

Summary & Motivation

It is not possible to serialize the full automation condition object if it is user-defined. In these cases, we only persist the snapshot.

Updates the graphql code to use this snapshot instead of the raw condition, and adds e2e test ensuring we can execute / get the correct label for these objects.

How I Tested These Changes

Changelog

NOCHANGELOG

OwenKephart · 2024-10-11T00:29:53Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @OwenKephart and the rest of your teammates on Graphite

OwenKephart

@gibsondan flagging this PR as at one point we removed this data from the snapshot due to (iirc?) unfounded beliefs that it was blowing up the workspace size -- just confirming that this was actually unfounded

gibsondan · 2024-10-11T01:09:34Z

I think it was in fact visibly increasing the workspace size but it remains unclear what the exact impact of that was - worth a quick chat maybe

…

On Thu, Oct 10, 2024 at 7:41 PM OwenKephart ***@***.***> wrote: ***@***.**** commented on this pull request. @gibsondan <https://github.com/gibsondan> flagging this PR as at one point we removed this data from the snapshot due to (iirc?) unfounded beliefs that it was blowing up the workspace size -- just confirming that this was actually unfounded — Reply to this email directly, view it on GitHub <#25211 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAPJCZP27KISBEL745WMADZ24NE7AVCNFSM6AAAAABPX3XBTCVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGNRRG4YDGNZSG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

gibsondan

is there a world where we only persist this new thing once you opt in to user-defined automation condition sensors? or does that make things way more complicated?

gibsondan

sounds like that world is indeed possible

updated

schrockn · 2024-10-15T18:45:57Z

python_modules/dagster-graphql/dagster_graphql/schema/asset_condition_evaluations.py

            item.description,
            item.children,
        )
+    elif isinstance(item, AutomationConditionSnapshot):
+        label, name, description, children = (
+            item.node_snapshot.label,
+            item.node_snapshot.name,
+            item.node_snapshot.description,
+            item.children,
+        )


[Re: lines +336 to +349]

can we not just _always_ use the snapshot?

See this comment inline on Graphite.

Ah we need to use the AutomationConditionEvaluation in the history UI, as we don't store the full snapshot there. We could put a method on AutomationConditionEvaluation that reconstructs an AutomationConditionSnapshot though, just seems out of scope for this PR. Can stack one a PR for that on top though

schrockn

Why don't we just always use the snapshots?

OwenKephart · 2024-10-15T21:09:44Z

In general, because we need to be able to execute the conditions on the daemon side of the process if executing in daemon mode. In the code sample you linked to, no good reason (and I removed it).

updated

schrockn · 2024-10-15T23:57:43Z

python_modules/dagster-graphql/dagster_graphql/schema/asset_graph.py

+        automation_condition = (
+            self._asset_node_snap.automation_condition_snapshot
+            or self._asset_node_snap.automation_condition
+        )
+        if automation_condition:
+            return GrapheneAutomationCondition(
+                # we only store one of automation_condition or automation_condition_snapshot
+                automation_condition
+                if isinstance(automation_condition, AutomationConditionSnapshot)
+                else automation_condition.get_snapshot()
+            )


Don't see why we don't unconditionally use the snapshot in the graphql server

As in why can't the asset_node_snap just have the snapshot?

Responded to this above

schrockn

This is just a weird mental model in my mind.

Seems like there should be just a snapshot but we should have special logic to reconstruct the "real" automation condition in the asset daemon from the snapshot.

bengotow

Just reviewed the UI + graphql portions of this but it looks good to me! Left one inline comment, might be nice to remove the autoMaterializePolicy fields from the graphql queries if we can.

js_modules/dagster-ui/packages/ui-core/src/assets/AssetTableFragment.tsx

OwenKephart · 2024-10-16T23:56:36Z

@schrockn The snapshot is designed to only contain information that's relevant to displaying a condition in the UI to a user. This means that it reduces down a lot of the complexity of the parameters etc. of a given condition to being just a simple string (e.g. the any_deps_match condition can have an "allow" or "ignore" parameter, which are rendered in the string like so: https://sourcegraph.com/github.com/dagster-io/dagster@a118eeb03d2c9a6f2a8ff1c152f4a0acc52e4a7a/-/blob/python_modules/dagster/dagster/_core/definitions/declarative_automation/operators/dep_operators.py?L76

In general, this means that we don't enforce any contract on how those names should be generated, and so trying to deserialize them is impossible, and much more the job of the serdes framework.

So if we wanted to avoid having both an AutomationCondition field and an AutomationConditionSnapshot field on the asset_node_snap, then we'd need to go the route of only populating the AutomationCondition field, which would necessitate having a special "NonExecutableAutomationCondition" or something that we could transform user-defined conditions into, whose purpose would be to allow us to get the parts of user-defined automation conditions that are necessary to render it in the UI across the serdes boundary. I explored this approach in a previous PR but we deemed that to be a confusing system, so that approach was abandoned, but I could be convinced to go back to that setup.

responded

…de conditions

OwenKephart · 2024-10-22T16:50:24Z

Merge activity

Oct 22, 12:50 PM EDT: A user started a stack merge that includes this pull request via Graphite.
Oct 22, 12:50 PM EDT: A user merged this pull request with Graphite.

…de conditions (dagster-io#25211) ## Summary & Motivation It is not possible to serialize the full automation condition object if it is user-defined. In these cases, we only persist the snapshot. Updates the graphql code to use this snapshot instead of the raw condition, and adds e2e test ensuring we can execute / get the correct label for these objects. ## How I Tested These Changes ## Changelog NOCHANGELOG

OwenKephart mentioned this pull request Oct 11, 2024

Export AutomationContext and AutomationResult #25210

Merged

OwenKephart force-pushed the 10-10-export_automationcontext_and_automationresult branch from d2572a5 to 5c39c20 Compare October 11, 2024 00:30

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch from 11b2b09 to 7a838b5 Compare October 11, 2024 00:30

OwenKephart requested a review from bengotow October 11, 2024 00:40

OwenKephart commented Oct 11, 2024

View reviewed changes

This was referenced Oct 11, 2024

Add test demonstrating that we do not hit the RESOURCE_EXHAUSTED error above a given threshold #25214

Merged

Add error for user code sensors with too many entities targeted #25215

Merged

gibsondan reviewed Oct 11, 2024

View reviewed changes

gibsondan previously requested changes Oct 11, 2024

View reviewed changes

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch from 7a838b5 to 10e5e25 Compare October 11, 2024 16:47

OwenKephart requested a review from gibsondan October 11, 2024 16:47

OwenKephart force-pushed the 10-10-export_automationcontext_and_automationresult branch from 5c39c20 to 8acc87b Compare October 11, 2024 17:18

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch from 10e5e25 to 2734223 Compare October 11, 2024 17:18

OwenKephart force-pushed the 10-10-export_automationcontext_and_automationresult branch from 8acc87b to 66d4fe6 Compare October 11, 2024 19:46

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch from 2734223 to 404dcbf Compare October 11, 2024 19:46

OwenKephart force-pushed the 10-10-export_automationcontext_and_automationresult branch from 66d4fe6 to 3a1e7c1 Compare October 11, 2024 20:01

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch from c02046f to 12ccaf1 Compare October 15, 2024 17:55

schrockn reviewed Oct 15, 2024

View reviewed changes

schrockn previously requested changes Oct 15, 2024

View reviewed changes

OwenKephart force-pushed the 10-10-export_automationcontext_and_automationresult branch from 5c31e88 to ddc6079 Compare October 15, 2024 18:54

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch from 12ccaf1 to 3bd0f6e Compare October 15, 2024 18:54

OwenKephart force-pushed the 10-10-export_automationcontext_and_automationresult branch from ddc6079 to ebe651d Compare October 15, 2024 20:08

Base automatically changed from 10-10-export_automationcontext_and_automationresult to master October 15, 2024 20:10

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch 2 times, most recently from 700d7e4 to b9a4a6e Compare October 15, 2024 21:08

OwenKephart requested a review from schrockn October 15, 2024 21:09

schrockn reviewed Oct 15, 2024

View reviewed changes

schrockn previously requested changes Oct 16, 2024

View reviewed changes

bengotow approved these changes Oct 16, 2024

View reviewed changes

js_modules/dagster-ui/packages/ui-core/src/assets/AssetTableFragment.tsx Show resolved Hide resolved

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch 2 times, most recently from 8bf74fd to 12f4f07 Compare October 16, 2024 23:45

OwenKephart requested a review from schrockn October 16, 2024 23:57

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch from 12f4f07 to cab309b Compare October 17, 2024 16:38

OwenKephart mentioned this pull request Oct 17, 2024

[1.9] Add automation_condition to observable_source_asset #25344

Merged

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch from cab309b to fad7f9a Compare October 21, 2024 23:04

OwenKephart mentioned this pull request Oct 22, 2024

[1.9] Replace Auto-materialize references in the UI with Automation condition #25414

Merged

Propagate automation condition information through the UI for user co…

9a74d41

…de conditions

OwenKephart force-pushed the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch from fad7f9a to 9a74d41 Compare October 22, 2024 16:48

OwenKephart merged commit fd9ffdc into master Oct 22, 2024
0 of 2 checks passed

OwenKephart deleted the 10-10-propagate_automation_condition_information_through_the_ui_for_user_code_conditions branch October 22, 2024 16:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Propagate automation condition information through the UI for user code conditions #25211

Propagate automation condition information through the UI for user code conditions #25211

OwenKephart commented Oct 11, 2024

OwenKephart commented Oct 11, 2024 •

edited

Loading

OwenKephart left a comment

gibsondan commented Oct 11, 2024 via email

gibsondan left a comment

gibsondan left a comment

schrockn Oct 15, 2024

OwenKephart Oct 16, 2024

schrockn left a comment

OwenKephart commented Oct 15, 2024

schrockn Oct 15, 2024

schrockn Oct 15, 2024

OwenKephart Oct 16, 2024

schrockn left a comment

bengotow left a comment

OwenKephart commented Oct 16, 2024

OwenKephart commented Oct 22, 2024 •

edited

Loading

Propagate automation condition information through the UI for user code conditions #25211

Propagate automation condition information through the UI for user code conditions #25211

Conversation

OwenKephart commented Oct 11, 2024

Summary & Motivation

How I Tested These Changes

Changelog

OwenKephart commented Oct 11, 2024 • edited Loading

OwenKephart left a comment

Choose a reason for hiding this comment

gibsondan commented Oct 11, 2024 via email

gibsondan left a comment

Choose a reason for hiding this comment

gibsondan left a comment

Choose a reason for hiding this comment

schrockn Oct 15, 2024

Choose a reason for hiding this comment

OwenKephart Oct 16, 2024

Choose a reason for hiding this comment

schrockn left a comment

Choose a reason for hiding this comment

OwenKephart commented Oct 15, 2024

schrockn Oct 15, 2024

Choose a reason for hiding this comment

schrockn Oct 15, 2024

Choose a reason for hiding this comment

OwenKephart Oct 16, 2024

Choose a reason for hiding this comment

schrockn left a comment

Choose a reason for hiding this comment

bengotow left a comment

Choose a reason for hiding this comment

OwenKephart commented Oct 16, 2024

OwenKephart commented Oct 22, 2024 • edited Loading

Merge activity

OwenKephart commented Oct 11, 2024 •

edited

Loading

OwenKephart commented Oct 22, 2024 •

edited

Loading