Suggest Edits: Claude keeps forgetting to add <operation> tags

[stippi]

Summary

The Suggest Edits feature should deal with incorrect/incomplete patches.

Steps to trigger the problem:

1. Use the Suggest Edits feature
2. Claude 3.5 Sonnet (New) will frequently output incomplete <edit> tags that lack the <operation> tag.

Actual Behavior:
In the Assistant panel, these edits are rendered incomplete (no target file). When clicking the patch, all edits with missing operation tags will be missing in the Diff editor that opens. Frequently, all edits lack the operation tag and the Diff editor is empty.

Expected Behavior:
Either the Suggest Edits feature can deal with it somehow, for example by auto-detecting the kind of patch or assuming <operation>update</operation>. Or it provides feedback to the LLM that the tag is missing. Or the system instructions emphasize the <operation> tag appropriately.

💡 What's nice, though, is that I can edit the assistant message myself and fix the patch invocation manually and everything in the UI automatically updates and is fixed accordingly.

Zed Version and System Specs

Zed: v0.172.10 (Zed)
OS: macOS 15.3.0
Memory: 36 GiB
Architecture: aarch64

[notpeter]

This is inherently hard to reproduce. I believe you are experiencing this, but given the non-deterministic nature of the LLM it may be hard to nail down a set of steps, but I'd like to try otherwise this isn't something we can really try and tackle.

1. Do you have any System prompts you use? Are you willing to share them / next time you run into this problem can you remove the system prompt and see if that changes your results. Check assistant: deploy prompt library to change your default prompts if defined.

2. Do you have an example file/context and prompt which triggered the behavior for you? Ideally something you could share with us (in an open source project, etc).

You can see the prompt that Zed is using for Suggested Edits here which specifies <operation> as required:

zed/assets/prompts/suggest_edits.hbs

Lines 19 to 25 in 22e2b8e

	- <operation> (required) - The type of change that should occur at the given range
	of the file. Must be one of the following values:
	- `update`: Replaces the entire range with the new text.
	- `insert_before`: Inserts the new text before the range.
	- `insert_after`: Inserts new text after the range.
	- `create`: Creates or overwrites a file with the given path and the new text.
	- `delete`: Deletes the specified range from the file.

If you'd like to experiment with these prompts, instructions for overriding these prompt templates here: https://zed.dev/docs/assistant/prompting#advanced-concepts

In the meantime I'm going to migrate this to discussions until there's something actionable here to do (other than "iterate on the suggest_edits.hbs prompt to try and better coerce Claude into following instructions")

[stippi]

Thanks for taking the time to provide such detailed feedback. I don't have a system prompt, at least I never created one and when I expand the "Default" item at the top of the Assistant panel, it is empty.