Skip to content

Commit

Permalink
Update content/blog/codegen-learnings/index.md
Browse files Browse the repository at this point in the history
Co-authored-by: Gavin <[email protected]>
  • Loading branch information
arturl and thtmnisamnstr authored Jan 4, 2025
1 parent 4b86a80 commit 274f954
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion content/blog/codegen-learnings/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ However, the first of these problems can be addressed by the approach known as "

We observed that many generated TypeScript programs that fail to typecheck contain only a few errors, and asking the LLM to fix these errors often produces a valid program. While this approach is still experimental, we see promising results where getting to a correct program requires only 1-2 iterations of self-debugging. The biggest challenge with this approach is doing it in real-time. The user is staring at the screen waiting for an answer, so latency is a major concern.

Monitoring these typechecking errors in production can also provide valuable insight into the quality of the RAG, and even suggest specific solutions. For example, failure to typecheck a member-access expression is a likely indicator of a missing type schema (a recall problem), or a "wrong" schema brought in by an irrelevant document (a precision problem).
Monitoring these typechecking errors in production can also provide valuable insight into the quality of the RAG and even suggest specific solutions. For example, failure to typecheck a member-access expression is a likely indicator of a missing type schema (a recall problem) or a "wrong" schema brought in by an irrelevant document (a precision problem).

Self-debugging can also be extended to include the `pulumi preview` command, which is a "dry run" operation before the actual deployment and can detect many real or potential problems such as destructive actions, incorrect configurations that cannot be detected at compile time, dependency conflicts and policy violations.

Expand Down

0 comments on commit 274f954

Please sign in to comment.