Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Agent improvements: Adopt system instructions and allow multiple command executions #717

Draft
wants to merge 29 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
71bc0ef
New system instructions
DonggeLiu Nov 12, 2024
3b25840
Apply system instructions
DonggeLiu Nov 12, 2024
3498faf
Refine priming
DonggeLiu Nov 12, 2024
3c8d99a
Minor correction
DonggeLiu Nov 12, 2024
9477c02
Bug fix
DonggeLiu Nov 12, 2024
8cdbaca
Do not emphasize on simple/minimum fuzz target
DonggeLiu Nov 12, 2024
1759a67
Complete conclusion protocol
DonggeLiu Nov 12, 2024
8b83ef1
Minimize priming
DonggeLiu Nov 12, 2024
8b17486
Visually separate RESPONSE/PROMPT and their content by a line break
DonggeLiu Nov 13, 2024
5272385
Strip empty lines and spaces from bash output
DonggeLiu Nov 13, 2024
adb9e72
Allow executing multiple bash commands in one response
DonggeLiu Nov 13, 2024
4c41553
Allow passing system instructions to LLM
DonggeLiu Nov 13, 2024
e8e737e
More concise objective and instructions
DonggeLiu Nov 13, 2024
4adf624
Make code consistent
DonggeLiu Nov 13, 2024
ebf36c0
lower input token limit by system instruction token size
DonggeLiu Nov 13, 2024
b0d6d3c
Simplify system instruction
DonggeLiu Nov 13, 2024
8d1de5b
Consider previous text in the same prompt when truncate new text
DonggeLiu Nov 14, 2024
dc007b4
minor fix
DonggeLiu Nov 14, 2024
f474d16
ASK LLM do not compile
DonggeLiu Nov 14, 2024
6286ae5
Prioritize understanding over retrying
DonggeLiu Nov 14, 2024
122b0a5
Remove the compile command so that LLM cannot learn
DonggeLiu Nov 14, 2024
1ca97d3
Debug truncating prompt
DonggeLiu Nov 14, 2024
611b4b7
Reduce unnecessary logs
DonggeLiu Nov 14, 2024
588afc9
Fix bug to remove compile command from build result
DonggeLiu Nov 14, 2024
76d7975
Simpler debugging
DonggeLiu Nov 14, 2024
2b24d05
Fix bug in truncation
DonggeLiu Nov 14, 2024
610efda
Set log level
DonggeLiu Nov 14, 2024
b2be748
Fix truncation and be more strict on individual output size limit
DonggeLiu Nov 14, 2024
f189d18
Retry on VertexAI's InternalServerError
DonggeLiu Nov 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Refine priming
DonggeLiu committed Nov 19, 2024
commit 3498fafeafb85e0b3c72f45e012ecdfae04c12fb
28 changes: 1 addition & 27 deletions prompts/agent/prototyper-priming.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,3 @@
<system>
As a security testing engineer, you must write an `int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)` fuzz target in {LANGUAGE}.
Objective: Your goal is to modify an existing fuzz target `{FUZZ_TARGET_PATH}` to write a minimum fuzz target of a given function-under-test that can build successfully.
</system>

<steps>
Follow these steps to write a minimum fuzz target:

@@ -87,28 +82,7 @@ Write a new build script only if the existing one (`/src/build.sh`) is insuffici

Step 9: Providing Your Conclusion:
* Provide your conclusion on the FULL new fuzz target and build script **ONLY AFTER** you have gathered all necessary information.
* **DO NOT SEND** any other content (e.g., bash tool commands) in the conclusion message. ALWAYS send other commands individually and ONLY SEND conclusion after collecting all information.
* Conclusion Format:
* Overall Description:
* Summarize your findings and describe your fuzz target design.
* Wrap this summary within <conclusion> and </conclusion> tags.
* Modified Fuzz Target:
* Provide the full code of the modified fuzz target.
* Wrap the code within <fuzz target> and </fuzz target> tags.
* Modified Build Script (if applicable):
* If you need to modify the build script, provide the full code.
* Wrap it within <build script> and </build script> tags.
* Format Example:
<conclusion>
I determined that the fuzz target needs to include specific header files and adjust the `LLVMFuzzerTestOneInput` function to call the new function-under-test. Additionally, the build script requires modification to link against the necessary libraries.
</conclusion>
<fuzz target>
[Your FULL fuzz target code here.]
</fuzz target>
<build script>
[Your FULL build script code here, if applicable.]
</build script>

* Follow the conclusion response protocol (see section <conclusion protocol>).
</steps>

{TYPE_SPECIFIC_PRIMING}