Crash Analyzer Agent #814

maoyixie · 2025-02-25T12:47:57Z

This PR mainly implements a crash analyzer that can interact with LLDB in the multi-agent framework, and supports GPT. In addition, this PR attempts to fix the problem of not replacing the fuzz target and build script. This PR is under testing. The main logic is no longer changing, and minor bugs are being fixed.

TODO:

Optimize the process of agent interaction with LLDB.
Solve the problem of missing debugging information for some projects.
Try to add LLM-based static methods to enhance the crash analyzer.

DonggeLiu · 2025-03-03T00:16:26Z

Thanks again for the pushing the code fixing the conflicts, @maoyixie!
I will make sure to read your code today.

Before that, let me start an experiment below so that we can see its results together later : )
There seems to be another conflict between your chat_llm with the one on main branch. The major difference seems to be:

self.conversation_history.extend(prompt.get())

DonggeLiu · 2025-03-03T00:17:23Z

/gcbrun exp -n mx -ag

DonggeLiu · 2025-03-03T02:11:01Z

agent/crash_analyzer.py

@@ -2,7 +2,7 @@
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# You may obtain a copy of the License a


DonggeLiu · 2025-03-03T02:29:09Z

agent/crash_analyzer.py

+      with open(os.path.join(generated_project_path, 'Dockerfile'), 'a') as f:
+        f.write('\nENV FUZZING_LANGUAGE={run_result.benchmark.language}\n'
+                '\nRUN sed -i.bak \'1i export CFLAGS="${CFLAGS} -g"\' '
+                '/src/build.sh\n'


Would it be simpler to modify CFLAGS in dockerfile? E.g., ENV CFLAGS="${CFLAGS} -g"?

Do we need to add -g to CXXFLAGS too?

DonggeLiu · 2025-03-03T02:33:33Z

agent/crash_analyzer.py

+                 trial=self.trial)
+    return prompt_builder.DefaultTemplateBuilder(self.llm).build([])
+
+  def _create_ossfuzz_project_with_lldb(self,


I reckon this function is derived from create_ossfuzz_project().
Some nits:

oss_fuzz_checkout.py is a better place for this function.
That file is designed to encapsulate and handle all OSS-Fuzz related functionalities, so that OSS-Fuzz-Gen does not have to know/consider them. Same with _prepare_project_image() in LLDBTool below. We plan to relocate create_ossfuzz_project() too soon.

Please try to avoid code duplication.
We will really appreciate this because if we need to modify create_ossfuzz_project() later, we don't have to remember to repeats the same steps for this function. For example, given your function's main task is appending lines to the Dockerfile, could we first call function create_ossfuzz_project() to create a new project, then add those new lines to the Dockerfile of the new project?

DonggeLiu · 2025-03-03T03:09:25Z

agent/crash_analyzer.py

+          '\nCOPY agent-build.sh /src/build.sh\n'
+          '\nENV FUZZING_LANGUAGE={run_result.benchmark.language}\n'
+          '\nRUN sed -i.bak \'1i export CFLAGS="${CFLAGS} -g"\' /src/build.sh\n'
+          '\nRUN apt-get update && apt-get install -y lldb\n')


I reckon this is another code block we can remove if we first call create_ossfuzz_project() to create a new project, and then add these lines to the Dockerfile in the new project.
We don't have to worry about agent-build.sh in that case.

DonggeLiu · 2025-03-03T03:27:36Z

agent/crash_analyzer.py

+    for command in self._parse_tags(response, 'bash'):
+      prompt_text += self._format_bash_execution_result(
+          tool.execute(command), previous_prompt=prompt) + '\n'
+      prompt.add_problem(prompt_text)


I reckon you overwrite _container_handle_bash_command() because you want to call add_problem() instead of append() for OpenAIPrompt.

Would it be simpler to implement the append() function in OpenAIPrompt so that we don't have to overwrite this function?
This will make the code more transparent between different models and largely lower the complexity when we read/modify the code in the future.

For example:

def append(self, text: str, role:str='user') -> None: """Constructs the prompt problem in the required format.""" self._prompt.append({ 'role': role, 'content': text, })

Or append the text to an existing role-content pair, whichever is better.

DonggeLiu · 2025-03-03T06:18:16Z

tool/lldb_tool.py

+  def _prepare_project_image(self) -> str:
+    """Prepares the project's OSS-Fuzz docker image and returns the image name.
+    """
+    image_name = f'gcr.io/oss-fuzz/{self.project}'


Would it be better to use gcr.io/oss-fuzz/{self.project}-lldb to distinguish it from the normal image without lldb?

DonggeLiu · 2025-03-03T06:19:37Z

tool/lldb_tool.py

+      logger.info('Successfully build project image for %s', self.project)
+      return image_name
+    except sp.CalledProcessError:
+      logger.info('Failed to build image for %s', self.project)


Again, would it be clearer to put this in oss_fuzz_checkout, or reuse some of its code?
Ideally, OSS-Fuzz-Gen, particularly agents, don't have to know OSS-Fuzz details.

DonggeLiu · 2025-03-03T06:23:33Z

tool/lldb_tool.py

+
+  def _execute_command(self,
+                       command: list[str],
+                       in_container: bool = False) -> sp.CompletedProcess:


If these functions are the same as _execute_command and _execute_command_in_container, could you make lldb_tool inherit from ProjectContainerTool so that we don't have to repeat them?

DonggeLiu · 2025-03-03T06:24:54Z

tool/lldb_tool.py

+          result.stderr)
+    return result
+
+  def _start_docker_container(self) -> str:


Could we reuse

oss-fuzz-gen/tool/container_tool.py

Line 108 in f08dd92

def _start_docker_container(self) -> str:

DonggeLiu · 2025-03-03T06:26:03Z

tool/lldb_tool.py

+    process.args = command
+    return process
+
+  def terminate(self) -> bool:


Reuse or inherit from ProjectContainerTool if possible : )

DonggeLiu · 2025-03-03T06:34:38Z

Hi @maoyixie, the code looks good in general, I've left some comments above, please take a look at your convenience : )
Also feel free to schedule a meeting, I am happy to go through them with you.

DonggeLiu · 2025-03-10T04:10:57Z

tool/container_tool.py

+  def __init__(self,
+               benchmark: Benchmark,
+               name: str = '',
+               project_name: str = '') -> None:


Is anything changed here?

DonggeLiu · 2025-03-10T04:17:11Z

tool/container_tool.py

    super().__init__(benchmark, name)
-    self.image_name = self._prepare_project_image()
+    project_name = project_name or benchmark.project


self.project_name, so that _prepare_project_image don't have to take this parameter.
Otherwise you need to fix all usages of _prepare_project_image.

DonggeLiu · 2025-03-13T11:06:44Z

Thanks @maoyixie for the hardwork!
I will run an exp below, hope it works as expected : )

DonggeLiu · 2025-03-13T11:06:56Z

/gcbrun exp -n mx -ag

DonggeLiu · 2025-03-13T11:38:48Z

https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2025-03-13-814-mx-comparison/index.html

DonggeLiu · 2025-03-13T23:32:45Z

@maoyixie analyzer failed on cloud experiment:
https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2025-03-13-814-mx-comparison/sample/output-libfuse-af_gb_alloc_data/02.html

Not sure if you have access to cloud build, I will paste the error message below:

2025-03-13 11:31:08 [Trial ID: 02] INFO [logger.info]: Executing Crash Analyzer
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/workspace/ofg/agent/base_agent.py", line 220, in <module>
    BaseAgent.cloud_main()
  File "/workspace/ofg/agent/base_agent.py", line 206, in cloud_main
    result = agent.execute(result_history)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ofg/agent/crash_analyzer.py", line 132, in execute
    evaluator_lib.Evaluator.create_ossfuzz_project_with_lldb(
  File "/workspace/ofg/experiment/evaluator.py", line 332, in create_ossfuzz_project_with_lldb
    Evaluator.create_ossfuzz_project(benchmark, name, target_file,
  File "/workspace/ofg/experiment/evaluator.py", line 295, in create_ossfuzz_project
    shutil.copyfile(
  File "/usr/lib/python3.11/shutil.py", line 256, in copyfile
    with open(src, 'rb') as fsrc:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/experiment/results/output-libfuse-af_gb_alloc_data/fuzz_targets/02.fuzz_target'

This is likely due to missing the follow line in your def execute(), could you please double-check and fix it?

oss-fuzz-gen/agent/prototyper.py

Line 428 in a3094bd

WorkDirs(self.args.work_dirs.base)

This is likely unproducible in local experiments, where those dirs (e.g., output-libfuse-af_gb_alloc_data/fuzz_targets/) were created earlier. Cloud builds start with a fresh env so it requires this line to recreate them.

DonggeLiu · 2025-03-14T04:20:01Z

/gcbrun exp -n mx -ag

DonggeLiu · 2025-03-14T05:50:57Z

/gcbrun exp -n mx1 -ag

…ew agent

DonggeLiu · 2025-03-14T08:58:29Z

/gcbrun exp -n mx1 -ag

DonggeLiu · 2025-03-20T12:10:44Z

/gcbrun exp -n mx1 -ag

DonggeLiu · 2025-03-20T12:28:44Z

/gcbrun exp -n mx1 -ag

DonggeLiu · 2025-03-20T20:58:43Z

/gcbrun exp -n mx1 -ag

arthurscchan · 2025-03-21T02:49:35Z

@maoyixie I have also fixed the chat_llm approach in #902. See if it matches part of the fixing in here. I think either way is good to me since I also see you trying to refractor the model specifically for ChatGPT.

@DonggeLiu @DavidKorczynski I am OK with either ways, I will leave #902 in draft until this is merged and see if additional fixes for chat_llm implementation is needed.

DonggeLiu · 2025-03-21T05:10:32Z

/gcbrun exp -n mx1 -ag

DonggeLiu · 2025-03-21T09:29:22Z

/gcbrun exp -n mx -ag

maoyixie added the Experiment-only A PR only to run experiments, do not merge it to main. label Feb 25, 2025

maoyixie requested a review from DonggeLiu March 2, 2025 14:04

DonggeLiu requested changes Mar 3, 2025

View reviewed changes

DonggeLiu reviewed Mar 10, 2025

View reviewed changes

fix typo

34b8d65

maoyixie mentioned this pull request Mar 13, 2025

Add chat_llm implementation for GPT model #807

Merged

maoyixie added 20 commits March 13, 2025 11:23

crash analyzer agent prototype

4eb99d2

add _execute_agent_cloud in crash analyzer

3d75099

save cloud testcase

eb556a7

support gpt chat model

c11a2ae

update lldb _handle_conclusion

ef90d46

update prompt, compile project

55fcbed

update CrashResult

50d379e

temp, revert later

98e216c

try to fix gpt model

f7096bd

try to fix gpt model

c10bd27

todo: fix openai prompt

705aa06

add tutorial with add_problem instead of append

6590fda

try to fix invalid type for message in gpt(todo)

8177330

matain gpt chat session, ugly fix invalid message

4d8e354

update

da6d96e

fix agent name

3e59be8

fix crash analyzer no benchmark

8e8f5e8

reformat

b589684

delete unused import

d10587b

fix line to long

31ac644

fix loss t

ca8f522

maoyixie requested a review from DonggeLiu March 13, 2025 07:24

DonggeLiu added 2 commits March 14, 2025 15:17

Merge branch 'main' into dev9

0ebad67

Fix 'No such file or directory' error

edd3a74

Fix 'NameError: name 'WorkDirs' is not defined'

7e6a99d

maoyixie and others added 3 commits March 14, 2025 15:15

Merge branch 'main' into dev9

2ffc79f

fix lint

a4f30a4

Write fuzz target and build script to the expected locations in the n…

406a1aa

…ew agent

DonggeLiu removed the Experiment-only A PR only to run experiments, do not merge it to main. label Mar 19, 2025

Fix artifact blobing

0f0d5d6

Merge branch 'main' into dev9

cf3b091

DavidKorczynski mentioned this pull request Mar 20, 2025

llm: Fix chat_llm for OpenAI model #902

Draft

fix author

1671da2

DonggeLiu added 2 commits March 21, 2025 16:05

Fix blobbing

af2ccfc

Various bug fixes

6fafd6c

Fix initialization

4d21afc

code move to _copy_cloud_artifact

e8df9e2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash Analyzer Agent #814

Crash Analyzer Agent #814

maoyixie commented Feb 25, 2025 •

edited

Loading

DonggeLiu commented Mar 3, 2025

DonggeLiu commented Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu Mar 3, 2025

DonggeLiu commented Mar 3, 2025

DonggeLiu Mar 10, 2025

DonggeLiu Mar 10, 2025

DonggeLiu commented Mar 13, 2025

DonggeLiu commented Mar 13, 2025

DonggeLiu commented Mar 13, 2025

DonggeLiu commented Mar 13, 2025 •

edited

Loading

DonggeLiu commented Mar 14, 2025

DonggeLiu commented Mar 14, 2025

DonggeLiu commented Mar 14, 2025

DonggeLiu commented Mar 20, 2025

DonggeLiu commented Mar 20, 2025

DonggeLiu commented Mar 20, 2025

arthurscchan commented Mar 21, 2025 •

edited

Loading

DonggeLiu commented Mar 21, 2025

DonggeLiu commented Mar 21, 2025

Crash Analyzer Agent #814

Are you sure you want to change the base?

Crash Analyzer Agent #814

Conversation

maoyixie commented Feb 25, 2025 • edited Loading

DonggeLiu commented Mar 3, 2025

DonggeLiu commented Mar 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DonggeLiu commented Mar 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DonggeLiu commented Mar 13, 2025

DonggeLiu commented Mar 13, 2025

DonggeLiu commented Mar 13, 2025

DonggeLiu commented Mar 13, 2025 • edited Loading

DonggeLiu commented Mar 14, 2025

DonggeLiu commented Mar 14, 2025

DonggeLiu commented Mar 14, 2025

DonggeLiu commented Mar 20, 2025

DonggeLiu commented Mar 20, 2025

DonggeLiu commented Mar 20, 2025

arthurscchan commented Mar 21, 2025 • edited Loading

DonggeLiu commented Mar 21, 2025

DonggeLiu commented Mar 21, 2025

maoyixie commented Feb 25, 2025 •

edited

Loading

DonggeLiu commented Mar 13, 2025 •

edited

Loading

arthurscchan commented Mar 21, 2025 •

edited

Loading