Crash Analyzer Agent #814

maoyixie · 2025-02-25T12:47:57Z

This PR mainly implements a crash analyzer that can interact with LLDB in the multi-agent framework, and supports GPT. In addition, this PR attempts to fix the problem of not replacing the fuzz target and build script. This PR is under testing. The main logic is no longer changing, and minor bugs are being fixed.

TODO:

Optimize the process of agent interaction with LLDB.
Solve the problem of missing debugging information for some projects.
Try to add LLM-based static methods to enhance the crash analyzer.

DonggeLiu · 2025-03-03T00:16:26Z

Thanks again for the pushing the code fixing the conflicts, @maoyixie!
I will make sure to read your code today.

Before that, let me start an experiment below so that we can see its results together later : )
There seems to be another conflict between your chat_llm with the one on main branch. The major difference seems to be:

self.conversation_history.extend(prompt.get())

DonggeLiu · 2025-03-03T00:17:23Z

/gcbrun exp -n mx -ag

DonggeLiu · 2025-03-03T02:11:01Z

agent/crash_analyzer.py

@@ -2,7 +2,7 @@
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
+# You may obtain a copy of the License a


DonggeLiu · 2025-03-03T02:29:09Z

agent/crash_analyzer.py

+      with open(os.path.join(generated_project_path, 'Dockerfile'), 'a') as f:
+        f.write('\nENV FUZZING_LANGUAGE={run_result.benchmark.language}\n'
+                '\nRUN sed -i.bak \'1i export CFLAGS="${CFLAGS} -g"\' '
+                '/src/build.sh\n'


Would it be simpler to modify CFLAGS in dockerfile? E.g., ENV CFLAGS="${CFLAGS} -g"?

Do we need to add -g to CXXFLAGS too?

DonggeLiu · 2025-03-03T02:33:33Z

agent/crash_analyzer.py

+                 trial=self.trial)
+    return prompt_builder.DefaultTemplateBuilder(self.llm).build([])
+
+  def _create_ossfuzz_project_with_lldb(self,


I reckon this function is derived from create_ossfuzz_project().
Some nits:

oss_fuzz_checkout.py is a better place for this function.
That file is designed to encapsulate and handle all OSS-Fuzz related functionalities, so that OSS-Fuzz-Gen does not have to know/consider them. Same with _prepare_project_image() in LLDBTool below. We plan to relocate create_ossfuzz_project() too soon.

Please try to avoid code duplication.
We will really appreciate this because if we need to modify create_ossfuzz_project() later, we don't have to remember to repeats the same steps for this function. For example, given your function's main task is appending lines to the Dockerfile, could we first call function create_ossfuzz_project() to create a new project, then add those new lines to the Dockerfile of the new project?

DonggeLiu · 2025-03-03T03:09:25Z

agent/crash_analyzer.py

+          '\nCOPY agent-build.sh /src/build.sh\n'
+          '\nENV FUZZING_LANGUAGE={run_result.benchmark.language}\n'
+          '\nRUN sed -i.bak \'1i export CFLAGS="${CFLAGS} -g"\' /src/build.sh\n'
+          '\nRUN apt-get update && apt-get install -y lldb\n')


I reckon this is another code block we can remove if we first call create_ossfuzz_project() to create a new project, and then add these lines to the Dockerfile in the new project.
We don't have to worry about agent-build.sh in that case.

DonggeLiu · 2025-03-03T03:27:36Z

agent/crash_analyzer.py

+    for command in self._parse_tags(response, 'bash'):
+      prompt_text += self._format_bash_execution_result(
+          tool.execute(command), previous_prompt=prompt) + '\n'
+      prompt.add_problem(prompt_text)


I reckon you overwrite _container_handle_bash_command() because you want to call add_problem() instead of append() for OpenAIPrompt.

Would it be simpler to implement the append() function in OpenAIPrompt so that we don't have to overwrite this function?
This will make the code more transparent between different models and largely lower the complexity when we read/modify the code in the future.

For example:

def append(self, text: str, role:str='user') -> None: """Constructs the prompt problem in the required format.""" self._prompt.append({ 'role': role, 'content': text, })

Or append the text to an existing role-content pair, whichever is better.

DonggeLiu · 2025-03-03T06:18:16Z

tool/lldb_tool.py

+  def _prepare_project_image(self) -> str:
+    """Prepares the project's OSS-Fuzz docker image and returns the image name.
+    """
+    image_name = f'gcr.io/oss-fuzz/{self.project}'


Would it be better to use gcr.io/oss-fuzz/{self.project}-lldb to distinguish it from the normal image without lldb?

DonggeLiu · 2025-03-03T06:19:37Z

tool/lldb_tool.py

+      logger.info('Successfully build project image for %s', self.project)
+      return image_name
+    except sp.CalledProcessError:
+      logger.info('Failed to build image for %s', self.project)


Again, would it be clearer to put this in oss_fuzz_checkout, or reuse some of its code?
Ideally, OSS-Fuzz-Gen, particularly agents, don't have to know OSS-Fuzz details.

DonggeLiu · 2025-03-03T06:23:33Z

tool/lldb_tool.py

+
+  def _execute_command(self,
+                       command: list[str],
+                       in_container: bool = False) -> sp.CompletedProcess:


If these functions are the same as _execute_command and _execute_command_in_container, could you make lldb_tool inherit from ProjectContainerTool so that we don't have to repeat them?

DonggeLiu · 2025-03-03T06:24:54Z

tool/lldb_tool.py

+          result.stderr)
+    return result
+
+  def _start_docker_container(self) -> str:


Could we reuse

oss-fuzz-gen/tool/container_tool.py

Line 108 in f08dd92

def _start_docker_container(self) -> str:

DonggeLiu · 2025-03-03T06:26:03Z

tool/lldb_tool.py

+    process.args = command
+    return process
+
+  def terminate(self) -> bool:


Reuse or inherit from ProjectContainerTool if possible : )

DonggeLiu · 2025-03-03T06:34:38Z

Hi @maoyixie, the code looks good in general, I've left some comments above, please take a look at your convenience : )
Also feel free to schedule a meeting, I am happy to go through them with you.

DonggeLiu · 2025-03-10T04:10:57Z

tool/container_tool.py

+  def __init__(self,
+               benchmark: Benchmark,
+               name: str = '',
+               project_name: str = '') -> None:


Is anything changed here?

DonggeLiu · 2025-03-10T04:17:11Z

tool/container_tool.py

    super().__init__(benchmark, name)
-    self.image_name = self._prepare_project_image()
+    project_name = project_name or benchmark.project


self.project_name, so that _prepare_project_image don't have to take this parameter.
Otherwise you need to fix all usages of _prepare_project_image.

DonggeLiu · 2025-03-13T11:06:44Z

Thanks @maoyixie for the hardwork!
I will run an exp below, hope it works as expected : )

DonggeLiu · 2025-03-13T11:06:56Z

/gcbrun exp -n mx -ag

DonggeLiu · 2025-03-13T11:38:48Z

https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2025-03-13-814-mx-comparison/index.html

DonggeLiu · 2025-03-13T23:32:45Z

@maoyixie analyzer failed on cloud experiment:
https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2025-03-13-814-mx-comparison/sample/output-libfuse-af_gb_alloc_data/02.html

Not sure if you have access to cloud build, I will paste the error message below:

2025-03-13 11:31:08 [Trial ID: 02] INFO [logger.info]: Executing Crash Analyzer
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/workspace/ofg/agent/base_agent.py", line 220, in <module>
    BaseAgent.cloud_main()
  File "/workspace/ofg/agent/base_agent.py", line 206, in cloud_main
    result = agent.execute(result_history)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ofg/agent/crash_analyzer.py", line 132, in execute
    evaluator_lib.Evaluator.create_ossfuzz_project_with_lldb(
  File "/workspace/ofg/experiment/evaluator.py", line 332, in create_ossfuzz_project_with_lldb
    Evaluator.create_ossfuzz_project(benchmark, name, target_file,
  File "/workspace/ofg/experiment/evaluator.py", line 295, in create_ossfuzz_project
    shutil.copyfile(
  File "/usr/lib/python3.11/shutil.py", line 256, in copyfile
    with open(src, 'rb') as fsrc:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/experiment/results/output-libfuse-af_gb_alloc_data/fuzz_targets/02.fuzz_target'

This is likely due to missing the follow line in your def execute(), could you please double-check and fix it?

oss-fuzz-gen/agent/prototyper.py

Line 428 in a3094bd

WorkDirs(self.args.work_dirs.base)

This is likely unproducible in local experiments, where those dirs (e.g., output-libfuse-af_gb_alloc_data/fuzz_targets/) were created earlier. Cloud builds start with a fresh env so it requires this line to recreate them.

DonggeLiu · 2025-03-14T04:20:01Z

/gcbrun exp -n mx -ag

DonggeLiu · 2025-03-14T05:50:57Z

/gcbrun exp -n mx1 -ag

DonggeLiu · 2025-03-14T08:58:29Z

/gcbrun exp -n mx1 -ag

DonggeLiu · 2025-03-20T12:10:44Z

/gcbrun exp -n mx1 -ag

DonggeLiu · 2025-03-20T12:28:44Z

/gcbrun exp -n mx1 -ag

DonggeLiu · 2025-03-20T20:58:43Z

/gcbrun exp -n mx1 -ag

arthurscchan · 2025-03-21T02:49:35Z

@maoyixie I have also fixed the chat_llm approach in #902. See if it matches part of the fixing in here. I think either way is good to me since I also see you trying to refractor the model specifically for ChatGPT.

@DonggeLiu @DavidKorczynski I am OK with either ways, I will leave #902 in draft until this is merged and see if additional fixes for chat_llm implementation is needed.

DonggeLiu · 2025-03-21T05:10:32Z

/gcbrun exp -n mx1 -ag

DonggeLiu · 2025-03-21T09:29:22Z

/gcbrun exp -n mx -ag

maoyixie · 2025-04-07T06:38:07Z

/gcbrun exp -n mx -ag

DonggeLiu · 2025-04-10T03:35:12Z

Hi @maoyixie, I am unsure if you could see the cloud build error, so I found and pasted it below:

ERROR: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: dockerfile parse error line 35: unknown instruction: APT-GET
ERROR:__main__:Docker build failed.

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/workspace/ofg/agent/base_agent.py", line 220, in <module>
    BaseAgent.cloud_main()
  File "/workspace/ofg/agent/base_agent.py", line 206, in cloud_main
    result = agent.execute(result_history)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ofg/agent/crash_analyzer.py", line 176, in execute
    self.analyze_tool = LLDBTool(benchmark,
                        ^^^^^^^^^^^^^^^^^^^
  File "/workspace/ofg/tool/lldb_tool.py", line 33, in __init__
    super().__init__(benchmark, name, project_name)
  File "/workspace/ofg/tool/container_tool.py", line 34, in __init__
    self.image_name = self._prepare_project_image(self.project_name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/ofg/tool/container_tool.py", line 50, in _prepare_project_image
    raise Exception(f'Failed to build image for {project_name}')
Exception: Failed to build image for hoextdown-hoedown_document_render_inline-hoedown_fuzzer-lldb-01

This is likely caused by these lines.

maoyixie · 2025-05-28T07:30:15Z

/gcbrun exp -n my -m vertex_ai_gemini-2-5-pro-chat -ag

DonggeLiu

Exp failed

DonggeLiu · 2025-05-29T12:51:40Z

agent/crash_analyzer.py

+    if not os.path.exists(last_result.artifact_path):
+      logger.error('Artifact path %s does not exist',
+                   last_result.artifact_path,
+                   trial=self.trial)


Good news:
We no longer see the error /workspace/crash-XXXX does not exist, hence the artifact seems to be correctly uploaded.

Bad news:
This error log was printed:

Already have image (with digest): gcr.io/cloud-builders/docker 2025-05-28 09:29:13 [Trial ID: 00] INFO [logger.info]: Checkign if we should use local FI 2025-05-28 09:29:13 [Trial ID: 00] INFO [logger.info]: This does not require a local FI. 2025-05-28 09:29:13 [Trial ID: 07] INFO [logger.info]: Executing Crash Analyzer 2025-05-28 09:29:13 [Trial ID: 07] ERROR [logger.error]: Artifact path /experiment/results/output-ada-url-ada_can_parse_with_base/artifacts/07.fuzz_target-F0-07/crash-5c013da7b11b7ccb2c437239fbcdbf4c53b20655 does not exist Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/workspace/ofg/agent/base_agent.py", line 280, in <module> BaseAgent.cloud_main() File "/workspace/ofg/agent/base_agent.py", line 266, in cloud_main result = agent.execute(result_history) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/ofg/agent/crash_analyzer.py", line 172, in execute evaluator_lib.Evaluator.create_ossfuzz_project_with_lldb( File "/workspace/ofg/experiment/evaluator.py", line 321, in create_ossfuzz_project_with_lldb shutil.copyfile( File "/usr/lib/python3.11/shutil.py", line 256, in copyfile with open(src, 'rb') as fsrc: ^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/experiment/results/output-ada-url-ada_can_parse_with_base/artifacts/07.fuzz_target-F0-07/crash-5c013da7b11b7ccb2c437239fbcdbf4c53b20655'

Would the artifact be at /workspace/crash-XXXX, because the cloud build step copied the artifact there and crash analyzer no longer executes self._copy_cloud_artifact(last_result.artifact_path)?

IIRC, we discussed that it's better to place the artifact at last_result.artifact_path (in this case, /experiment/results/output-ada-url-ada_can_parse_with_base/artifacts/07.fuzz_target-F0-07/crash-5c013da7b11b7ccb2c437239fbcdbf4c53b20655), because it is:

Transparent: It will be the same path for both local and cloud exp.

Straightforward: we can always trust last_result.artifact_path.

Thinking more about this:
Would it be easier to directly download the artifact to last_result.artifact_path (instead of /workspace/artifact-XXX) in the cloud build step?
The agent does not need to copy it, it can safely assume the artifact is at last_result.artifact_path for both local and cloud setup.
We may need to add a step to create the parent dir of last_result.artifact_path in the cloud build before download it.

DonggeLiu · 2025-05-29T12:54:46Z

common/cloud_builder.py

+                    f'/workspace/{os.path.basename(artifact_path)}'
+                ],
+                'allowFailure': True,
+            },


For example, before this step, we have something like:

{ 'name': 'gcr.io/cloud-builders/gsutil', 'entrypoint': 'bash', 'args': [ '-c', f'mkdir -p {os.path.dirname(artifact_path)}' ] },

…ath does not exist

maoyixie · 2025-06-01T06:21:32Z

/gcbrun exp -n my -m vertex_ai_gemini-2-5-pro-chat -ag

maoyixie · 2025-06-01T08:09:24Z

/gcbrun exp -n my1 -m vertex_ai_gemini-2-5-pro-chat -ag

DonggeLiu · 2025-06-01T23:43:12Z

/gcbrun exp -n my -m vertex_ai_gemini-2-5-pro-chat -ag

… `/workspace` instead.

DonggeLiu · 2025-06-02T02:00:14Z

/gcbrun exp -n my2 -m vertex_ai_gemini-2-5-pro-chat -ag

DonggeLiu · 2025-06-02T02:58:39Z

/gcbrun exp -n my3 -m vertex_ai_gemini-2-5-pro-chat -ag

DonggeLiu · 2025-06-02T04:06:46Z

/gcbrun exp -n my4 -m vertex_ai_gemini-2-5-pro-chat -ag

DonggeLiu · 2025-06-02T05:03:39Z

Crash Analyzer is finally working again:
https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2025-06-02-814-my4-comparison/sample/output-hoextdown-hoedown_document_render_inline/02.html

LLDB output on the report is always empty, even though it seems LLM has received the output:
https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2025-06-02-814-my4-comparison/sample/output-hoextdown-hoedown_document_render_inline/02.html#:~:text=The%20output%20%60(hoedown_document%20*)%20%241%20%3D%200x000056215b000000%60%20in%20frame%20%238%20(LLVMFuzzerTestOneInput)%20shows%20that%20the%20%60document%60%20pointer%20has%20the%20same%20high%2C%20likely%20invalid%2C%20address%20as%20seen%20in%20frame%20%230.%20This%20suggests%20that%20%60hoedown_document_new%60%20itself%20is%20returning%20this%20problematic%20pointer.

DonggeLiu · 2025-06-02T05:05:11Z

I will merge this to avoid further conflicts.
We can fix output parsing in a new PR.

This PR mainly implements a crash analyzer that can interact with LLDB in the multi-agent framework, and supports GPT. In addition, this PR attempts to fix the problem of not replacing the fuzz target and build script. This PR is under testing. The main logic is no longer changing, and minor bugs are being fixed. TODO: Optimize the process of agent interaction with LLDB. Solve the problem of missing debugging information for some projects. Try to add LLM-based static methods to enhance the crash analyzer. --------- Co-authored-by: Dongge Liu <[email protected]>

maoyixie added the Experiment-only A PR only to run experiments, do not merge it to main. label Feb 25, 2025

maoyixie requested a review from DonggeLiu March 2, 2025 14:04

DonggeLiu requested changes Mar 3, 2025

View reviewed changes

DonggeLiu reviewed Mar 10, 2025

View reviewed changes

maoyixie mentioned this pull request Mar 13, 2025

Add chat_llm implementation for GPT model #807

Merged

maoyixie force-pushed the dev9 branch from 20b810c to d52b48b Compare March 13, 2025 04:15

maoyixie requested a review from DonggeLiu March 13, 2025 07:24

DonggeLiu removed the Experiment-only A PR only to run experiments, do not merge it to main. label Mar 19, 2025

DavidKorczynski mentioned this pull request Mar 20, 2025

llm: Fix chat_llm for OpenAI model #902

Merged

This was referenced Mar 27, 2025

Refactor Result Classes with Unified Data Container and Improved Extensibility #941

Closed

Refactor Result Classes with Unified Data Container and Improved Extensibility #948

Open

maoyixie added 8 commits May 28, 2025 14:38

optimize lldb prompt

c59e934

fix git rebase

21a2173

fix model

3f671b6

fix artifact_url missing

1d88c95

lint

5d1331d

try to fix cloud exp file not found failure

c45197c

possible reason

d01e21b

more debug info

98b8924

maoyixie force-pushed the dev9 branch from 22c9aca to 98b8924 Compare May 28, 2025 06:39

upload artifact to gcs

8bde106

DonggeLiu reviewed May 29, 2025

View reviewed changes

cp to artifact_path, ensure artifact_path consistency, fix artifact p…

a79b121

…ath does not exist

allow artifact mkdir failure

1e4f62f

mount /experiment dir, which stores the crash artifact.

4d7c8db

/experiment will not exist after each step, saving it to persistent…

9e6c822

… `/workspace` instead.

Allow cp artifact fail (for non-crash analyzer agents)

2100a11

Do not delete the work_dir, which may contain the artifact.

a2d0186

DonggeLiu self-requested a review June 2, 2025 05:05

DonggeLiu approved these changes Jun 2, 2025

View reviewed changes

DonggeLiu merged commit 0acc389 into google:main Jun 2, 2025
6 checks passed

Crash Analyzer Agent #814

Crash Analyzer Agent #814

Conversation

maoyixie commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DonggeLiu commented Mar 3, 2025

Uh oh!

DonggeLiu commented Mar 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DonggeLiu commented Mar 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DonggeLiu commented Mar 13, 2025

Uh oh!

DonggeLiu commented Mar 13, 2025

Uh oh!

DonggeLiu commented Mar 13, 2025

Uh oh!

DonggeLiu commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DonggeLiu commented Mar 14, 2025

Uh oh!

DonggeLiu commented Mar 14, 2025

Uh oh!

DonggeLiu commented Mar 14, 2025

Uh oh!

DonggeLiu commented Mar 20, 2025

Uh oh!

DonggeLiu commented Mar 20, 2025

Uh oh!

DonggeLiu commented Mar 20, 2025

Uh oh!

arthurscchan commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DonggeLiu commented Mar 21, 2025

Uh oh!

DonggeLiu commented Mar 21, 2025

Uh oh!

maoyixie commented Apr 7, 2025

Uh oh!

DonggeLiu commented Apr 10, 2025

Uh oh!

maoyixie commented May 28, 2025

Uh oh!

DonggeLiu left a comment

Choose a reason for hiding this comment

Uh oh!

DonggeLiu May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maoyixie commented Jun 1, 2025

maoyixie commented Feb 25, 2025 •

edited

Loading

DonggeLiu commented Mar 13, 2025 •

edited

Loading

arthurscchan commented Mar 21, 2025 •

edited

Loading

DonggeLiu May 29, 2025 •

edited

Loading

DonggeLiu commented Jun 2, 2025 •

edited

Loading