Sancov based ctx/ngram LibAFL fuzzers #1956

tokatoka · 2024-02-21T18:36:44Z

Hello, fuzzbench team.

We implemented ctx and ngram coverage based on sanitizer coverage.
(libafl_ctx_large_map, libafl_ctx_mid_map, libafl_ctx_small_map, libafl_ngram_large_map, libafl_ngram_mid_map, libafl_ngram_small_map)
The previous implementation was based on AFL's llvm pass which has negative impact on performance.

Therefore we want to compare how this new implementation compares to the baseline.
Both ctx, ngram has 3 variants depending on the map size we use.

tokatoka · 2024-02-23T12:22:27Z

This one is ready

Ths command is
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-02-23-libafl --fuzzers libafl_ctx_large_map libafl_ctx_mid_map libafl_ctx_small_map libafl_ngram_large_map libafl_ngram_mid_map libafl_ngram_small_map

@DonggeLiu
Could you run this experiment? :)

DonggeLiu · 2024-02-25T23:30:06Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-02-26-libafl --fuzzers libafl_ctx_large_map libafl_ctx_mid_map libafl_ctx_small_map libafl_ngram_large_map libafl_ngram_mid_map libafl_ngram_small_map

DonggeLiu · 2024-02-25T23:31:25Z

Ops, could you please add a dummy change to enable PR experiments?
Thanks :)

tokatoka · 2024-02-28T15:35:20Z

hi i made the change also i added the more recent libafl to compare as the baseline

the command would be

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-02-23-libafl --fuzzers libafl_ctx_large_map libafl_ctx_mid_map libafl_ctx_small_map libafl_ngram_large_map libafl_ngram_mid_map libafl_ngram_small_map libafl_280224

DonggeLiu · 2024-02-28T21:15:59Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-02-29-libafl --fuzzers libafl_ctx_large_map libafl_ctx_mid_map libafl_ctx_small_map libafl_ngram_large_map libafl_ngram_mid_map libafl_ngram_small_map libafl_280224

tokatoka · 2024-03-06T18:04:20Z

Hello @DonggeLiu
Could you do another run?

In this change, we

- make LibAFL fuzzers base on the same commit as #1840 so we can make a fair comparison (sorry previous run was based on the latest LibAFL, it is my fault :man_facepalming: 
- add value-profile (named as libafl_fuzzbench_vp_alter) and ngram-8 fuzzers, both are sancov based new implementation

The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-06-libafl --fuzzers libafl_fuzzbench_ngram4 libafl_fuzzbench_ngram8 libafl_fuzzbench_ctx libafl_fuzzbench_vp_alter

DonggeLiu · 2024-03-11T03:13:28Z

~~Thanks @tokatoka, unfortunately we are making some change to FuzzBench at this moment and may need a few days before it is ready for experiments :)~~

It seems to be ready now, I will do it below.

DonggeLiu · 2024-03-11T03:22:07Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-03-11-libafl --fuzzers libafl_fuzzbench_ngram4 libafl_fuzzbench_ngram8 libafl_fuzzbench_ctx libafl_fuzzbench_vp_alter

DonggeLiu · 2024-03-11T03:26:35Z

Experiment 2024-03-11-libafl data and results will be available later at:
The experiment data.
The experiment report.

tokatoka · 2024-04-12T15:43:42Z

this is done. thank you! 👍

…ibafl_exp_ctx

tokatoka · 2024-04-17T12:21:58Z

Hi @DonggeLiu. This is not the preparatory test for the long-run experiment that I was talking about last week, but it is additional experiment for our fuzzer comparison paper

In this experiment we want to evaluate the degree of two fuzzer's component's interference. (like if the component X and component Y of the fuzzer will have better or worse interaction)

Can you run the experiment for the next 5 fuzzers? The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-17-libafl --fuzzers libafl_fuzzbench_fast libafl_fuzzbench_fast_ngram4 libafl_fuzzbench_fast_value_profile libafl_fuzzbench_ngram libafl_fuzzbench_value_profile

DonggeLiu · 2024-04-17T22:19:44Z

Sure, could you please fix the presubmit failure? Thanks!

tokatoka · 2024-04-19T11:16:45Z

done!

DonggeLiu · 2024-04-22T03:34:39Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-22-libafl --fuzzers libafl_fuzzbench_fast libafl_fuzzbench_fast_ngram4 libafl_fuzzbench_fast_value_profile libafl_fuzzbench_ngram libafl_fuzzbench_value_profile

DonggeLiu · 2024-04-22T04:04:22Z

The experiment failed to launch because of invalid fuzzer name.
Maybe libafl_fuzzbench_ngram should be libafl_fuzzbench_ngram4 or libafl_fuzzbench_ngram8?

tokatoka · 2024-04-22T10:00:02Z

I'm sorry 😔
It is libafl_fuzzbench_ngram8

so
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-22-libafl --fuzzers libafl_fuzzbench_fast libafl_fuzzbench_fast_ngram4 libafl_fuzzbench_fast_value_profile libafl_fuzzbench_ngram8 libafl_fuzzbench_value_profile

DonggeLiu · 2024-04-22T23:47:51Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-libafl --fuzzers libafl_fuzzbench_fast libafl_fuzzbench_fast_ngram4 libafl_fuzzbench_fast_value_profile libafl_fuzzbench_ngram8 libafl_fuzzbench_value_profile

tokatoka · 2024-04-24T12:31:57Z

Hi. I adjusted the map size because previously it was using a map that was too large.

Can you run it again along with an additional fuzzer of libafl_fuzzbench_ngram4 ?

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-4-libafl --fuzzers libafl_fuzzbench_fast_ngram4 libafl_fuzzbench_ngram4

DonggeLiu · 2024-04-24T13:00:42Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-24-libafl --fuzzers libafl_fuzzbench_fast_ngram4 libafl_fuzzbench_ngram4

tokatoka · 2024-05-02T15:35:40Z

Hi @DonggeLiu , I waited for a few days, for the 2024-04-24-libafl experiment but the results do not look complete.
I downloaded the csv from here https://www.fuzzbench.com/reports/experimental/2024-04-24-libafl/data.csv.gz.
However, the result does not contain the result for 24 hours. and most experiments ends in 17.5 hours. I suspect it is because of out of resource since there was another different experiment conducted on the same day (2024-04-24-full-muttfuzz).

Could you run this again for me? thanks

DonggeLiu · 2024-05-03T00:44:02Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-03-libafl --fuzzers libafl_fuzzbench_fast_ngram4 libafl_fuzzbench_ngram4

DonggeLiu · 2024-05-03T00:50:19Z

Hi @DonggeLiu , I waited for a few days, for the 2024-04-24-libafl experiment but the results do not look complete. I downloaded the csv from here https://www.fuzzbench.com/reports/experimental/2024-04-24-libafl/data.csv.gz. However, the result does not contain the result for 24 hours. and most experiments ends in 17.5 hours. I suspect it is because of out of resource since there was another different experiment conducted on the same day (2024-04-24-full-muttfuzz).

Thanks for reporting this, @tokatoka! I've re-started the experiment in case it is flaky.
Usually, we expect the experiment to finish within four days. Please feel free to ping me if it takes longer.

While waiting for the new experiment to complete, did you have a chance to look into the run log of your experiment?
Sometimes, the fuzzer's run log may have clues, too.
I am curious to learn the reason and see if there is anything we can improve.

tokatoka · 2024-05-03T15:07:40Z

While waiting for the new experiment to complete, did you have a chance to look into the run log of your experiment?
Sometimes, the fuzzer's run log may have clues, too.

For the LibAFL log, we do not log anything, so I can only guess by the last corpus update time.

These days I almost always see an weird restart (or you could say vm preemption) with the results.
That is,

First the experiments starts.
Then after 10 ~ 12 hours, suddenly most instances stop, and then the result page will also revert back.
After that the instances start again, this time they will run for 23 hours.

For me, I'm fine as long as they give the complete result in 3.
However 3. didn't happen this time, as long as i can tell from the result, they didn't finish its 23-hours run.
When I quickly look through these results. https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-04-24-libafl/experiment-folders/
then the instances with bigger trial id (these are the restarted instances in the above 3.) somehow all stopped fuzzing around on 26th April 1PM to 2PM.
So I would assume something has happend during that period.

tokatoka · 2024-05-03T15:13:53Z

And on the other hand,
https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-04-24-full-muttfuzz

Oddly here, the last corpus time is at latest around 26th April's morning. So in theory, there should be no "out of resource" at the time of 26th April's afternoon. Because at this time, this experiment is already finished

DonggeLiu · 2024-05-04T00:51:40Z

However 3. didn't happen this time, as long as i can tell from the result, they didn't finish its 23-hours run. When I quickly look through these results. https://storage.googleapis.com/fuzzbench-data/index.html?prefix=2024-04-24-libafl/experiment-folders/ then the instances with bigger trial id (these are the restarted instances in the above 3.) somehow all stopped fuzzing around on 26th April 1PM to 2PM. So I would assume something has happend during that period.

Yes, this is indeed strange why no trial instances are running:

There is no trial instance of 2024-04-24-libafl, but the main dispatcher VM still exists:

NAME                         STATUS      CREATION_TIMESTAMP   PREEMPTIBLE  ZONE
d-2024-04-24-libafl          RUNNING     2024-04-25T00:03:46  FALSE        us-central1-c

The report shows the experiment has not finished: https://storage.googleapis.com/www.fuzzbench.com/reports/experimental/2024-04-24-libafl/index.html
The gcloud log shows it has not finished either:

I will kill d-2024-04-24-libafl and let's see how the new experiment goes.

DonggeLiu · 2024-05-04T00:59:28Z

BTW, this code ensures trial instances will eventually complete within 2 days:

fuzzbench/experiment/scheduler.py

Lines 301 to 312 in 162ca0c

    
           # Don't keep creating preemptible instances forever. Don't create 
        
           # them if the experiment has already taken a certain amount of time 
        
           # longer than the equivalent nonpreemptible experiment. 
        
           # *NOTE*: preemptible_window_passed is slightly broken. When 
        
           # the measurer uses this method it may produce slightly different 
        
           # results than the scheduler because the initial trials may be 
        
           # different. This is unlikely to happen in the real world. It is 
        
           # probably benign as well because the measurer may think the window 
        
           # end is slightly later than the scheduler. The effect of this will 
        
           # simply be that the measurer may measure for slightly longer than 
        
           # needed. 
        
           return False

If a trial was preempted after 1 day, then a non-preemptible instance will be used.
Not sure why they did not produce result in the last exp, though.

tokatoka · 2024-05-06T11:17:14Z

It's complete thank you 👍

tokatoka added 12 commits February 21, 2024 19:32

adding fuzzers

523a54e

small fix

9a4686b

fixes

7b7676b

add another map size variant

4715f30

fomatto

324c1b6

add adaptive

26b1f82

add adaptive

6d6fc6f

update commit

42af850

bring back small map

e88e347

c++14

37ea80a

another fix?

43844f0

make linter pass

d8576eb

tokatoka added 3 commits February 28, 2024 16:30

dummy comment

d5a81aa

add latest libafl

a3b3a2a

fix

c1b5460

tokatoka added 4 commits March 4, 2024 20:29

adding fuzzers

4c253cd

delete

097ecde

upd

d80b8df

value_profile_alternative_impl

0f0ead3

tokatoka closed this Apr 12, 2024

tokatoka added 3 commits April 16, 2024 14:02

Merge branch 'google:master' into libafl_exp_ctx

8802f42

add

2055966

Merge branch 'libafl_exp_ctx' of github.com:tokatoka/fuzzbench into l…

74c5da4

…ibafl_exp_ctx

tokatoka reopened this Apr 16, 2024

cx

4eb332a

FMT

853e047

tokatoka added 2 commits April 24, 2024 14:13

fix map size

f37a18d

fix

62052cd

DonggeLiu mentioned this pull request May 3, 2024

Adding muttfuzz and requesting an experiment #1967

Open

tokatoka closed this May 6, 2024

tokatoka deleted the libafl_exp_ctx branch November 19, 2024 16:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sancov based ctx/ngram LibAFL fuzzers #1956

Sancov based ctx/ngram LibAFL fuzzers #1956

tokatoka commented Feb 21, 2024 •

edited

Loading

tokatoka commented Feb 23, 2024 •

edited

Loading

DonggeLiu commented Feb 25, 2024

DonggeLiu commented Feb 25, 2024

tokatoka commented Feb 28, 2024

DonggeLiu commented Feb 28, 2024

tokatoka commented Mar 6, 2024 •

edited

Loading

DonggeLiu commented Mar 11, 2024 •

edited

Loading

DonggeLiu commented Mar 11, 2024

DonggeLiu commented Mar 11, 2024

tokatoka commented Apr 12, 2024

tokatoka commented Apr 17, 2024

DonggeLiu commented Apr 17, 2024

tokatoka commented Apr 19, 2024

DonggeLiu commented Apr 22, 2024

DonggeLiu commented Apr 22, 2024

tokatoka commented Apr 22, 2024

DonggeLiu commented Apr 22, 2024

tokatoka commented Apr 24, 2024

DonggeLiu commented Apr 24, 2024

tokatoka commented May 2, 2024 •

edited

Loading

DonggeLiu commented May 3, 2024

DonggeLiu commented May 3, 2024

tokatoka commented May 3, 2024 •

edited

Loading

tokatoka commented May 3, 2024

DonggeLiu commented May 4, 2024 •

edited

Loading

DonggeLiu commented May 4, 2024

tokatoka commented May 6, 2024

Sancov based ctx/ngram LibAFL fuzzers #1956

Sancov based ctx/ngram LibAFL fuzzers #1956

Conversation

tokatoka commented Feb 21, 2024 • edited Loading

tokatoka commented Feb 23, 2024 • edited Loading

DonggeLiu commented Feb 25, 2024

DonggeLiu commented Feb 25, 2024

tokatoka commented Feb 28, 2024

DonggeLiu commented Feb 28, 2024

tokatoka commented Mar 6, 2024 • edited Loading

DonggeLiu commented Mar 11, 2024 • edited Loading

DonggeLiu commented Mar 11, 2024

DonggeLiu commented Mar 11, 2024

tokatoka commented Apr 12, 2024

tokatoka commented Apr 17, 2024

DonggeLiu commented Apr 17, 2024

tokatoka commented Apr 19, 2024

DonggeLiu commented Apr 22, 2024

DonggeLiu commented Apr 22, 2024

tokatoka commented Apr 22, 2024

DonggeLiu commented Apr 22, 2024

tokatoka commented Apr 24, 2024

DonggeLiu commented Apr 24, 2024

tokatoka commented May 2, 2024 • edited Loading

DonggeLiu commented May 3, 2024

DonggeLiu commented May 3, 2024

tokatoka commented May 3, 2024 • edited Loading

tokatoka commented May 3, 2024

DonggeLiu commented May 4, 2024 • edited Loading

DonggeLiu commented May 4, 2024

tokatoka commented May 6, 2024

tokatoka commented Feb 21, 2024 •

edited

Loading

tokatoka commented Feb 23, 2024 •

edited

Loading

tokatoka commented Mar 6, 2024 •

edited

Loading

DonggeLiu commented Mar 11, 2024 •

edited

Loading

tokatoka commented May 2, 2024 •

edited

Loading

tokatoka commented May 3, 2024 •

edited

Loading

DonggeLiu commented May 4, 2024 •

edited

Loading