Port random fuzzing parameters to AFLplusplus

[jiradeto]

This PR ports the idea of randomizing fuzzing parameters (i.e. random weight selection, uniformly random selection, etc.) implemented in Practical-Formal-Methods/AFL-public#6 to afl++.

[jiradeto]

The experiment result is available:

• bloaty
• freetype
• sqlite
• libxml

Configurations:

• app_frozen: AFLplusplus with stable branch (353d402aaf9296c7dbd47e66fbbc6e59179c4e44)
• afl_wrs_rf_rp:
• afl_wrs_rp: AFL_DISABLE_RF=1
• app_rf_u_wrs: AFL_ENABLE_UF=1, AFL_DISABLE_RP=1

[wuestholz]

@jiradeto Thanks! Looks like afl_wrs_rp is the most competitive, but it's only better than the baseline for bloaty.... I should probably take a closer look at the code.

As discussed, something might be off with the RF option. Let's try to tweak the parameters for AFL first.

[jiradeto]

@wuestholz the experiment result of the recent fix is available. I compare the buggy version app_wrs_rf_old against the fixed version app_wrs_rf_new to highlight the difference. The statistics of the experiment is also recored in this sheet.

• sqlite
• bloaty
• libxml
• libjpeg
• freetype

The fewer executions we previously had is clearly due to the uninitialized fuzzing parameters as we expected :) Thank you for noticing this strange behaviour in the first place.

[wuestholz]

@jiradeto Great! Thanks for the update! That looks much better. We actually seem to outperform vanilla AFL++ for some benchmarks.

On the other hand, for some benchmarks we're still something like 2x slower. I wonder what that might be due to. Any ideas? Did you by any chance measure the overhead for specific code we added?

[jiradeto]

On the other hand, for some benchmarks we're still something like 2x slower. I wonder what that might be due to. Any ideas? Did you by any chance measure the overhead for specific code we added?

This experiment setting does not include the afl++ variant that measures the overhead. I am starting another experiment to collect this information and will keep you updated.

[wuestholz]

Great! Thanks a lot! 👍

[jiradeto]

@wuestholz I pushed the latest implementation that changes how we compute the favored inputs as we discussed. This PR is ready for further review. Thank you in advance.

[wuestholz]

@jiradeto Thanks! I left some comments. Have you already tried this for sqlite?

[jiradeto]

@jiradeto Thanks! I left some comments. Have you already tried this for sqlite?

@wuestholz Thank you for your feedback. I haven't started any experiment yet as I was unsure about my implementation. Now I think it's ready for the fuzzbench experiment.

[wuestholz]

@jiradeto Great! Thanks! Yeah, looks ready to run some experiments.

[jiradeto]

As discussed I would merge this PR and request the fuzzbench experiment with the following fuzzers that are inspired by our previous experiment:

1. aflplusplus (vanilla AFL++)
2. app_no_favs: DISABLE_WRS && DISABLE_RF && !ENABLE_UF && DISABLE_FAVS
3. app_wrs_rf: !DISABLE_WRS && !DISABLE_RF && !ENABLE_UF && !DISABLE_FAVS
4. app_rf: DISABLE_WRS && !DISABLE_RF && !ENABLE_UF && !DISABLE_FAVS
5. app_rf_u: DISABLE_WRS && !DISABLE_RF && ENABLE_UF && !DISABLE_FAVS
6. app_wrs: !DISABLE_WRS && DISABLE_RF && !ENABLE_UF && !DISABLE_FAVS

What do you think? @wuestholz

[wuestholz]

@jiradeto Thanks! Yeah, please merge this.

I think we should also consider the AFL_DISABLE_RP option. Or did we somehow determine that it's not useful?

What do you think about the following?

1. aflplusplus (vanilla AFL++)
2. app_no_favs: DISABLE_WRS && DISABLE_RF && !ENABLE_UF && DISABLE_RP && DISABLE_FAVS
3. app_wrs_rf_rp: !DISABLE_WRS && !DISABLE_RF && !ENABLE_UF && !DISABLE_RP && !DISABLE_FAVS
4. app_wrs_rf: !DISABLE_WRS && !DISABLE_RF && !ENABLE_UF && DISABLE_RP && !DISABLE_FAVS
5. app_wrs_rp: !DISABLE_WRS && DISABLE_RF && !ENABLE_UF && !DISABLE_RP && !DISABLE_FAVS
6. app_wrs: !DISABLE_WRS && DISABLE_RF && !ENABLE_UF && DISABLE_RP && !DISABLE_FAVS

[jiradeto]

@wuestholz, seem like I missed those essential configurations. Thank you for your suggestion.

BTW, I think I don't have permission on this repository. Could you please merge this PR?

[wuestholz]

@jiradeto Now that I think about it, we don't need to merge after all. :) I thought there was a separate PR for the optimization, but it's already in this PR. We can just keep this one open.