fuzzing poke with afl & afl++

[Dan Čermák]

Hi list,

I've had some time during the last days and used it to run some fuzzing
tests on poke (essentially I've let poke get bombarded by tons of
mutated input files and watched out for "interesting" things to happen,
e.g. crashes).

If you are interested in the technical details: I've written them down
into the `fuzz` subdirectory in the defolos/fuzzing branch.

Long story short: I've let afl++ fuzz poke's script mode overnight on my
laptop (12 fuzzing processes were running, one in deterministic mode and
11 in random mode) for about 5 hours until poke filled up my /tmp/ and
the fuzzer died (because it was running on the same partition to not
murder my SSD). It found 600-something input files that made poke crash,
but of these only 164 are distinct crashes (distinct according to afl,
at least the failed assertions could be maybe only a handful of
bugs). I've put them into the following categories according to the type
of crash that occurred:

assertion failed: 122
double free:      11
signal FPE:       2
signal SEGV:      19
use after free:   8
misc:             2 (these just result in an "Aborted" message)

The resulting files are attached in the archive crashes.tar.gz.


afl also produced quite a number of hangs (attached in the
hangs.tar.gz). Some of them are relatively boring endless loops that the
fuzzer produced, e.g.:
--8<---------------cut here---------------start------------->8---
defvar i = 0;

while (i < 10)
  {
if (i == 2)
      break;
    print "foo\n";
    i== i + 1;
  }
--8<---------------cut here---------------end--------------->8---

but others are more interesting, like this one:
--8<---------------cut here---------------start------------->8---
defvar x= 4#B;
defvar a = [0,0] as int[x + 4#0];
--8<---------------cut here---------------end--------------->8---


All in all I must say that I'm pretty impressed by poke's code base:
most C code bases that are "battle tested" often die with a ton of
memory errors once someone starts fuzzing them. Poke in contrast mostly
aborts due to failed assertions and I haven't found a single buffer
overflow. I have to note though, that I've fuzzed it for a short time
and probably still only covered a small portion of poke's code base.


I have one suggestion though: could we add a flag to poke (either
compile- or run-time), so that it will not create diagnostic files in
/tmp/? This is not really useful when running poke a few million times
with mostly invalid input.


Cheers,

Dan

crashes.tar.gz
Description: GNU Zip compressed data

hangs.tar.gz
Description: GNU Zip compressed data

signature.asc
Description: PGP signature