[Qemu-devel] [RFC 00/19] Add virtual device fuzzing support

[Oleinik, Alexander]

As part of Google Summer of Code 2019, I'm working on integrating
fuzzing of virtual devices into QEMU [1]. This is a highly WIP patchset
adding this functionality.

Fuzzers provide random data to a program and monitor its execution for
errors. Coverage-guided fuzzers also observe the parts of the program
that are exercised by each input, and use this information to
mutate/guide the inputs to reach additional parts of the program. They
are quite effective for finding bugs in a wide range of software. 

Summary:
 - The virtual-device fuzzers use libfuzzer [2] for coverage-guided
   in-process fuzzing.
 - To fuzz a device, create a new fuzz "target" - i.e. a function that
   exercises QEMU based on inputs provided by the fuzzer.
 - Fuzz targets rely on qtest and libqos to turn inputs into actions.
 - Since libfuzzer does in-process fuzzing, the QEMU state needs to be
   reset after each fuzz run. These patches provide three methods for
   resetting state.
 - There are currently few targets, but they have already helped
   discover bugs in the console, and virtio-net, and have reproduced
   previously-reported vulnerabilities.

Here are some main implementation details:
 - The fuzzing occurs within a single process. QTest and QOS are
   modified so the QTest client and server coexist within the same
   process. They communicate with each other through direct function
   calls. Similar to qtest, the fuzzer uses a lightweight accelerator to
   skip CPU emulation. The fuzzing target is responsible for manually
   executing the main loop.
 - Since the same process is reused for many fuzzing runs, QEMU state
   needs to be reset at the end of each run. There are currently three
   implemented options for resetting state: 
   1. Reboot the guest between runs.
      Pros: Straightforward and fast for simple fuzz targets. 
      Cons: Depending on the device, does not reset all device state. If
      the device requires some initialization prior to being ready for
      fuzzing (common for QOS-based targets), this initialization needs
      to be done after each reboot.
      Example target: --virtio-net-ctrl-fuzz
   2. vmsave the state to RAM, once, and restore it after each run.
      Alternatively, only save the device state
      (savevm.c:qemu_save_device_state)
      Pros: Do not need to initialize devices prior to each run.
      VMStateDescriptions often specify more state than the device
      resetting functions called during reboots.
      Cons: Restoring state is often slower than rebooting. There is
      currently no way to save the QOS object state, so the objects
      usually needs to be re-allocated, defeating the purpose of
      one-time device initialization.
      Example target: --qtest-fuzz
   3. Run each test case in a separate forked process and copy the 
      coverage information back to the parent. This is fairly similar to
      AFL's "deferred" fork-server mode [3]
      Pros: Relatively fast. Devices only need to be initialized once.
      No need to do slow reboots or vmloads.
      Cons: Not officially supported by libfuzzer and the implementation
      is very flimsy. Does not work well for devices that rely on
      dedicated threads.
      Example target: --qtest-fork-fuzz
 - Fuzz targets are registered using QEMU's module system, similar to
   QOS test cases. Base qtest targets are registed with fuzz_add_target
   and QOS-based targets with fuzz_add_qos_target.
 - There are two entry points for the fuzzer:
    LLVMFuzzerInitialize: Run once, prior to fuzzing. Here, we set up
   qtest/qos, register the fuzz targets and partially execute vl.c:main.
   This is also where we would take a snapshot, if using the vmsave
   approach to resetting.
    LLVMFuzzerTestOneInput: Run for each fuzzing input. This function is
   responsible for taking care of device initialization, calling the
   actual fuzz target, and resetting state at the end of each run.
   Both of these functions are defined in tests/fuzz/fuzz.c
 - There are many libfuzzer flags which should be used to configure the
   coverage metrics and storage of interesting fuzz inputs. [2] These
   flags can also be helpful in evaluating fuzzing performance through
   metrics such as inputs/seconds and line-coverage.

Here are some key issues with the current state of the code:
 - The patches change vl.c, main-loop.c, qtest.c, tests/libqtest.c,
   savevm.c, memory.c. I wrapped the changes with #ifdef CONFIG_FUZZ,
   but many of these changes can and should be avoided.
 - tests/fuzz/qos_helpers.c is largely a copy of tests/qos-test.c.
 - The fuzzer is not properly integrated into the build system.
   Currently I simply added all of the necessary objects to
   target/i386/Makefile.objs, but there should be a simple way to build
   for other arches. The binary needs to be linked against libqemuutil,
   libqtest, qos and the qos objects, and the requirements for softmmu
   targets.
 - Some of the fuzz targets leak memory during state-resetting that need
   to be tracked down and fixed. 
 - As mentioned already, running each test in a separate process does
   not seem to be supported by libfuzzer, and the implementation
   reflects this (tests/fuzz/fuzzer_hooks.c)
 - The existing fuzz targets should be cleaned up as they have issues
   with memory alignment and contain redundant checks. The should also
   use qtest's clock_step. The fork fuzz targets are dependant on
   a hard-coded section size.

Building and running:
Libfuzzer requires clang.
  $ CC=clang-7 CXX=clang++-7 ./configure --enable-fuzzing
  $ make i386-softmmu/all
  $ i386-softmmu/qemu-system-i386 --qtest-dma-fuzz -detect_leaks=0

Here "qtest-dma-fuzz" is the fuzz target name. Running qemu-system-i386
without any arguments should print all of the available fuzz targets.
The -help=1 command prints out the available libfuzzer options.

There are more details, including instructions for adding new fuzz
targets in docs/devel/fuzzing.txt

In the coming weeks I would like to fix the issues listed above, more
fuzzing targets, and ideally work on getting QEMU into oss-fuzz[4],
where it can be fuzzed continuously.

I appreciate any feedback. Thanks
-Alex

[1] https://wiki.qemu.org/Internships/ProjectIdeas/QtestOssFuzz
[2] Trophy Case section: http://lcamtuf.coredump.cx/afl/
[3] https://llvm.org/docs/LibFuzzer.html
[4] https://github.com/mirrorer/afl/blob/master/llvm_mode/README.llvm#L82
[5] https://github.com/google/oss-fuzz


Alexander Oleinik (19):
  fuzz: add configure option and linker objects
  fuzz: add FUZZ_TARGET type to qemu module system
  fuzz: add fuzz accelerator
  fuzz: Add qos support to fuzz targets
  fuzz: expose qemu_savevm_state & skip state header
  fuzz: Add ramfile for fast vmstate/vmload
  fuzz: Modify libqtest to directly invoke qtest.c
  fuzz: add shims to intercept libfuzzer init
  fuzz: use mtree_info to find mapped addresses
  fuzz: expose real_main (aka regular vl.c:main)
  fuzz: add direct send/receive in qtest client
  fuzz: hard-code all of the needed files for build
  fuzz: add ctrl vq support to virtio-net in libqos
  fuzz: hard-code a main-loop timeout
  fuzz: add fuzz accelerator type
  fuzz: add general fuzzer entrypoints
  fuzz: add general qtest fuzz target
  fuzz: Add virtio-net tx and ctrl fuzz targets
  fuzz: Add documentation about the fuzzer to docs/

 accel/fuzz.c                 |  47 ++++++
 configure                    |  11 ++
 docs/devel/fuzzing.txt       | 145 +++++++++++++++++
 include/qemu/module.h        |   7 +-
 include/sysemu/fuzz.h        |  15 ++
 include/sysemu/qtest.h       |   7 +-
 include/sysemu/sysemu.h      |   4 +
 memory.c                     |  34 ++++
 migration/savevm.c           |   8 +-
 migration/savevm.h           |   3 +
 qtest.c                      |  19 ++-
 target/i386/Makefile.objs    |  19 +++
 tests/fuzz/fuzz.c            | 262 +++++++++++++++++++++++++++++++
 tests/fuzz/fuzz.h            |  96 ++++++++++++
 tests/fuzz/fuzzer_hooks.c    | 106 +++++++++++++
 tests/fuzz/fuzzer_hooks.h    |   9 ++
 tests/fuzz/qos_fuzz.c        |  63 ++++++++
 tests/fuzz/qos_fuzz.h        |  29 ++++
 tests/fuzz/qos_helpers.c     | 295 +++++++++++++++++++++++++++++++++++
 tests/fuzz/qos_helpers.h     |  17 ++
 tests/fuzz/qtest_fuzz.c      | 261 +++++++++++++++++++++++++++++++
 tests/fuzz/qtest_fuzz.h      |  38 +++++
 tests/fuzz/ramfile.c         | 127 +++++++++++++++
 tests/fuzz/ramfile.h         |  20 +++
 tests/fuzz/virtio-net-fuzz.c | 226 +++++++++++++++++++++++++++
 tests/libqos/virtio-net.c    |   2 +-
 tests/libqtest.c             |  53 ++++++-
 tests/libqtest.h             |   6 +
 util/main-loop.c             |   3 +
 vl.c                         |  21 ++-
 30 files changed, 1945 insertions(+), 8 deletions(-)
 create mode 100644 accel/fuzz.c
 create mode 100644 docs/devel/fuzzing.txt
 create mode 100644 include/sysemu/fuzz.h
 create mode 100644 tests/fuzz/fuzz.c
 create mode 100644 tests/fuzz/fuzz.h
 create mode 100644 tests/fuzz/fuzzer_hooks.c
 create mode 100644 tests/fuzz/fuzzer_hooks.h
 create mode 100644 tests/fuzz/qos_fuzz.c
 create mode 100644 tests/fuzz/qos_fuzz.h
 create mode 100644 tests/fuzz/qos_helpers.c
 create mode 100644 tests/fuzz/qos_helpers.h
 create mode 100644 tests/fuzz/qtest_fuzz.c
 create mode 100644 tests/fuzz/qtest_fuzz.h
 create mode 100644 tests/fuzz/ramfile.c
 create mode 100644 tests/fuzz/ramfile.h
 create mode 100644 tests/fuzz/virtio-net-fuzz.c

-- 
2.20.1