[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v4 00/20] Add virtual device fuzzing support
|
From: |
Darren Kenny |
|
Subject: |
Re: [PATCH v4 00/20] Add virtual device fuzzing support |
|
Date: |
Tue, 5 Nov 2019 16:47:39 +0000 |
|
User-agent: |
NeoMutt/20180716 |
On Tue, Nov 05, 2019 at 11:28:59AM -0500, Alexander Oleinik wrote:
On 11/5/19 8:57 AM, Darren Kenny wrote:
Hi Alexander,
I've been trying out these patches, and I'm seeing a high volume of
crashes - where for v3, there were none in a run of over 3 weeks -
so it was a bit of a surprise :)
The question is what may have changed that is causing that level of
crashes - are you seeing this for the virtio-net-fork-fuzz tests?
Good question - my guess is that it may have to do with the change in
how we run the main loop. I have not looked into it in much detail,
but the crash below is likely triggered only after running the
main_loop several times (the events handled in the first loop,
schedule additional BHs). In v3, I believe the main_loop only ran once
before the forked process exited. There are also changes to the
linker-script used to facilitate communication between the forked
process and the parent, but I think that would only impact the
coverage information passed back to the parent.
OK, sounds like this is a genuine issue then - great :)
But also, I've been trying to debug some of these crashes - and the
expectation is that you pass the crash-XXXX file as an argument to
the qemu-fuzz-* binary - and when I do, I see the crash - but when I
try to debug it, it ends up running through and exiting
My assumption is that because of the fork in the test, the crash is
in one of the children.
Right! Seems you are already using the follow-fork-mode option.
Yup
(ASIDE: I think it might be worth adding a debugging/analysing
section to the documentation you've added to help people debug such
crashes)
Will do. Although it did not make it into v4, I am also working on an
option to dump a trace of the qtest commands leading to a crashing
input, which can then be replayed with a standard qtest program
"replay.c". This seems like a good way to provide a reproducer to the
device developers who may not be familiar with the fuzzer, or have
time to build it.
That would be great!
For me the biggest problem was that I didn't have a version of clang
new enough (OL7/RHEL7) to have Libfuzzer, so if people don't have
it, then building with it's support is a bit more involved since you
also need to build clang, etc.
Setting follow-fork-mode to child does get me there, and each crash
seems, at least in the samples that I've taken, to be in iov_copy:
Yes - this is what I have been using as well.
#0 0x00007ffff4cff377 in raise () from /lib64/libc.so.6
#1 0x00007ffff4d00a68 in abort () from /lib64/libc.so.6
#2 0x00007ffff4cf8196 in __assert_fail_base () from
/lib64/libc.so.6
#3 0x00007ffff4cf8242 in __assert_fail () from /lib64/libc.so.6
#4 0x00005555574d4026 in iov_copy ()
#5 0x000055555640dbd8 in virtio_net_flush_tx ()
#6 0x000055555640c8ef in virtio_net_tx_bh ()
#7 0x00005555574a05bb in aio_bh_call ()
#8 0x00005555574a0a34 in aio_bh_poll ()
#9 0x00005555574b1687 in aio_dispatch ()
#10 0x00005555574a35f9 in aio_ctx_dispatch ()
#11 0x00007ffff5e5d099 in g_main_context_dispatch () from
/lib64/libglib-2.0.so.0
#12 0x00005555574ae9fd in glib_pollfds_poll ()
#13 0x00005555574ad972 in os_host_main_loop_wait ()
#14 0x00005555574ad62c in main_loop_wait ()
#15 0x000055555736c653 in flush_events ()
#16 0x00005555573710a4 in virtio_net_fork_fuzz ()
#17 0x000055555736cb85 in LLVMFuzzerTestOneInput ()
...
Have you seen these kind of crashes, or is this just me?
Not just you :) I posted a fix for this, but it may have not been
complete. I think the fuzzer found it before we added forking, just by
doing reboots in between runs:
https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg04882.html
Ah! So I guess this hasn't been pulled into master yet (or at least
wasn't when I pulled from there yesterday.
Thanks,
Darren.
Just wondering if I should dig into it as a real issue, or some
mis-merge I've done (not all the patches were cleanly applied for
me when I cloned from master).
Thanks,
Darren.
On Wed, Oct 30, 2019 at 02:49:47PM +0000, Oleinik, Alexander wrote:
This series adds a framework for coverage-guided fuzzing of
virtual-devices. Fuzzing targets are based on qtest and can make use of
the libqos abstractions.
V4:
* add/transfer license headers to new files
* restructure the added QTestClientTransportOps struct
* restructure the FuzzTarget struct and fuzzer skeleton
* fork-based fuzzer now directly mmaps shm over the coverage bitmaps
* fixes to i440 and virtio-net fuzz targets
* undo the changes to qtest_memwrite
* possible to build /fuzz and /all in the same build-dir
* misc fixes to address V3 comments
V3:
* rebased onto v4.1.0+
* add the fuzzer as a new build-target type in the build-system
* add indirection to qtest client/server communication functions
* remove ramfile and snapshot-based fuzzing support
* add i440fx fuzz-target as a reference for developers.
* add linker-script to assist with fork-based fuzzer
V2:
* split off changes to qos virtio-net and qtest server to other patches
* move vl:main initialization into new func: qemu_init
* moved useful functions from qos-test.c to a separate object
* use struct of function pointers for add_fuzz_target(), instead of
arguments
* move ramfile to migration/qemu-file
* rewrite fork-based fuzzer pending patch to libfuzzer
* pass check-patch
Alexander Oleinik (20):
softmmu: split off vl.c:main() into main.c
libqos: Rename i2c_send and i2c_recv
fuzz: Add FUZZ_TARGET module type
qtest: add qtest_server_send abstraction
libqtest: Add a layer of abstraciton to send/recv
module: check module wasn't already initialized
qtest: add in-process incoming command handler
tests: provide test variables to other targets
libqos: split qos-test and libqos makefile vars
libqos: move useful qos-test funcs to qos_external
libqtest: make qtest_bufwrite send "atomic"
libqtest: add in-process qtest.c tx/rx handlers
fuzz: add configure flag --enable-fuzzing
fuzz: Add target/fuzz makefile rules
fuzz: add fuzzer skeleton
fuzz: add support for fork-based fuzzing.
fuzz: add support for qos-assisted fuzz targets
fuzz: add i440fx fuzz targets
fuzz: add virtio-net fuzz target
fuzz: add documentation to docs/devel/
Makefile | 16 ++-
Makefile.objs | 4 +
Makefile.target | 18 ++-
configure | 39 ++++++
docs/devel/fuzzing.txt | 119 ++++++++++++++++++
exec.c | 12 +-
include/qemu/module.h | 4 +-
include/sysemu/qtest.h | 4 +
include/sysemu/sysemu.h | 4 +
main.c | 52 ++++++++
qtest.c | 30 ++++-
tests/Makefile.include | 75 +++++------
tests/fuzz/Makefile.include | 11 ++
tests/fuzz/fork_fuzz.c | 51 ++++++++
tests/fuzz/fork_fuzz.h | 23 ++++
tests/fuzz/fork_fuzz.ld | 37 ++++++
tests/fuzz/fuzz.c | 177 ++++++++++++++++++++++++++
tests/fuzz/fuzz.h | 66 ++++++++++
tests/fuzz/i440fx_fuzz.c | 176 ++++++++++++++++++++++++++
tests/fuzz/qos_fuzz.c | 232 +++++++++++++++++++++++++++++++++++
tests/fuzz/qos_fuzz.h | 33 +++++
tests/fuzz/virtio_net_fuzz.c | 123 +++++++++++++++++++
tests/libqos/i2c-imx.c | 8 +-
tests/libqos/i2c-omap.c | 8 +-
tests/libqos/i2c.c | 10 +-
tests/libqos/i2c.h | 4 +-
tests/libqos/qos_external.c | 168 +++++++++++++++++++++++++
tests/libqos/qos_external.h | 28 +++++
tests/libqtest.c | 109 ++++++++++++++--
tests/libqtest.h | 4 +
tests/pca9552-test.c | 10 +-
tests/qos-test.c | 140 +--------------------
util/module.c | 7 ++
vl.c | 36 ++----
34 files changed, 1601 insertions(+), 237 deletions(-)
create mode 100644 docs/devel/fuzzing.txt
create mode 100644 main.c
create mode 100644 tests/fuzz/Makefile.include
create mode 100644 tests/fuzz/fork_fuzz.c
create mode 100644 tests/fuzz/fork_fuzz.h
create mode 100644 tests/fuzz/fork_fuzz.ld
create mode 100644 tests/fuzz/fuzz.c
create mode 100644 tests/fuzz/fuzz.h
create mode 100644 tests/fuzz/i440fx_fuzz.c
create mode 100644 tests/fuzz/qos_fuzz.c
create mode 100644 tests/fuzz/qos_fuzz.h
create mode 100644 tests/fuzz/virtio_net_fuzz.c
create mode 100644 tests/libqos/qos_external.c
create mode 100644 tests/libqos/qos_external.h
--
2.23.0