[Tinycc-devel] Testing with AFL and Valgrind

[Edmund Grimley Evans]

Somebody recently reported on a short file, discovered by afl-fuzz,
that caused an old version of tcc to crash: echo 'm[in(){}' | ./tcc -
That bug seems to have been fixed in the meantime, but it inspired
me to try AFL on the mob repo version of tcc.

AFL quickly discovered an infinite loop when the input finishes with a
'*' inside a C-style comment, for example: printf '/**' | ./tcc -
I didn't find anything else, and I wasn't really expecting to: AFL
is a clever tool, but a C compiler is not the sort of program that
it works best on.

In the course of playing with AFL I built tcc with clang rather than
gcc, and that provided a couple of new warnings, which led me to make
commits eafd7a7 and c899659.

If anyone else wants to try with AFL, here's the Quick Start Guide:
http://lcamtuf.coredump.cx/afl/QuickStartGuide.txt

Apparently the initial test cases can make a big difference, and you
can give it a "dictionary". Perhaps with a more intelligent approach
AFL would find another bug in TCC.

Afterwards I tried with another, even more famous tool: Valgrind.
Here's the quick start guide:
http://valgrind.org/docs/manual/quick-start.html

Here's what I did:

apt-get install valgrind

cd .../tinycc
./configure
make
mv tcc tcc.real
cat > tcc <<'EOF'
#!/bin/sh
valgrind -q --leak-check=no $0.real "$@" # ignore leaks for now
EOF
chmod +x tcc
# Delete test3 for now:
perl -i -ne 'print unless /^ test3/;' tests/Makefile
make test &> log

This revealed a fairly interesting bug in tccelf.c, which I have
fixed, I hope, with commit f0b7566, so look at the log if you're
interested. It also revealed another bug, which I haven't fixed:
"Conditional jump or move depends on uninitialised value(s)" at
macro_subst (tccpp.c:3232).

The code there is inspecting the last "char" of the token string, to
see whether it is a space, but the last "char" might be (an
uninitialised) part of a CString, or perhaps part of an "nwchar_t".
You could, of course, make Valgrind stop complaining by calling memset
when allocating the buffer, but that would not fix the real bug, which
is probably quite fundamental. As far as I can tell, line 928 of
tccpp.c, for example, is doing an illegal cast, but that code has been
in TCC since 2004.

Valgrind also detects memory leaks, but I didn't look any further to
discover whether that really is a bug.

To summarise, the tools did not find a lot of bugs, but they did both
find a bug, and with very little human effort. I think one should
regard the bugs they found as a sample: there are probably a lot of
other bugs, and perhaps similar bugs, that the tools did not find. One
should try to learn a more general lesson from each bug: what is wrong
with the coding style that allowed those bugs to arise?

Edmund