Re: [Security-discuss] handling integer overflows

[Paul Eggert]

Browsing an old mailing list thread I saw this:

> From: Nikos Mavrogiannopoulos
> Date: Sun, 01 Apr 2012 07:54:36 +0200

> I was wondering on how other gnu programs handle integer overflows.

I've added some macros to gnulib to handle the common cases,
and this is used in applications like coreutils and tar.
For example, the macro INT_MULTIPLY_OVERFLOW (a, b)
returns true if a*b would overflow.  This is defined by the
intprops module in gnulib, which consists of a single standalone file
intprops.h which you should be able to use in your application:

http://git.sv.gnu.org/gitweb/?p=gnulib.git;a=blob_plain;f=lib/intprops.h

Unfortunately the overflow checks currently in libtasn1/lib/decoding.c
are not correct.  For example:

    last = ans;
    ans = (ans*256) + der[punt++];

    if (ans < last)
       /* we wrapped around, no bignum support... */
       return -2;

This isn't correct, since it's possible that an overflow
occurred but the above code will not catch it.  On a typical
host with 32-bit words, for example, if last == 0x1020000
and ans == 0x2000000, then overflow has occurred but
it's not the case that ans < last.  You need something
like this instead:

    if (INT_MULTIPLY_OVERFLOW (ans, 256))
       return -2;
    ans = (ans*256) + der[punt++];

I would not be surprised if there were other overflow errors
in that code, too, but I didn't check it carefully.