[bug-gnulib] Re: iconvme again

[Paul Eggert]

Simon Josefsson <address@hidden> writes:

> It was proposed in
>
> http://sources.redhat.com/bugzilla/show_bug.cgi?id=756
>
> to simplify the overflow check to
>
> if (outbuf_size == 0)
>   {
>     errno = ENOMEM;
>     return NULL;
>   }

No, that check is incorrect.  outbuf_size isn't set to zero when an
overflow occurs.  Unsigned arithmetic wraps around, so outbuf_size
could be any of a wide variety of values in the size_t range.

For example, suppose MB_LEN_MAX is 16, inbytes_remaining is 2**24, and
we are on a 32-bit machine.  Then

  size_t outbuf_size = (inbytes_remaining + 1) * MB_LEN_MAX;

would set inbytes_remaining to 16, not zero.

I have updated the glibc bug report accordingly.