Re: glibc 2.3.2 pthread_create() mixed with fork() - problem with free()

[Wolfram Gloger]

Hi,

> However, in the web server, once it has been determined that the request is
> a simple, static, report, fork() is used.  This allows the report to be
> created from a snapshot of the internal tables.  By virtue of copy-on-write,
> the cost of the static copy is minimal.  The report runs, returns data to
> the user and the forked thread exits, e.g.:
> 
> main thread
>   -> data1 thread
>   -> data2 thread
>   -> resolver thread
>   -> web server thread
>        -> report process (fork())

This looks OK and is supposed to work with glibc, even though the
relevant standards only allow you to call async-signal safe functions
after fork().

> Unfortunately - and I'll be the first to admit this is a bug - deep in the
> code for the report, a common routine is invoked of the 'if we haven't found
> x out yet, inquire and update the data structure'.  Because this is in the
> fork()ed static copy, the expected behavior is:
> 
>    1. the data structure is updated in the static copy.
>    2. the report - built from the static copy - should reflect the data.
>    3. the main and worker threads (pthread_create()ed), would never see the
> update.

I can't see a bug here.  However, I would expect that there _has_ to
be some sort of synchronization with your other threads here, because
if your data structure is anything but trivial, it could be
inconsistent at fork() time, so the 'static copy' is corrupt.

> Under RedHat Linux 9 (glibc 2.3.2) (and if one updates the RH8 glibc to
> 2.3.2), a problem surfaces.  The free() appears to corrupt the memory
> allocation routines, causing the fork()ed routine to segment fault at
> various and random times, locations, etc.  I'm Ivory soap sure (99.94%) that
> the free() is the problem - if I change the code to skip the free in the
> fork()ed child, the 'random' problems ALL disappear.

OK so you are 100% sure the crash happens in the fork()ed routine,
yes?

> So I'm not surprised to find different behavior in edge/undefined cases.

There isn't supposed to be different behaviour -- if it was correct in
2.2 it should work in 2.3 as well.

> My working hypothesis is, therefore, that something in the new malloc() et
> al code, is sharing the allocated memory across the threads, regardless of
> whether pthread_create() - which should share or fork() - which should not -
> was used to create it.

That is _highly_ unlikely, it would in fact be a bug in fork().

> I've looked at pthread_atfork, but according to it's man page, it's really
> aimed at resolving cross-fork() mutex issues:
...
> Nothing in there indicates anything about memory allocation routines.

Note: pthread_atfork() is aimed at resolving cross-fork() _data
consistency_ issues.  Most of the time, this means taking mutexes in
the "before_fork" handler and releasing them in the "after_fork"
handler, but it could be more complex as well.

So I would expect that your code would definitely _need_ to set up an
atfork handler to protect your "data structure" as mentioned above.
Otherwise, it could be in just about _any_ state at the time the
fork() happens.  For example, the string pointer you mentioned could
be "in the middle" of being updated to a new value by some other
thread.  Then, in the forked() thread, the corrupt pointer will cause
a crash when free()d.

This is just a hypothesis of course.  There could be a bug in glibc,
too.  Please try to send sample code in that case that shows the
problem.

There is a test for malloc/atfork issues at

http://www.malloc.de/tests/fork-malloc.c

I haven't run this in a long time, so you could try it first on your
glibc-2.3 system to make sure that there isn't a regression with
respect to earlier glibc versions (which would explain the problem in
your more complex code).

Regards,
Wolfram.