Re: libio/libioP.h:ALLOC_BUF() unconditionally uses mmap()

[Wayne Whitney]

Dear Wolfram, John and others,

Thank you for your responses.  I think I'm having a little trouble
expressing myself, so let me try afresh:

The problem I've described with current MAGMA, current glibc, and current
Linux kernel could absolutely be resolved by modifying MAGMA or modifying
the Linux kernel.  As for modifying MAGMA to use malloc() instead of
sbrk(), I can't speak authoritatively, as I am not the author and have not
read the source code.  I can say that the author indicated that malloc()
is not a drop-in replacement for the allocator he implemented eons ago on
top of sbrk().

On the Linux kernel side, there is definitely alot to be said for
modifying the kernel to allow TASK_UNMAPPED_BASE to be set on a
per-process basis.  I've discussed that a bit on the LKML and am planning
to look at doing it myself.

But since this is the bug-glibc list, I was hoping to consider here how
glibc fits in this picture, and what the relative difficulty and merit of
modifying it is.  I'm not advocating any particular change here, just
brainstorming.

So:

1) Wolfram, you said that the particular use of mmap() in
glibc-2.2.4/libio/libioP.h:ALLOC_BUF() had the benefits of a) allowing the
main program to modify malloc(), b) thread-safety, and c) giving a
page-aligned buffer without memory fragmentation.

For (a), ALLOC_BUF() has to use some allocation mechanism (although when
I've overridden __mmap() to always return MAP_FAILED, stdio still seems to
work), so some allocation will occur in a way the main program can not
tune.  You suggest that it is better for this allocation to be an mmap()
than a malloc().  Why is this?  I might like to override or tune mmap()
just as much as I might like to override or tune malloc(), no?

As for (b), I'm clueless: I would naively think that each thread would
have its own stdio buffer, and that it would construct and destruct the
buffer itself, so there would be no thread issue.  What am I missing?

And I assume (c) is not a real issue, just icing on the cake, as if the
stdio buffer should be page-aligned, it would not be so bad to waste
(upto) an extra page per process.  Or would it?

As I mentioned, the argument for using malloc() in ALLOC_BUFF() is that,
beyond solving the immediate issue I have, it would seem more consistent:  
one would expect the malloc tuning parameters to apply to all of an
executable's allocations, both glibc and the main program.

2) If the use of mmap() in ALLOC_BUF() is necessary, what about the option
of having some more control over it?  For example, it would be enough for
my purposes to be able to tell glibc to attempt the mmap() at a particular
address, with fallback to an anonymous mapping if that fails.  Then a
program (like MAGMA) could ask for it higher up, or perhaps even move it
to just before the code page.

3) Lastly, failing even that, it would be enough to have a hook of some
sort to allow something to occur before the mmap().  For example, to allow
the program to intercept mmap(), putting in a higher requested address to
each call.  Or to execute a really big brk() before the mmap(), that would
allow the reservation of address space for the heap.  Or, with John
Reiser's setrlimit() proposal, to call setrlimit().

Of course, MAGMA could maintain a private patch against glibc to implement
one of these ideas, and link statically against it, since static linking
is required for large heap space at present anyway.  But I know there are
other programs out there that want a large heap space, I'm in touch with
another other Linux user who has this issue.

Thank you for your time, and any comments you have about the relative and
absolute virtues of the above possibilities would be very helpful.

Best wishes,
Wayne