Changes for 2nd expansion

[Paul Smith]

Hi Boris;

This is an FYI and to see if you have any thoughts.

I'm doing a bunch of work to try to reduce memory usage in make (there's
a bug reported by glibc, among others, about these kinds of issues).
The main thrust of this so far is to combine the same string (filenames
essentially) so all users of that filename use a pointer to the same
memory.  This is the strcache module (I can't remember if this was in
3.81 or added later).  An added bonus to this (not yet implemented) is
that we can do away with a lot of the strcmp() operations, and switch to
simple pointer comparisons (I hope to use this to offset the performance
loss from the string lookups etc.)

After my initial implementation, I discovered that I really hadn't saved
any memory.  Looking at it more carefully, I found that the problem was
the way prereqs were handled.  Since I added the unexpanded values to
the strcache in read.c, then I expanded them later in expand_deps() and
added THOSE as well, basically I used twice as much memory as necessary.

So, I have started rewriting the way 2nd expansion is handled.  I've
abstracted some of the operations from expand_deps() so they can be
invoked early or late, as needed.  I moved the expansion of prereqs back
into record_file(), for any prereq that does not need 2nd expansion
(where either .SECONDEXPANSION is not set, OR there are no "$" in the
prereq list for explicit rules).  For any prereq that DOES need 2nd
expansion, I don't add it to the strcache; it uses normal allocated
memory which is then freed when the prereq is parsed in expand_deps().

This means that for any prereq in the list which does not have the
needs_2nd_expansion flag set, it's an already-parsed prereq and we don't
need to parse it again.

I also removed the need for the special handling of "updated" flag, by
putting prereqs into the deps list in the right order from the beginning
rather than in reversed order.  During expand_deps() I insert the
expanded deps into the list at that point, so that order is always
preserved.  It's possible that there is a small performance decrease
(but I don't see it immediately since I keep pointers to the end of the
list where possible) but it makes it simpler for me to understand so
unless I can find a serious problem I'll leave it like this.

Everything is working now, up to the pattern rule expansion which is
what I'm looking at now.  This seems pretty complex, with the conversion
to struct idep etc.  And, it's not necessary to re-expand all the
prerequisites since most of them will already be expanded (of course we
still have to handle patterns if they exist but this can be done simply,
using the old-style %-replacement).

I'm thinking of dropping the idep stuff and instead creating a new set
of struct dep from the expanded patterns, and using that (sort of
similar to what was done in the past but with an added expansion step,
so we can actually get multiple struct dep's from a single struct dep).

I know this is asking for you to remember way back, but if you have any
thoughts/caveats/things to look for/special hidden gotchas that you can
think of, please let me know.

I hope to finish this up this week.

Cheers!

-- 
-------------------------------------------------------------------------------
 Paul D. Smith <address@hidden>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.mad-scientist.us
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist