Re: Crippling change in ld memory/cpu consumption in v2.12/v2.13

[Marc Paterno]

Hi all,

Previous measurements have demonstrated a problem we have encountered
with excessive memory use by ld when linking programs from large numbers
(thousands) of object modules. The programs in which we are interested
require several tens of thousands of object modules. Along with the
excessive memory usage, we find linking times that are very slow. In
these comparisons, "excessive memory use" and "very slow" is in
comparison of ld 2.12 (and ld 2.13) to ld 2.11.

We reported these measurements to the binutils mailing list, and
received a prompt response from Alan Modra. He provided a patch that
reduced the default hash table size in one of the source files
(opencls.c) used in ld.

We patched the source code for ld 2.13 with the patch he supplied, and
tried our measurements again. The result was a small improvement. Where
ld 2.13 took about 25 kbytes of memory per object module linked, the
patched version (ld 2.13p, hereafter) took about 23 kbytes of memory per
object module. In comparison with ld 2.11, which took only about 4
kbytes of memory per object module, the memory usage was still
excessive.


This prompted us to try a more radical patch. We modified hash.c so that
the default size of all hash tables was reduced to the same size Alan
suggested in his patch for opencls.c. We call this version ld 2.13rh
(reduced hash). It uses slightly less than 8 kbytes of memory per object
module linked.

Plots showing the results can be found at:
http://www-cdserver.fnal.gov/cd_public/cpd/aps/mfp/web/CPlusPlusWG/ld-issue/
summary2.htm

We're now looking at the speed of linking our most important
executables, with these modified versions of ld, and will report the
results when they are available.

If any of the experts has reason to believe reducing the default hash
table size globally will have deleterious effect, please let us know.

best regards,
Marc