RE: Crippling change in ld memory/cpu consumption in v2.12/v2.13

[Marc Paterno]

Hello,

Several weeks ago, Robert Kennedy of Fermilab wrote to this forum,
asking for advice concerning a problem we at Fermilab have been
encountering. The problem is a large increase in the memory usage of
ld, between version 2.11 and 2.12. This follow-up message provides
some results of measurements we have done to quantify the problem, in
the hopes it will help one of the experts to solve the problem.

We have generated data on the memory and time usage of ld versions
2.11 and 2.12. We have previously observed that the time taken by ld
v2.12 to link large executables is much greater than the time taken by
ld 2.11, and that the memory requirements of the new ld are also much
greater than that of the old. Our measurements quantify these
observations, and identify the culprit as the number of object modules
given to the linker.

Our observations are:

[1] The memory usage of both ld v2.11 and v2.12 increase linearly with
the number of object modules used in the program. The total number of
symbols in the program is much less important than the number of
object modules.

[2] ld v2.11 uses about 4.2 kbytes memory per object file; ld v2.12
uses about 26 kbytes memory per object file. This large increase is
causing significant problems for us when we are linking large
executables; executables of interest require of order 30,000 object
modules.

[3] The time usage of both ld v2.11 and v2.12 are slightly worse than
quadratic in the number of symbols in the program. This is cause for
alarm with both versions of ld, for large executables.  </p> </li>

[4] The size of the symbols is not an important factor in the amount
of memory used by ld, even when the symbol size is made huge
(thousands of bytes).

We are interested in finding out what change in ld was made in going
from version 2.11 to version 2.12 that resulted in the dramatic
increase in memory usage, and especially in finding out if the
previous memory profile can be recovered.

If it is not possible to quickly modify ld to regain its past levels
of performance, is it safe for use to continue to use the older
version of ld with new versions of GCC? If so, should we be using the
full set of products from binutils v2.11, or should we use only ld
v2.11, and the newest versions of the other tools in binutils?

A plot is available (at URL
http://www-cdserver.fnal.gov/cd_public/cpd/aps/mfp/web/CPlusPlusWG/ld-issue/
summary.htm)
which shows the amount of memory used, as reported by ld, for varying
number of object files. The memory usage is clearly linear in the
number of object files, and much greater in ld 2.12 than in 2.11. The
number of symbols is clearly not as important as the number of object
files.

We would appreciate any assistance in solving this problem.

best regards,

Marc Paterno
Fermi National Accelerator Laboratory
Computing Division - Computational Physics Department