Make slow on Solaris

[Krzysztof Cieniuch]

Hi,

I  developed make build system that extensively uses eval and call and 
does lot's of string conversions and variable assignments
it is enough to say that no rule is written by hand everything is 
generated by make during parsing stage.
So this make system is very computing intensive but also memory access 
is intensive.
I'm building product on four platforms Linux on intel  xeon chips, 
hp-ux: ia64 (Itanium) architecture  and Solaris 10 running on T series 
processors and UltraIII.
Currently in build system I have around 350 makefiles the build system 
follows pattern borrowed from Painlels Non-Recursive Make article i.e.
in project makefiles you include make header set some variables and then 
include make footer that generates rules depending on set variables
I run into strange problem with make performance. The makefile parsing 
stage on Solaris is very different from other machines
here is info :

Linux (3Ghz chips) makefile parsing stage: 12sec
HP-UX: (1,6Ghz chips) makefile parsing stage: 16sec
Solaris (1,2Ghz chips UltraIII) makefile parsing stage: 1 minute 20seconds
Solaris (1,1Ghz chips Tseries) makefile parsing stage: 3 minutes 18seconds

All source code is located on network drive so the disk io is similar 
for all machines.
when I use -d I can see how parsing process is getting slower at the times
I can  tell that when I see how below debug output is printed on terminal.
[..]
Reading makefile `mk/install.make' (search path) (no ~ expansion)...
Reading makefile `some/module/path/Makefile.mk' (search path) (no ~ 
expansion)...
Reading makefile `mk/init.make' (search path) (no ~ expansion)...
Reading makefile `mk/make.make' (search path) (no ~ expansion)...
[..]
There are times when make "hangs for a 3-4 seconds until it prints out 
next line of "Reading makefile ..."
I have this issue only on Solaris other platforms work with similar 
performance.

running truss shows that majority of that time is spent in user space
syscall               seconds   calls  errors
_exit                    .000       1
read                     .368    5161
write                    .818    7671
open                     .000       7       1
close                    .086    1919
chdir                    .000       2
brk                      .092    2770
stat                     .002      34      27
getpid                   .000       1
ioctl                    .052    1703    1703
execve                   .000       1
fcntl                    .006     218
openat                   .081     218
sigaction                .000       9
sigfillset               .000       1
getcontext               .000       1
setustack                .000       1
mmap                     .002      26
munmap                   .000       7
getrlimit                .000       1
memcntl                  .000       5
sysconfig                .000       1
llseek                   .054    1703
schedctl                 .000       1
resolvepath              .000       8
getdents64               .017     426
stat64                  1.877    6881    4188
fstat64                  .129    3626
setrlimit64              .000       1
getrlimit64              .000       1
open64                  1.945    8460    6757
getcwd                   .000       2
                    --------  ------   ----
sys totals:             5.538   40867  12676
usr time:             182.147
elapsed:              271.670

Do you have any clues why Solaris is so different ??? I looked briefly 
through make source code and couldn't find
anything special for Sun platform the code is pretty generic but just 
wanted to confirm that.

I'm using precompiled make form sunfreeware site. But maybe there are some
magic compiler flags :-) when compiling make that would solve this issue 
in my case if so I could recompile make by myself?

As I said before parsing stage is memory access intensive so it might be 
that Sun hardware is the bottleneck
and there is nothing I can do about this :-(

On all platforms I'm using make 3.81

Thanks
Chris

PS. here is pstack output taken when make was "hanging" in parsing of 
some module makefile to give you the
idea how complicated the system is :-) (yes its one stack trace :-)
pstack `pgrep -u chris make`
24143:  /usr/local/bin/make -r
00021488 handle_function (ffbf1a3c, ffbf1a38, 0, 9c2b8, ff000000, 
ffbf1a38) + d4
00018db0 variable_expand_string (0, 6f6c0, ffffffff, ff256eb0, 71, 71) 
+ 234
00019574 variable_expand (6f6c0, 696d8, e70c4, 0, 601a0, 18928c) + 14
000196c8 variable_expand_for_file (6f6c0, 0, ff343800, 0, ff352a00, 
696d8) + 20
00019a80 allocated_variable_expand_for_file (6f6c0, 0, 11b4ac, 0, 
696d8, 85fffc2c) + 40
00018af8 recursively_expand_for_file (696d8, 0, ffffffff, 9c238, 
4aa0e0, 696d8) + 1c4
000194d4 reference_variable (9bef8, ffbf1e1a, 3, ffbf1e1a, 71, 
ffbf1e1d) + ec
000192c8 variable_expand_string (9bef8, ffbf1e18, 6, 6, 696d8, 5fd30) + 74c
00021cb4 func_call (9bef8, ffbf1f50, 4b300, 0, 29, 5fa00) + 48c
00021398 expand_builtin_function (9bef8, 3, ffbf1f50, 5fa00, ffec5e20, 
1d6108) + d0
00021798 handle_function (ffbf205c, ffbf2058, 0, 9bef8, ff000000, 
ffbf2058) + 3e4
00018db0 variable_expand_string (0, 1d60d8, ffffffff, 0, ff352a00, 
ff2563a0) + 234
00019574 variable_expand (1d60d8, 73736900, 19feb4, 737369, ff00, 
ff0000) + 14
000196c8 variable_expand_for_file (1d60d8, 0, 1d8b60, 9bef0, 1b90, 0) + 20
00019a80 allocated_variable_expand_for_file (1d60d8, 0, 38, ff256eb0, 
0, 1d60d8) + 40
00019600 expand_argument (1d60d8, 1d6108, 1d6108, 80808080, 1d6107, 
1d60d8) + 74
00020238 func_if  (1d6009, ffbf23c8, 4b320, 448f0, ff33e3c0, 5fa30) + 84
00021398 expand_builtin_function (1d6009, 2, ffbf23c8, 5fa30, 29, 
1d6129) + d0
00021798 handle_function (ffbf24d4, ffbf24d0, 1, 1d6009, ff00, 
ffbf24d0) + 3e4
00018db0 variable_expand_string (0, 1d5dd0, ffffffff, 0, ff352a00, 
ff2563a0) + 234
00019574 variable_expand (1d5dd0, 73736900, 19feb4, 737369, ff00, 
ff0000) + 14
000196c8 variable_expand_for_file (1d5dd0, 0, e71a0, ff347344, 0, 
19feb0) + 20
00019a80 allocated_variable_expand_for_file (1d5dd0, 0, ff343800, 0, 
ff352a00, 1d5f03) + 40
0001f20c func_foreach (1d5ce9, ffbf27e0, 4b2f8, 448f0, ff33e3c0, 5f9f4) 
+ 15c
00021398 expand_builtin_function (1d5ce9, 3, ffbf27e0, 5f9f4, 29, 
1d5e28) + d0
00021798 handle_function (ffbf28f4, ffbf28f0, 1, 1d5ce9, ff000000, 
ffbf28f0) + 3e4
00018db0 variable_expand_string (0, ffbf2ae8, ffffffff, 63320000, 70, 
70) + 234
00019574 variable_expand (ffbf2ae8, 5f9ac, 488, ff256eb0, 5fd34, 64374) 
+ 14
000196c8 variable_expand_for_file (ffbf2ae8, 0, 29, 0, ff352a00, 5f9ac) 
+ 20
00019a80 allocated_variable_expand_for_file (ffbf2ae8, 0, 0, 0, 
ffb42e2b, 306) + 40
0001968c expand_argument (afcbd, affc3, ffffffff, 6, 125f63, 1) + 100
0002172c handle_function (ffbf2fc4, ffbf2fc0, 0, 1d5c18, 1, ffbf2fc0) + 378
00018db0 variable_expand_string (0, afca8, ffffffff, 67657400, 74, 74) 
+ 234
00019574 variable_expand (afca8, a8fd0, e70c4, 1d5c10, 601a0, 187920) + 14
000196c8 variable_expand_for_file (afca8, 0, ff343800, 0, ff352a00, 
a8fd0) + 20
00019a80 allocated_variable_expand_for_file (afca8, 0, 1ca1b4, 0, 
a8fd0, 85fffc2c) + 40
00018af8 recursively_expand_for_file (a8fd0, 0, ffffffff, 1d5be0, 
5e2431, a8fd0) + 1c4
000194d4 reference_variable (1d57d1, ffbf33a2, f, ffbf33a2, 74, 
ffbf33b1) + ec
000192c8 variable_expand_string (1d57d1, ffbf33a0, 12, 6, a8fd0, 5fd30) 
+ 74c
00021cb4 func_call (1d57d1, ffbf34e8, 4b300, 0, 29, 5fa00) + 48c
00021398 expand_builtin_function (1d57d1, 3, ffbf34e8, 5fa00, 20, 
1d56c6) + d0
00021798 handle_function (ffbf35fc, ffbf35f8, 1, 1d57d1, ff00, 
ffbf35f8) + 3e4
00018db0 variable_expand_string (0, 1d5674, ffffffff, 9bef0, 0, 0) + 234
00019574 variable_expand (1d5674, 1d5f60, 1c5808, 9bef0, 1b90, 0) + 14
000196c8 variable_expand_for_file (1d5674, 0, 210, 196c8, 10c3, 1d6178) 
+ 20
00019a80 allocated_variable_expand_for_file (1d5674, 0, e71a0, 
ff347344, 0, 1d5674) + 40
00019600 expand_argument (1d5674, 0, ff343800, 0, ff352a00, 80808080) + 74
000202c0 func_if  (1d55b3, ffbf3968, 4b320, 448f0, ff33e3c0, 5fa30) + 10c
00021398 expand_builtin_function (1d55b3, 2, ffbf3968, 5fa30, 20, 
1d57c8) + d0
00021798 handle_function (ffbf3aac, ffbf3aa8, 1, 1d55b3, ff00, 
ffbf3aa8) + 3e4
00018db0 variable_expand_string (0, 1d53b2, ffffffff, 0, ff352a00, 
ff3472bc) + 234
00019574 variable_expand (1d53b2, 18, ff33e3c0, 8, a5958, ff0000) + 14
000196c8 variable_expand_for_file (1d53b2, 0, e7974, 196c8, 1a, 1a) + 20
00019a80 allocated_variable_expand_for_file (1d53b2, 0, e71a0, 
ff347344, 0, 1d53b2) + 40
00019600 expand_argument (1d53b2, 0, ff343800, 0, ff352a00, 80808080) + 74
000202c0 func_if  (1d4ef1, ffbf3e18, 4b320, 448f0, ff33e3c0, 5fa30) + 10c
00021398 expand_builtin_function (1d4ef1, 3, ffbf3e18, 5fa30, 20, 
1d556e) + d0
00021798 handle_function (ffbf3f6c, ffbf3f68, 1, 1d4ef1, ff000000, 
ffbf3f68) + 3e4
00018db0 variable_expand_string (0, 1d4fdb, ffffffff, 0, ff352a00, 
ff3472bc) + 234
00019574 variable_expand (1d4fdb, 6500756e, 1c4bfc, 6500, ff000000, 
ff0000) + 14
000196c8 variable_expand_for_file (1d4fdb, 0, e71a0, ff347344, 0, 
1c4be0) + 20
00019a80 allocated_variable_expand_for_file (1d4fdb, 0, ff343800, 0, 
ff352a00, 1d52f9) + 40
0001f20c func_foreach (1d4ef1, ffbf4278, 4b2f8, 448f0, ff33e3c0, 5f9f4) 
+ 15c
00021398 expand_builtin_function (1d4ef1, 3, ffbf4278, 5f9f4, 20, 
1d51b8) + d0
00021798 handle_function (ffbf43d4, ffbf43d0, 1, 1d4ef1, 1, ffbf43d0) + 3e4
00018db0 variable_expand_string (0, ac018, ffffffff, 74683000, 30, 30) 
+ 234
00019574 variable_expand (ac018, a9210, 1e9, 1eb, 601a0, 188330) + 14
000196c8 variable_expand_for_file (ac018, 0, ff343800, 0, ff352a00, 
a9210) + 20
00019a80 allocated_variable_expand_for_file (ac018, 0, a09e0, 0, a9210, 
85fffc2c) + 40
00018af8 recursively_expand_for_file (a9210, 0, ffffffff, 1d8c30, 
50a9db, a9210) + 1c4
000194d4 reference_variable (ff18b, ffbf47b2, f, ffbf47b2, 30, 
ffbf47c1) + ec
000192c8 variable_expand_string (ff18b, ffbf47b0, 12, 6, a9210, 5fd30) 
+ 74c
00021cb4 func_call (ff18b, ffbf48f8, 4b300, 0, 5d, 5fa00) + 48c
00021398 expand_builtin_function (ff18b, 6, ffbf48f8, 5fa00, 20, 
1d4a07) + d0
00021798 handle_function (ffbf4a0c, ffbf4a08, 1, ff18b, 2, ffbf4a08) + 3e4
00018db0 variable_expand_string (0, 1d48b4, ffffffff, 9bef0, 0, 0) + 234
00019574 variable_expand (1d48b4, 1d5090, a0940, 9bef0, 1b90, 0) + 14
000196c8 variable_expand_for_file (1d48b4, 0, 10e0, 196c8, 10c3, 
1d6178) + 20
00019a80 allocated_variable_expand_for_file (1d48b4, 0, e71a0, 
ff347344, 0, 1d48b4) + 40
00019600 expand_argument (1d48b4, 0, ff343800, 0, ff352a00, 80808080) + 74
000202c0 func_if  (ff0f4, ffbf4d78, 4b320, 448f0, ff33e3c0, 5fa30) + 10c
00021398 expand_builtin_function (ff0f4, 2, ffbf4d78, 5fa30, 20, 
1d4a08) + d0
00021798 handle_function (ffbf4ebc, ffbf4eb8, 1, ff0f4, ff00, ffbf4eb8) 
+ 3e4
00018db0 variable_expand_string (0, 1d46c2, ffffffff, 0, ff352a00, 
ff3472b8) + 234
00019574 variable_expand (1d46c2, 18, ff33e3c0, 8, a55d0, ff0000) + 14
000196c8 variable_expand_for_file (1d46c2, 0, e7974, 196c8, 1a, 1a) + 20
00019a80 allocated_variable_expand_for_file (1d46c2, 0, e71a0, 
ff347344, 0, 1d46c2) + 40
00019600 expand_argument (1d46c2, 0, ff343800, 0, ff352a00, 80808080) + 74
000202c0 func_if  (fec11, ffbf5228, 4b320, 448f0, ff33e3c0, 5fa30) + 10c
00021398 expand_builtin_function (fec11, 3, ffbf5228, 5fa30, 20, 
1d487e) + d0
00021798 handle_function (ffbf537c, ffbf5378, 1, fec11, ff000000, 
ffbf5378) + 3e4
00018db0 variable_expand_string (0, fecfb, ffffffff, 0, ff352a00, 
ff3472b8) + 234
00019574 variable_expand (fecfb, 6e005043, a546e, 6e00, ff000000, 
ff0000) + 14
000196c8 variable_expand_for_file (fecfb, 0, e71a0, ff347344, 0, a5458) 
+ 20
00019a80 allocated_variable_expand_for_file (fecfb, 0, ff343800, 0, 
ff352a00, ff001) + 40
0001f20c func_foreach (fec11, ffbf5688, 4b2f8, 448f0, ff33e3c0, 5f9f4) 
+ 15c
00021398 expand_builtin_function (fec11, 3, ffbf5688, 5f9f4, 20, feed8) 
+ d0
00021798 handle_function (ffbf57e4, ffbf57e0, 1, fec11, 1, ffbf57e0) + 3e4
00018db0 variable_expand_string (0, ac018, ffffffff, 74683000, 30, 30) 
+ 234
00019574 variable_expand (ac018, a9210, 59, 5b, 601a0, 188330) + 14
000196c8 variable_expand_for_file (ac018, 0, ff343800, 0, ff352a00, 
a9210) + 20
00019a80 allocated_variable_expand_for_file (ac018, 0, fdc74, 0, a9210, 
85fffc2c) + 40
00018af8 recursively_expand_for_file (a9210, 0, ffffffff, 997c8, 
5253c8, a9210) + 1c4
000194d4 reference_variable (11af88, ffbf5bc2, f, ffbf5bc2, 30, 
ffbf5bd1) + ec
000192c8 variable_expand_string (11af88, ffbf5bc0, 12, 6, a9210, 5fd30) 
+ 74c
00021cb4 func_call (11af88, ffbf5d08, 4b300, 2172c, 5d, 5fa00) + 48c
00021398 expand_builtin_function (11af88, 6, ffbf5d08, 5fa00, 524f78, 
ffbf6048) + d0
00021798 handle_function (ffbf5e1c, ffbf5e18, 0, 11af88, ff000000, 
ffbf5e18) + 3e4
00018db0 variable_expand_string (0, ffbf6010, ffffffff, 1d8b18, 74, 74) 
+ 234
00019574 variable_expand (ffbf6010, 5f9a0, 78, 19a80, 5fd34, 64294) + 14
000196c8 variable_expand_for_file (ffbf6010, 0, ff343800, 0, ff352a00, 
5f9a0) + 20
00019a80 allocated_variable_expand_for_file (ffbf6010, 0, ffffffff, 0, 
29, 38) + 40
0001968c expand_argument (ffbf63d7, ffbf640f, ffffffff, 74680020, 
507788, 1) + 100
0002172c handle_function (ffbf61dc, ffbf61d8, 0, fdb58, ff000000, 
ffbf61d8) + 378
00018db0 variable_expand_string (0, ffbf63d0, ffffffff, 63320000, 70, 
70) + 234
00019574 variable_expand (ffbf63d0, 5f9ac, 78, ff256eb0, 5fd34, 64374) + 14
000196c8 variable_expand_for_file (ffbf63d0, 0, ff343800, 0, ff352a00, 
5f9ac) + 20
00019a80 allocated_variable_expand_for_file (ffbf63d0, 0, ffffffff, 0, 
29, 40) + 40
0001968c expand_argument (b00cd, b010d, ffffffff, 6, 12898b, 1) + 100
0002172c handle_function (ffbf65a4, ffbf65a0, 0, 1d8a50, 1, ffbf65a0) + 378
00018db0 variable_expand_string (0, b00b8, ffffffff, 74680020, 
50617468, 200000) + 234
00019574 variable_expand (b00b8, b09f0, 69, 6b, 601a0, 188270) + 14
000196c8 variable_expand_for_file (b00b8, 0, ff343800, 0, ff352a00, 
b09f0) + 20
00019a80 allocated_variable_expand_for_file (b00b8, 0, a6188, 0, b09f0, 
85fffc2c) + 40
00018af8 recursively_expand_for_file (b09f0, 0, ffffffff, 19de70, 
5d6093, b09f0) + 1c4
000194d4 reference_variable (1cca13, ffbf6982, e, ffbf6982, 68, 
ffbf6990) + ec
000192c8 variable_expand_string (1cca13, ffbf6980, 11, 6, b09f0, 5fd30) 
+ 74c
00021cb4 func_call (1cca13, ffbf6ac8, 4b300, 0, 29, 5fa00) + 48c
00021398 expand_builtin_function (1cca13, 3, ffbf6ac8, 5fa00, 3d, 
ffbf6e0d) + d0
00021798 handle_function (ffbf6bd4, ffbf6bd0, c, 1cca13, ff00, 
ffbf6bd0) + 3e4
00018db0 variable_expand_string (0, ffbf6dc8, ffffffff, c9, 6c, 6c) + 234
00019574 variable_expand (ffbf6dc8, 5fa60, 0, 1c00, 5fd34, 64314) + 14
000196c8 variable_expand_for_file (ffbf6dc8, 0, 1161f8, 191, 3, 5fa60) + 20
00019a80 allocated_variable_expand_for_file (ffbf6dc8, 0, ffffffff, 0, 
29, 45) + 40
0001968c expand_argument (11aa6a, 11aaaf, ffffffff, 0, 20, 1) + 100
0002172c handle_function (ffbf6f94, ffbf6f90, 1, 1d8cca, 2, ffbf6f90) + 378
00018db0 variable_expand_string (0, 11a9e7, ffffffff, 0, ff352a00, 
ff3472b8) + 234
00019574 variable_expand (11a9e7, 1d8ec8, a0940, 1d8cc0, c8, 1d8cc0) + 14
000196c8 variable_expand_for_file (11a9e7, 0, 1161f8, c9, 2, 2) + 20
00019a80 allocated_variable_expand_for_file (11a9e7, 0, e71a0, 
ff347344, 0, 11a9e7) + 40
00019600 expand_argument (11a9e7, 0, ff343800, 0, ff352a00, 80808080) + 74
000202c0 func_if  (fd90b, ffbf7300, 4b320, 448f0, ff33e3c0, 5fa30) + 10c
00021398 expand_builtin_function (fd90b, 2, ffbf7300, 5fa30, 20, 
11ab30) + d0
00021798 handle_function (ffbf7424, ffbf7420, 1, fd90b, 1, ffbf7420) + 3e4
00018db0 variable_expand_string (0, 11ac1a, ffffffff, 0, ff352a00, 
ff3472b8) + 234
00019574 variable_expand (11ac1a, 92c0, 10dc0c, 0, ff000000, 80808080) + 14
000196c8 variable_expand_for_file (11ac1a, 0, e71a0, ff347344, 0, 
10dbf8) + 20
00019a80 allocated_variable_expand_for_file (11ac1a, 0, ff343800, 0, 
ff352a00, 1d8940) + 40
0001f20c func_foreach (1ccdd0, ffbf7730, 4b2f8, 448f0, ff33e3c0, 5f9f4) 
+ 15c
00021398 expand_builtin_function (1ccdd0, 3, ffbf7730, 5f9f4, ceef0, 
11af7c) + d0
00021798 handle_function (ffbf788c, ffbf7888, 0, 1ccdd0, ff000000, 
ffbf7888) + 3e4
00018db0 variable_expand_string (0, fdee0, 386, ffbf798c, 3, 3) + 234
00032cc4 eval     (ffbf7ab0, 1, 7d000, 2, 3, ffbf7ab0) + 1200
000319a8 eval_makefile (78ec0, a, ff343800, 0, a, 65) + 448
00032a00 eval     (ffbf7ce8, 1, bc6b0, 2, 3, ffbf7ce8) + f3c
000319a8 eval_makefile (ff5e0, a, ff343800, 0, a, 65) + 448
00032a00 eval     (ffbf7f20, 1, ae0f0, 80808080, ff0000, 60120) + f3c
00031a88 eval_buffer (ae0a8, ffbf7fb0, 1161f8, d9, 2, ffbf7fb0) + 94
00020620 func_eval (995ed, ffbf80a0, 4b340, 0, 6b, 5fa60) + 30
00021398 expand_builtin_function (995ed, 1, ffbf80a0, 5fa60, 20, 
119c66) + d0
00021798 handle_function (ffbf81a4, ffbf81a0, 1, 995ed, ff00, ffbf81a0) 
+ 3e4
00018db0 variable_expand_string (0, 119b33, ffffffff, a60f0, 0, 0) + 234
00019574 variable_expand (119b33, 10fe38, 98e30, ae0a0, d8, ae0a0) + 14
000196c8 variable_expand_for_file (119b33, 0, ae0a8, d1, 1, 1) + 20
00019a80 allocated_variable_expand_for_file (119b33, 0, e71a0, 
ff347344, 0, 119b33) + 40
00019600 expand_argument (119b33, 0, ff343800, 0, ff352a00, 80808080) + 74
000202c0 func_if  (19f28c, ffbf8510, 4b320, 448f0, ff33e3c0, 5fa30) + 10c
00021398 expand_builtin_function (19f28c, 3, ffbf8510, 5fa30, 20, 
119d67) + d0
00021798 handle_function (ffbf8654, ffbf8650, 1, 19f28c, 1, ffbf8650) + 3e4
00018db0 variable_expand_string (0, 9da9c, ffffffff, 0, ff352a00, 0) + 234
00019574 variable_expand (9da9c, 536861, 95258, 0, ff000000, 80808080) + 14
000196c8 variable_expand_for_file (9da9c, 0, e71a0, ff347344, 0, 95220) 
+ 20
00019a80 allocated_variable_expand_for_file (9da9c, 0, ff343800, 0, 
ff352a00, bc54c) + 40
0001f20c func_foreach (9bc28, ffbf8960, 4b2f8, 448f0, ff33e3c0, 5f9f4) 
+ 15c
00021398 expand_builtin_function (9bc28, 3, ffbf8960, 5f9f4, 2a423, 
9de68) + d0
00021798 handle_function (ffbf8acc, ffbf8ac8, 0, 9bc28, 1, ffbf8ac8) + 3e4
00018db0 variable_expand_string (0, 717f8, ffffffff, 6d6b0000, 
655f6d6b, 0) + 234
00019574 variable_expand (717f8, 69838, 39, 3b, 601a0, 1888cc) + 14
000196c8 variable_expand_for_file (717f8, 0, ff343800, 0, ff352a00, 
69838) + 20
00019a80 allocated_variable_expand_for_file (717f8, 0, 10a0c0, 657200, 
69838, 85fffc2c) + 40
00018af8 recursively_expand_for_file (69838, 0, ffffffff, ff5a8, 
50caa8, 69838) + 1c4
000194d4 reference_variable (105950, ffbf8eaa, a, ffbf8eaa, 6b, 
ffbf8eb4) + ec
000192c8 variable_expand_string (105950, ffbf8ea8, d, 6, 69838, 5fd30) 
+ 74c
00021cb4 func_call (105950, ffbf8fe8, 4b300, 0, 29, 5fa00) + 48c
00021398 expand_builtin_function (105950, 2, ffbf8fe8, 5fa00, 64620, 
a1358) + d0
00021798 handle_function (ffbf90ec, ffbf90e8, 0, 105950, ff000000, 
ffbf90e8) + 3e4
00018db0 variable_expand_string (0, a1330, 28, ffbf91ec, 3, 3) + 234
00032cc4 eval     (ffbf9310, 1, 7d000, 2, 3, ffbf9310) + 1200
000319a8 eval_makefile (1a5b68, a, ff343800, 0, a, 65) + 448
00032a00 eval     (ffbf9548, 1, ad220, 2, 3, ffbf9548) + f3c
000319a8 eval_makefile (9fba8, a, ff343800, 0, a, 65) + 448
00032a00 eval     (ffbf9780, 1, 120d07, 80808080, ff00, 60120) + f3c
00031a88 eval_buffer (120cd0, ffbf9810, 99860, c9, 2, ffbf9810) + 94
00020620 func_eval (974e5, ffbf9900, 4b340, 0, 6b, 5fa60) + 30
00021398 expand_builtin_function (974e5, 1, ffbf9900, 5fa60, 20, 98cf6) 
+ d0
00021798 handle_function (ffbf9a04, ffbf9a00, 1, 974e5, ff00, ffbf9a00) 
+ 3e4
00018db0 variable_expand_string (0, 98bc3, ffffffff, 0, ff352a00, 0) + 234
00019574 variable_expand (98bc3, 974e0, 1c91d8, 1c91d8, c8, c9) + 14
000196c8 variable_expand_for_file (98bc3, 0, 1c91e0, c9, 1, 1) + 20
00019a80 allocated_variable_expand_for_file (98bc3, 0, e71a0, ff347344, 
0, 98bc3) + 40
00019600 expand_argument (98bc3, 0, ff343800, 0, ff352a00, 80808080) + 74
000202c0 func_if  (9fcb4, ffbf9d70, 4b320, 448f0, ff33e3c0, 5fa30) + 10c
00021398 expand_builtin_function (9fcb4, 3, ffbf9d70, 5fa30, 20, 98df7) 
+ d0
00021798 handle_function (ffbf9eb4, ffbf9eb0, 1, 9fcb4, 1, ffbf9eb0) + 3e4
00018db0 variable_expand_string (0, 9907c, ffffffff, 0, ff352a00, 0) + 234
00019574 variable_expand (9907c, 6e747300, a1a24, 6e7473, ff00, ff0000) 
+ 14
000196c8 variable_expand_for_file (9907c, 0, e71a0, ff347344, 0, a1a00) 
+ 20
00019a80 allocated_variable_expand_for_file (9907c, 0, ff343800, 0, 
ff352a00, 97453) + 40
0001f20c func_foreach (98898, ffbfa1c0, 4b2f8, 448f0, ff33e3c0, 5f9f4) 
+ 15c
00021398 expand_builtin_function (98898, 3, ffbfa1c0, 5f9f4, 27093, 
99448) + d0
00021798 handle_function (ffbfa32c, ffbfa328, 0, 98898, 1, ffbfa328) + 3e4
00018db0 variable_expand_string (0, 717f8, ffffffff, 6d6b0000, 
655f6d6b, 0) + 234
00019574 variable_expand (717f8, 69838, 140, ff256eb0, 601a0, 88364) + 14
000196c8 variable_expand_for_file (717f8, 0, ff343800, 0, ff352a00, 
69838) + 20
00019a80 allocated_variable_expand_for_file (717f8, 0, 9888c, 69838, 
69838, 87fff82c) + 40
00018af8 recursively_expand_for_file (69838, 0, ffffffff, 97750, 
4983b0, 69838) + 1c4
000194d4 reference_variable (92ab8, ffbfa70a, a, ffbfa70a, 6b, 
ffbfa714) + ec
000192c8 variable_expand_string (92ab8, ffbfa708, d, 6, 69838, 5fd30) + 74c
00021cb4 func_call (92ab8, ffbfa848, 4b300, 0, 29, 5fa00) + 48c
00021398 expand_builtin_function (92ab8, 2, ffbfa848, 5fa00, ffffb1f0, 
978f0) + d0
00021798 handle_function (ffbfa94c, ffbfa948, 0, 92ab8, ff000000, 
ffbfa948) + 3e4
00018db0 variable_expand_string (0, 978c8, 28, ffbfaa4c, 3, 3) + 234
00032cc4 eval     (ffbfab70, 1, 7d000, 2, 3, ffbfab70) + 1200
000319a8 eval_makefile (74b38, a, ff343800, 0, a, 65) + 448
00032a00 eval     (ffbfada8, 1, 92da0, 2, 3, ffbfada8) + f3c
000319a8 eval_makefile (92e28, a, ff343800, 0, a, 65) + 448
00032a00 eval     (ffbfafe0, 1, 92bb4, 80808080, ff0000, 60120) + f3c
00031a88 eval_buffer (92b88, ffbfb070, 92ab8, c9, 2, ffbfb070) + 94
00020620 func_eval (8c6cd, ffbfb160, 4b340, 0, 6b, 5fa60) + 30
00021398 expand_builtin_function (8c6cd, 1, ffbfb160, 5fa60, 20, 929ae) 
+ d0
00021798 handle_function (ffbfb264, ffbfb260, 1, 8c6cd, ff00, ffbfb260) 
+ 3e4
00018db0 variable_expand_string (0, 9287b, ffffffff, 8c6c0, 0, 8c2a0) + 234
00019574 variable_expand (9287b, 927f8, 8c6c0, ff256eb0, ff347350, 0) + 14
000196c8 variable_expand_for_file (9287b, 0, 92ab8, c9, 1, 1) + 20
00019a80 allocated_variable_expand_for_file (9287b, 0, e71a0, ff347344, 
0, 9287b) + 40
00019600 expand_argument (9287b, 0, ff343800, 0, ff352a00, 80808080) + 74
000202c0 func_if  (8c5fc, ffbfb5d0, 4b320, 448f0, ff33e3c0, 5fa30) + 10c
00021398 expand_builtin_function (8c5fc, 3, ffbfb5d0, 5fa30, 20, 92aaf) 
+ d0
00021798 handle_function (ffbfb714, ffbfb710, 1, 8c5fc, 1, ffbfb710) + 3e4
00018db0 variable_expand_string (0, 919d4, ffffffff, 0, ff352a00, 
ff3472bc) + 234
00019574 variable_expand (919d4, 0, 85a74, 0, ff000000, 80808080) + 14
000196c8 variable_expand_for_file (919d4, 0, e71a0, ff347344, 0, 85a58) 
+ 20
00019a80 allocated_variable_expand_for_file (919d4, 0, ff343800, 0, 
ff352a00, 8c540) + 40
0001f20c func_foreach (89c68, ffbfba20, 4b2f8, 448f0, ff33e3c0, 5f9f4) 
+ 15c
00021398 expand_builtin_function (89c68, 3, ffbfba20, 5f9f4, 18463, 
91da0) + d0
00021798 handle_function (ffbfbb8c, ffbfbb88, 0, 89c68, 1, ffbfbb88) + 3e4
00018db0 variable_expand_string (0, 717f8, ffffffff, 6d6b0000, 
655f6d6b, 0) + 234
00019574 variable_expand (717f8, 69838, e70c4, 0, 601a0, 88364) + 14
000196c8 variable_expand_for_file (717f8, 0, ff343800, 0, ff352a00, 
69838) + 20
00019a80 allocated_variable_expand_for_file (717f8, 0, 859fc, 69838, 
69838, 87fff82c) + 40
00018af8 recursively_expand_for_file (69838, 0, ffffffff, 91f58, 
491518, 69838) + 1c4
000194d4 reference_variable (8d480, ffbfbf6a, a, ffbfbf6a, 6b, 
ffbfbf74) + ec
000192c8 variable_expand_string (8d480, ffbfbf68, d, 6, 69838, 5fd30) + 74c
00021cb4 func_call (8d480, ffbfc0a8, 4b300, 0, 29, 5fa00) + 48c
00021398 expand_builtin_function (8d480, 2, ffbfc0a8, 5fa00, ffffc100, 
913a8) + d0
00021798 handle_function (ffbfc1ac, ffbfc1a8, 0, 8d480, ff000000, 
ffbfc1a8) + 3e4
00018db0 variable_expand_string (0, 91380, 28, ffbfc2ac, 3, 3) + 234
00032cc4 eval     (ffbfc3d0, 1, 7d000, 2, 3, ffbfc3d0) + 1200
000319a8 eval_makefile (74a90, a, ff343800, 0, a, 65) + 448
00032a00 eval     (ffbfc608, 1, 898b8, 2, 3, ffbfc608) + f3c
000319a8 eval_makefile (8d580, a, ff343800, 0, a, 65) + 448
00032a00 eval     (ffbfc840, 1, 75698, 80808080, ff0000, 60120) + f3c
00031a88 eval_buffer (75670, ffbfc8d0, 754d0, c9, 2, ffbfc8d0) + 94
00020620 func_eval (8dabd, ffbfc9c0, 4b340, 0, 6b, 5fa60) + 30
00021398 expand_builtin_function (8dabd, 1, ffbfc9c0, 5fa60, 20, 8d9ae) 
+ d0
00021798 handle_function (ffbfcac4, ffbfcac0, 1, 8dabd, ff00, ffbfcac0) 
+ 3e4
00018db0 variable_expand_string (0, 8d87b, ffffffff, 8d478, 0, 0) + 234
00019574 variable_expand (8d87b, 754d0, 7cc40, 8d7f0, 3a0, 0) + 14
000196c8 variable_expand_for_file (8d87b, 0, 8d480, e1, 1, 1) + 20
00019a80 allocated_variable_expand_for_file (8d87b, 0, e71a0, ff347344, 
0, 8d87b) + 40
00019600 expand_argument (8d87b, 0, ff343800, 0, ff352a00, 80808080) + 74
000202c0 func_if  (8d3b4, ffbfce30, 4b320, 448f0, ff33e3c0, 5fa30) + 10c
00021398 expand_builtin_function (8d3b4, 3, ffbfce30, 5fa30, 20, 8daaf) 
+ d0
00021798 handle_function (ffbfcf74, ffbfcf70, 1, 8d3b4, 1, ffbfcf70) + 3e4
00018db0 variable_expand_string (0, 894e4, ffffffff, 0, ff352a00, 
ff3472b8) + 234
00019574 variable_expand (894e4, 6677, 617b0, 0, ff000000, 80808080) + 14
000196c8 variable_expand_for_file (894e4, 0, e71a0, ff347344, 0, 61798) 
+ 20
00019a80 allocated_variable_expand_for_file (894e4, 0, ff343800, 0, 
ff352a00, 86204) + 40
0001f20c func_foreach (72390, ffbfd280, 4b2f8, 448f0, ff33e3c0, 5f9f4) 
+ 15c
00021398 expand_builtin_function (72390, 3, ffbfd280, 5f9f4, b8b, 
898b0) + d0
00021798 handle_function (ffbfd3ec, ffbfd3e8, 0, 72390, 1, ffbfd3e8) + 3e4
00018db0 variable_expand_string (0, 717f8, ffffffff, 6d6b0000, 
655f6d6b, 0) + 234
00019574 variable_expand (717f8, 69838, e70c4, 0, 601a0, 88364) + 14
000196c8 variable_expand_for_file (717f8, 0, ff343800, 0, ff352a00, 
69838) + 20
00019a80 allocated_variable_expand_for_file (717f8, 0, 61590, 0, 69838, 
85fffc2c) + 40
00018af8 recursively_expand_for_file (69838, 0, ffffffff, 8ceb0, 
469ff0, 69838) + 1c4
000194d4 reference_variable (677b8, ffbfd7ca, a, ffbfd7ca, 6b, 
ffbfd7d4) + ec
000192c8 variable_expand_string (677b8, ffbfd7c8, d, 6, 69838, 5fd30) + 74c
00021cb4 func_call (677b8, ffbfd908, 4b300, 0, 29, 5fa00) + 48c
00021398 expand_builtin_function (677b8, 2, ffbfd908, 5fa00, ffff1ed8, 
75908) + d0
00021798 handle_function (ffbfda0c, ffbfda08, 0, 677b8, ff000000, 
ffbfda08) + 3e4
00018db0 variable_expand_string (0, 758e0, 28, ffbfdb0c, 3, 3) + 234
00032cc4 eval     (ffbfdc30, 1, 7d000, 2, 3, ffbfdc30) + 1200
000319a8 eval_makefile (78ed8, a, ff343800, 0, a, 65) + 448
00032a00 eval     (ffbfde68, 1, 6adb0, 2, 3, ffbfde68) + f3c
000319a8 eval_makefile (ffbfefb1, 0, ff343800, 0, ff352a00, ffbfdf1c) + 448
000311d0 read_all_makefiles (672f0, d, 4cd10, 2, 0, 60598) + 198
0002b858 main     (9, ffbfecf4, ffbfed1c, 6059c, ff3500c0, 0) + 1420
00013c4c _start   (0, 0, 0, 0, 0, 0) + 5c