[Automake-NG] Avoding issues with command-line length limits (was: Re: [PATCH 2/2] [ng] perf: optimize 'am__strip_suffixes' for speed)

[Stefano Lattarini]

Hi Bob.

On 05/22/2012 04:34 AM, Bob Friesenhahn wrote:
> On Mon, 21 May 2012, Akim Demaille wrote:
>>> + In mainline automake (commit v1.12-75-gd89da9c):
>>>  - "make all"                           0.9 seconds
>>>  - "make recheck"                       3.5 seconds
>>>  - "make check RECHECK_LOGS="          14.1 seconds
>>
>> Congrats!
> 
> Yes, this is excellent work/progress.
> 
> Please add performance for 'make clean' to the list of
> commands to make faster.  This one should be low-hanging
> fruit.
> 
> Currently Automake does a 'rm -f' for many files individually
> and this is very slow.
>
Yes, known issue: <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=10697>

> Presumably GNU make can help make this better.  If we are
> willing to require GNU make, then perhaps we can be willing
> to require a working 'xargs' implementation to help optimize
> performance within command-line length limits.
>
This would not help sadly, because the command-line length limits might
still trip the shell executing the recipe; i.e., a rule like

   clean:
       rm -f TOO-LONG-LIST

would cause the make-spawned shell itself to die for "command line
length exceeded"; rewriting that as

   clean:
       echo TOO-LONG-LIST | xargs rm -f

obviously won't help (we've even *increased* the command line length,
in fact!)

What we need is some smart GNU make function that acts like xargs,
so that:

  clean:
      $(am__xargs,rm -f,TOO-LONG-LIST)

will "expand" to something like:

  clean:
      rm -f SUBLIST-1
      ...
      rm -f SUBLIST-n

where all the sublist are ensured to be short enough not to cause
command line length issues, and where of course

   SUBLIST-1 ... SUBLIST-n = TOO-LONG-LIST

Alternatively, if we could assume GNU make >= 3.83 (still unreleased at the
time of writing), we might take advantage of the new $(file) builin to write
the long list to a temporary file, and processing it from there:

  clean:
      $(file > t,TOO-LONG-LIST)
      $(am__base_list) <t | while read files; do rm -f $$files; done;
      rm -f t

Regards,
  Stefano