Re: [PATCH 1/2] maint.mk: Split long argument lists

[Roman Bolshakov]

On Tue, Nov 27, 2018 at 07:19:43PM +0100, Bruno Haible wrote:
> Hi,
> 
> > The workaround is to split argument list into chunks that operating
> > system can process. "getconf ARG_MAX" is used to determine size of the
> > chunk.
> 
> Two questions on this:
> 
> 1) People say that 'getconf ARG_MAX' returns the appromixate number
>    of bytes in a command line. [1]
>    But you use it with 'xargs -n', which gives a limit on the number of
>    arguments. Shouldn't the patch use 'xargs -s' instead?
> 

Hi Bruno,

You're right about "-s", the patch should probably use it. I started
from "-n" with a reasonably low value of arguments (IIRC it was 8000
arguments), then tried to lookup if there's system limit of arguments. I
haven't found one, used ARG_MAX instead.

> 2) The really available values are slightly smaller.
> 
>    On Linux:
>    $ getconf ARG_MAX
>    2097152
>    $ LC_ALL=C xargs --show-limits
>    Your environment variables take up 4744 bytes
>    POSIX upper limit on argument length (this system): 2090360
>    POSIX smallest allowable upper limit on argument length (all systems): 4096
>    Maximum length of command we could actually use: 2085616
>    Size of command buffer we are actually using: 131072
> 
>    On FreeBSD/x86_64:
>    $ getconf ARG_MAX
>    262144
>    $ LC_ALL=C xargs --show-limits
>    Your environment variables take up 353 bytes
>    POSIX upper limit on argument length (this system): 259743
>    POSIX smallest allowable upper limit on argument length (all systems): 4096
>    Maximum length of command we could actually use: 259390
>    Size of command buffer we are actually using: 131072
> 
>    On macOS:
>    $ getconf ARG_MAX
>    262144
>    $ LC_ALL=C xargs --show-limits
>    Your environment variables take up 1262 bytes
>    POSIX upper limit on argument length (this system): 258834
>    POSIX smallest allowable upper limit on argument length (all systems): 4096
>    Maximum length of command we could actually use: 257572
>    Size of command buffer we are actually using: 131072
> 
>    How about being conservative and dividing the limit by 2, to avoid
>    this margin error?
> 
> Could it be that your patch works only because xargs uses a command buffer
> of length 131072, regardless of the value you pass to '-n'?
> 
> Bruno
> 
> [1] 
> https://www.cyberciti.biz/faq/linux-unix-arg_max-maximum-length-of-arguments/
> 

This is an interesting coincidence. If we pick big enough value for "-n"
that drains command buffer length, the number of arguments is going
to be limited by default value of "-s" flag. And ARG_MAX of arguments
will always drain command buffer length up to the limit.

Here are related excerpts from macOS xargs man page:
  -n _number_
               Set the maximum number of arguments taken from standard
               input for each invocation of utility.  An invocation of
               utility will use less than _number_ standard input arguments
               if the number of bytes accumulated (see the -s option)
               exceeds the specified _size_ or there are fewer than _number_
               arguments remaining for the last invocation of utility.
               The current default value for _number_ is 5000.
  -s _size_
               Set the maximum number of bytes for the command line length
               provided to utility.  The sum of the length of the utility
               name, the arguments passed to utility (including NULL
               terminators) and the current environment will be less than
               or equal to this number.  The current default value for
               _size_ is ARG_MAX - 4096.

And from GNU xargs:
  -n _max-args_, --max-args=_max-args_
                Use  at most _max-args_ arguments per command line.
                Fewer than _max-args_ arguments will be used if the size
                (see the -s option) is exceeded, unless the -x option is
                given, in which case xargs will exit
  -s _max-chars_, --max-chars=_max-chars_
                Use  at  most _max-chars_ characters per command line,
                including the command and initial-arguments and the
                terminating nulls at the ends of the argument strings.
                The largest allowed value is system-dependent, and is
                calculated as the argument length limit for exec, less
                the size of your environment, less 2048 bytes of
                headroom.  If this value is more than 128KiB, 128Kib is
                used as the default value; otherwise, the default value
                is the maximum.  1KiB is 1024 bytes.  xargs
                automatically adapts to tighter constraints.


Given that even if the patch is kept as is it should work properly on
macOS, FreeBSD (the docs are very close to macOS xargs) and all systems
with GNU xargs. I can correct commit message to note the observation.

Alternatively, we can replace "-n" with "-s", as you pointed out in 1).
But then we will need to correct calculation of VC_ARG_MAX. We can take
formulae from [2]:
expr `getconf ARG_MAX` - `env|wc -c` - `env|egrep '^[^ ]+='|wc -l` \* 4 - 2048

But it's a bit higher than effective limit of command line buffer on
macOS/FreeBSD. To fix that we need to replace 2048 with 4096 in the
formulae (according to man page above), so I think the final VC_ARG_MAX
should be:
expr `getconf ARG_MAX` - `env|wc -c` - `env|egrep '^[^ ]+='|wc -l` \* 4 - 4096

[2] https://www.in-ulm.de/~mascheck/various/argmax/

Thank you,
Roman