[Findutils-patches] added new xargs option -G, --load-limit

[Sven C. Dack]

Hello,

I'm sending you a patch, which adds load limiting to the xargs command. 
The option is '-G <max-load>' or '--load-limit <max-load>'.

The option allows one to respect the current system load while running 
processes in parallel through xargs. This is most useful for 
environments with changing system loads (i.e. multi-user environments, 
build systems, etc.) where shell scripts shall make the most use of 
multiple CPU cores without running idle or taking up too many system 
resources. So do some processes make use of multiple cores on their own, 
while others do not, or they can become idle while waiting for I/O 
operations to complete. This can making it difficult to find a good 
number for running processes in parallel with xargs. Being able to limit 
the number of processes with respect to the current system load can 
produce a fair balance for such cases.

Some test examples:

$ seq 1 100 | xargs -n1 -I@ -P4 -- sh -c "echo @; dd if=/dev/urandom 
bs=256k count=@ status=none | base64 | zstd -19 -T4 >/dev/null"

This will produce a series of 100 files with an increasing size for zstd 
to compress. zstd can run with multiple threads (here 4 with '-T4') and 
thus the '-P4' given to xargs will start 4 parallel instances of zstd, 
which can lead to a system load beyond 4.

$ seq 1 100 | xargs -n1 -I@ -P4 -G4 -- sh -c "echo @; dd if=/dev/urandom 
bs=256k count=@ status=none | base64 | zstd -19 -T2 >/dev/null"

Here '-P4 -G4' will start up to 4 parallel instances of zstd, but once 
the system load reaches 4 will xargs slow down and start fewer 
processes, and the system load begins to converge towards 4.

With '-P0 -G4' can one run any number of processes in parallel, only 
limited by the system load. A '-P8 -G4' will run between 1 to 8 
processes. Note that the option '-G' will not completely halt xargs, but 
only slow it down to execute one process at a time.

The implementation of the option relies on /proc/loadavg as found on 
Linux systems. It reads the number of active threads and processes from 
the file (the 4th value) before the executing a new command. Note, this 
is not the same as using the load averages. This value is updated by the 
kernel in real-time and represents the number of currently active, 
running threads and processes. Thus the option is only supported for 
Linux or on systems with a similar /proc/loadavg file.

The configure script will determine automatically if a usable 
/proc/loadavg exists, but it can also be overwritten with 
--enable-load-limiter=yes,no.

Cheers,
Sven

load-limit.patch
Description: Text Data