[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Findutils-patches] added new xargs option -G, --load-limit

From: Sven C. Dack
Subject: [Findutils-patches] added new xargs option -G, --load-limit
Date: Tue, 22 Aug 2017 15:56:12 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1


I'm sending you a patch, which adds load limiting to the xargs command. The option is '-G <max-load>' or '--load-limit <max-load>'.

The option allows one to respect the current system load while running processes in parallel through xargs. This is most useful for environments with changing system loads (i.e. multi-user environments, build systems, etc.) where shell scripts shall make the most use of multiple CPU cores without running idle or taking up too many system resources. So do some processes make use of multiple cores on their own, while others do not, or they can become idle while waiting for I/O operations to complete. This can making it difficult to find a good number for running processes in parallel with xargs. Being able to limit the number of processes with respect to the current system load can produce a fair balance for such cases.

Some test examples:

$ seq 1 100 | xargs -n1 -I@ -P4 -- sh -c "echo @; dd if=/dev/urandom bs=256k count=@ status=none | base64 | zstd -19 -T4 >/dev/null"

This will produce a series of 100 files with an increasing size for zstd to compress. zstd can run with multiple threads (here 4 with '-T4') and thus the '-P4' given to xargs will start 4 parallel instances of zstd, which can lead to a system load beyond 4.

$ seq 1 100 | xargs -n1 -I@ -P4 -G4 -- sh -c "echo @; dd if=/dev/urandom bs=256k count=@ status=none | base64 | zstd -19 -T2 >/dev/null"

Here '-P4 -G4' will start up to 4 parallel instances of zstd, but once the system load reaches 4 will xargs slow down and start fewer processes, and the system load begins to converge towards 4.

With '-P0 -G4' can one run any number of processes in parallel, only limited by the system load. A '-P8 -G4' will run between 1 to 8 processes. Note that the option '-G' will not completely halt xargs, but only slow it down to execute one process at a time.

The implementation of the option relies on /proc/loadavg as found on Linux systems. It reads the number of active threads and processes from the file (the 4th value) before the executing a new command. Note, this is not the same as using the load averages. This value is updated by the kernel in real-time and represents the number of currently active, running threads and processes. Thus the option is only supported for Linux or on systems with a similar /proc/loadavg file.

The configure script will determine automatically if a usable /proc/loadavg exists, but it can also be overwritten with --enable-load-limiter=yes,no.


Attachment: load-limit.patch
Description: Text Data

reply via email to

[Prev in Thread] Current Thread [Next in Thread]