Re: parallel refusing to take my request

[Ecks Hecker]

_Request for help_

Reiterating after 2 more weeks of failing to get what i want...

basically, i am using a script to create a directory of sparse files,
mainly like this:
> find "$SOURCE" -type f > filelist
> wc filelist # (returns 6917    6917  797053)
> time createdirectoryfrom "$SOURCE" "$TARGET" < filelist # (shows about
> 1 minute before it is done)
i tried to split the last task onto cores like this:
> cat filelist | parallel --pipe -j24 createdirectoryfrom "$SOURCE"
> "$TARGET"
but i keep getting the following warning:
> Warning: Only enough file handles to run 252 jobs in parallel.
> parallel: Warning: Running 'parallel -j0 -N 252 --pipe parallel -j0' or
> parallel: Warning: raising 'ulimit -n' or 'nofile' in
> /etc/security/limits.conf
> parallel: Warning: or /proc/sys/fs/file-max may help.
And from the time used, i can conclude, that still only one thread is used.
As noted below, none of the suggestions does help, and even the message
itself does not seem to be strictly correct. The server, i am on , has 2
CPUs with 16 Cores each, ulimit -n shows > 1000.

What am i doing wrong, or is this really a bug in parallel?
I am using
> GNU parallel 20201122
Please hint me out of my stuck state. :-)
Ecks

Am 08.12.2020 um 12:59 schrieb Ecks Hecker:
> Hi,
>
> after years of using parallel happily, silently on my private pc, i am
> presumeably facing a PEBKAC, and could not resolve it in several days.
>
> Basically, i am building a tree of sparse files resembling the content
> of an external device, intending it to be searchable easily with find,
> even if is disconnected.
>
> But since there are almost 10000 files, just using find on all of them
> gives around 1M of filenames.
>
> To execute a combination of truncate and touch on them on a single core
> takes about a minute, which is calling for using parallel (on my 32 cpu
> server). But ...
>
> I am running into this (on debian stable):
>
>> parallel: Warning: Only enough file handles to run 252 jobs in parallel.
>> parallel: Warning: Running 'parallel -j0 -N 252 --pipe parallel -j0' or
>> parallel: Warning: raising 'ulimit -n' or 'nofile' in
>> /etc/security/limits.conf
>> parallel: Warning: or /proc/sys/fs/file-max may help.
> Well, i did raise the 'nofile' limit as indicated, even tried the --pipe
> construct, to no avail.
>
> ulimit -n gives 1024
>
> Apart from that, there seems to be some misunderstanding from my side,
> as i am going round in circles without making progress. Any hints would
> be greatly appreciated. Apparently, this one is above my paygrade and
> skill level. :-(
>
> regards, Ecks