Re: Properly scheduling multi-threaded/process jobs

[FUNG Yik Him Zachary]

Hi,

Sorry for digging up an old thread from Aug-2014.

 

I come with a situation that the inputs of my application vary in a wide range such that their running speed could swing from I/O-bounded to computation-bounded, and therefore there are different efficient numbers of CPU cores to be used in different jobs. It would be also nice to me if GNU parallel would be able aware of the job size (job slots to be occupied by each job).

 

I try to use GNU parallel instead of queue systems because there are nice data handling features “--basefile/--return” and spamming the cluster queue with thousands of small tasks is not appreciated by the cluster’s admin. That’s why I am echoing this old idea =]

 

Zachary