Re: GNU Parallel Bug Reports feature suggestion: approximate scheduler

[Ole Tange]

On Thu, Sep 3, 2015 at 12:56 AM, Håkan Johansson <address@hidden> wrote:
:
> Otherwise, description of a typical invocation:
:
> - a list of 2 - 15 machines of varying speed (within a factor ~4)
>   and number of cores

I am a bit surprised by this: Are you saying that you have machines
with effective CPU speeds varying from around 1 GHz to 4 GHz?

> What I would imagine is something like this:
>
> - an option to request  parallel to try to schedule the jobs such that they
>   end more simultaneously (and thus sooner).  Given no other information,
>   the estimate  would be that it takes time proportional to the input {}
>   file size to complete the job.  I.e. start scheduling the largest
>   files and end with the smallest ones.

Unless --shuf is given, GNU Parallel schedules the jobs in the order
they are read.

Your suggestion assumes {} is a single file, and that cannot generally
be assumed for input to GNU Parallel - especially not when using
multiple input sources. Therefore I will suggest you simply write a
wrapper for your purpose that sorts the files by size:

ls | perl -e 'chomp(@a=<>);print map{"$_\n"} sort { -s $b <=> -s $a }
@a' | parallel ...

> - an option to read a machine-performance file, which gives the relative
>   performance of the machines.  (If some is missing, assume a default
>   value of e.g. 1).
>
>   Especially as the end is being reached, completetion may be quicker
>   if parallel would not use the slowest machines at all, or only for
>   small enough files.

If sorted by size this becomes less of an issue.

If it is still an issue remember that you can change the machines to
use by changing the file given for --slf while GNU Parallel is
running: So when 90% of the jobs are done you could simply remove all
the slow machines from the file.

Also the 'ncpu/hostname' syntax may be useful to you.

> - an option to read an file of input-file difficulty corrections, i.e.
>   telling (relatively) how 'large' each file effectively is.

This is too specific for your task. Write a wrapper for sorting the
jobs with this correction.

> - an option to produce a log file listing for each job:
>
>   date/time of job,  machine,  input-file,  size-of-input-file,  time used

Again you here assume the input is a file. That is not generally the
case. But have a look at --joblog. It will do most of these things.

> A separate program could then process (multiple) such files to produce the
> machine and input-file correction files.

Looks like a task for the wrapper.

> I have made a mock-up test of the equation solving by producing the log-file
> by wrapping each job in a script that measures the time etc, and then a
> small perl script to produce the equation system for solving with e.g.
> octave.  It produces machine performance and file difficulty values which
> look reasonable.  If interesting, I'll be happy to share the script etc.

Post on address@hidden and see if others would be interested. If
there is a general interest, I will be happy to consider distributing
the wrapper as an additional file.


/Ole