Re: Feature Request: Special input value to create a barrier

[Achim Gratz]

Ole Tange writes:
> I can see how this would make sense from a user perspective, so I am
> not opposed to the idea.

OK.

> You _could_ just do:
>
>   parallel --barrier + do_something '{}' ::: job1 job2 job3
>   parallel --barrier + do_something '{}' ::: job4 job5
>   parallel --barrier + do_something '{}' ::: job6 job7
>
> but I can see how the syntactic sugar could make some tasks easier.

I know that I can already do what I asked by running multiple
invocations of parallel (or sem, as I already do for other stuff).
However, that'd require to split things someplace else in the particular
piepeline I am using.  So far I've had everything just serialized in
dependency order, but since I have a much more powerful machine to run
things by now it'd be useful to extract the available parallelism
inherent in the dependency ordering.

> What should this do:
>
>   parallel --barrier + dostuff ::: 1 2 3 + 4 5 + 6 7 ::: a + b c + d e
> f + g + h :::+ A B C D E + F G H

Well, that doesn't really make sense unless the resulting sequence has
the barrier in all arguments simultaneously.  Combinatorial
parallelization is not likely to have a dependency graph leading to a
rank ordering anyway.

> We already have -E to make GNU Parallel stop reading when hitting a
> special value. This gets us partly there.

Interesting idea.  That might actually be workable if I can run the next
invocation(s) of parallel by telling to skip the first (and following)
"virtual" EOF until it finally hits the real one.  At least in the
application I have in mind, reading the input multiple times is no
problem of any kind.

> I think, however, it will be quite tricky make GNU Parallel resume
> after it finished all running jobs. Nothing in the design is made for
> resuming. I am not saying it cannot be done, but I will be surprised
> if it is an easy change.

As I outlined above, that might not be quite as tricky as you seem to
think as long as we can have two kinds of EOF.

> So patch welcome (but I have a feeling it will be a rather big change).

I'll put it on my todo list.  No promises of when I'll have time to look
at it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra