Re: [Help-bash] add and expand

[Greg Wooledge]

On Thu, Jun 08, 2017 at 02:02:20AM +0000, Val Krem wrote:
> I am trying to develop a pipeline procedure in SLURM
> (https://hpc.nih.gov/docs/job_dependencies.html).
> 
>   
> I have two jobs  job1.sh and job2.sh. Job2.sh deepened on job1.sh  and give 
> job dependency syntax  when  I am submitting job2.sh
> 
> 
> In SLURM,  when I submit the job the syntax is as follow
> 
> 
> JJ1=$(sbatch job1.sh)

This is not a background process.  This is a FOREGROUND process.  The
sbatch command runs in the foreground, and the command substitution and
variable assignment (and therefore the entire bash script) are "paused",
waiting until the sbatch command's standard output is closed.

Once sbatch closes stdout (usually by terminating), the entire contents
that were written to stdout are grabbed, trailing newlines are trimmed,
NUL bytes are removed (with or without a warning, depending on the version
of bash), and then the result is stored in the JJ1 variable.

Calling this a "job" is incredibly misleading.  The use of the English
word "job" in a shell programming context implies something running
in the background.

> JJ1 will give the job id or PID (numeric value) for the job1.sh.
> echo ${JJ1}  will produce  number like this  11254323

Wrong wrong wrong wrong wrong.

...

Unless you're claiming that your "sbatch" command is self-backgrounding.
If "sbatch" is an evil self-backgrounding abomination, which forks and
abandons a child process, and then writes the child's process ID to
stdout, then... what you say may be literally true, but I would run
away, screaming.  This is a horror show.  It's the worst mistakes of
the 1980s, risen anew.

> when I submit job2.sh
> 
> JJ2=$(sbatch --dependency=afterok:${JJ1} job2.sh)
> 
> 
> In this case Job2.sh runs only if job1.sh finishes.
> For single job submission it is ok.

Oh dear god, no.

I'm out.  Sorry.  I can't even ... no.  Just no.

You would be better off throwing this entire infrastructure away, and
building your own from scratch.

Did you know that you can create dependencies among systemd units?
Maybe that would be a better framework for doing this thing of yours.
Set up, say, one systemd service that provides the things your second
service needs.  Let's call this first service "dirt", because that is a
BETTER and MORE USEFUL name than "job1.sh".  Let's call the other service
"grass", because grass depends on dirt.  This is a BETTER and MORE USEFUL
name than "job2.sh".

So, then you ask yourself: "What does dirt do?"  Well, we don't
know, because you haven't SAID a single useful thing ANYWHERE.  All
we know is that you need it to run.  To be running?  To have executed
at least one time since boot?  Let's assume it has to be continuously
running, like a database engine or something.

To do that, you would figure out whatever command you have to run to
make the dirt service "go".  To be up and running, listening on its
unix domain socket, or whatever it does, so that the grass service can
use it.

IF AT ALL POSSIBLE, the command you write to invoke the dirt service
should be a FOREGROUND process.  It should not self-background.  It should
not fork and abandon a child, like some 1980s BSD abomination.  Then you
set up a systemd service that will run this foreground process, as the
correct user, with the correct environment variables, in the correct
directory, etc.

I won't try to show this here.

Then you set up the grass service, and tell systemd that it depdends
on dirt.  If you also run grass as a foreground process, then it may
be something as simple as this:

==== example /etc/systemd/system/grass.service
[Unit]
Description=Grass service, grows in the dirt
Requires=dirt.service
After=dirt.service

[Service]
ExecStart=/usr/local/bin/grass ...
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target
====

I'm not saying systemd is great, and I'm not yet ready to attempt to
write real documentation for systemd (although god knows it NEEDS it,
BADLY).  But it sounds like it could be a match for your poorly defined,
obfuscated task.

This is quite a bit outside the scope of "help-bash".  Setting up system
services with dependencies on each other is a system administration task,
not a bash programming task.  Bash isn't even INVOLVED.

If your target system doesn't use systemd, then... well.  Go to your
operating system's mailing list, explain what you are actually trying to
do, with *real names*, and actually explain what each of your processes
*does*, and the actual nature of their relationship, and maybe someone
can come up with a solution.