Re: [Help-bash] add and expand

[Dennis Williamson]

On Jun 7, 2017 9:07 PM, "Val Krem" <address@hidden> wrote:

Hi Greg and all,

Let me explain  and give a practical example again.
I am trying to develop a pipeline procedure in SLURM
(https://hpc.nih.gov/docs/job_dependencies.html).


I have two jobs  job1.sh and job2.sh. Job2.sh deepened on job1.sh  and give
job dependency syntax  when  I am submitting job2.sh


In SLURM,  when I submit the job the syntax is as follow


JJ1=$(sbatch job1.sh)

JJ1 will give the job id or PID (numeric value) for the job1.sh.
echo ${JJ1}  will produce  number like this  11254323

when I submit job2.sh

JJ2=$(sbatch --dependency=afterok:${JJ1} job2.sh)


In this case Job2.sh runs only if job1.sh finishes.
For single job submission it is ok.

My issue is when I  am submitting in an array.

There three jobs runs separately  for each city and the final job combine

all the results of the three jobs and do some analysis.


#! /bin/bash
    #sbatch is a command in SLURM


city= "NY LA  DC"

for tr in  ${city};
         do
              dp[${#tr]=$(sbatch  city${city}.sh );
  done

When I put echo statement within the do loop

echo  ${dp[${#tr}]} I got
14547
14548
14549


The next job depends on these three job ids.
if I knew these number at the time of job submission then I do rgis
    comb=(sbatch --dependency=afterok:14547,14548,14549  combine.sh )

will work fine.

Since I don't know these numbers at the time of the submission,I have to
pass it as variable

Let us assume the variable newvar contain the above numbers  =
14547,14548,14549

comb=(sbatch --dependency=afterok:${newvar)  combine.sh )
will do the job


The complete job submission for the two jobs


#! /bin/bash
#sbatch is a command in SLURM

city= "NY LA DC"

# Job1

for tr in ${city};
     do
         dp[${#tr]=$(sbatch city${city}.sh );
    don
#job2.

comb=(sbatch --dependency=afterok:${newvar) combine.sh )


So my question is how to get those three job ids out of the for loop.
Here is my revised  attempt



#! /bin/bashcity= "NY LA  DC"
count=0

# job1.

for tr in  ${city};
    do
                 dp[${#tr]=$(sbatch  city${city}.sh );
                 array[$count]=${dp[${#tr}]}
                 ((count++))
          done

H1="address@hidden"
newvar=$(echo "$H1" | sed 's/[[:blank:]]/,/g'); export newvar
echo "$newvar"


# Job2.

comb=(sbatch --dependency=afterok:${newvar) combine.sh )

Is there a better and clean way of doing this?










On Tuesday, June 6, 2017 4:13 PM, Greg Wooledge <address@hidden> wrote:



On Tue, Jun 06, 2017 at 08:48:25PM +0000, Val Krem wrote:
> I have two jobs running one after the other (job2 run after job1). Job1
has three jobs running within for loop.

Background processes?

> val="Four five six"
>
> Job1.
>    for a in $val

STOP THIS!  Stop putting "lists" in a string variable with spaces between
things.  Use an array.

>             do
>           Within this loop three jobs are submitted
>         done

Are you running *background jobs*?  Or what?  What do you do inside the
loop?

> Job2. This job should run after the three jobs completed.

Are they background jobs?  Did you capture their PIDs with $! one by
one?  Where did you store the PIDs?  Or do you simply want to call
"wait" to wait for all the background jobs to complete?

> My issue to extract the three PIDs and export them as one variable

EXPORT?!  Why?

> Here is my attempt
>   for tr in ${val};

I thought your loop iterator was "a", not "tr".  Why did you change it?
What does "a" mean?  What does "tr" mean?  Why did you choose these
variables?

>          do
>
>            job[${#tr}]= some process;

You are creating an array named job and indexing it by the LENGTH of
the string variable tr?  Huh?  What?

What is "some process"?

Your assignment has a god damned SPACE after the = sign so it doesn't
even work.

>                 tt1="${job[${#tr}]}"     #### gets the PIDs for each job
>             echo $tt1
>         done

Can't even bring myself to comment any more.  So tired.

> The echo statement within the for loop  produced the three PID like
>
> 1009
> 1010
> 1011

"... BY SOME MIRACLE I have an array with three elements."

Let's just assume you got these values somehow.  They are in an array
named "job".  I can't for the life of me figure out HOW you got them,
because your code isn't even code, and even your *fake* code is a
damned disaster, but let's say you have an array.

job=(1009 1010 1011)

> I want this three  PIDs  to be  exported as one variable like,
>
> test2=$1009,$1010,$1011

I told you how to do this already.  You use the [*] expansion in
single quotes to join an array into a string variable, using the
first character of IFS.

IFS=,
export test2="${job[*]}"
unset IFS

Are those dollar signs supposed to be part of the string?  Then you
could do a fancypants expansion.

IFS=,
export test2="${job[*]/#/\$}"
unset IFS


WHY are you exporting this crazy string into the environment with a
useless name like "test2" and inscrutable contents?  What are you doing
with it?

If you just want to launch 3 background processes and wait for them all
to finish, you don't need ANY of this crap.

for i in 1 2 3; do
    ./my background job "$i" &

done
wait

That's it!


What is the purpose of the dp array? You store the output of snatch, which
I presume is your job numbers, in the *SAME* position in the dp array on
*EACH* iteration of the loop! You're thus wasting the array!

You could, if you do it properly, save the job numbers in an array and
convert that to a comma separated string. Or you could just build a string
as you go.

for i in "$var"
do
    string=$separator$(foo "$var")
    separator=","
done

bar "$string"

I really don't think you need to use export since you're passing variables'
values as positional arguments.

By the way, if you come here and start talking about jobs without
qualifying what you mean by that, we all think you're talking about what
Bash considers to be jobs,