Re: [Help-bash] add and expand

[Val Krem]

Hi Greg and all,

Let me explain  and give a practical example again.
I am trying to develop a pipeline procedure in SLURM
(https://hpc.nih.gov/docs/job_dependencies.html).

  
I have two jobs  job1.sh and job2.sh. Job2.sh deepened on job1.sh  and give job 
dependency syntax  when  I am submitting job2.sh


In SLURM,  when I submit the job the syntax is as follow


JJ1=$(sbatch job1.sh)

JJ1 will give the job id or PID (numeric value) for the job1.sh.
echo ${JJ1}  will produce  number like this  11254323

when I submit job2.sh

JJ2=$(sbatch --dependency=afterok:${JJ1} job2.sh)


In this case Job2.sh runs only if job1.sh finishes.
For single job submission it is ok.

My issue is when I  am submitting in an array. 

There three jobs runs separately  for each city and the final job combine 

all the results of the three jobs and do some analysis.


#! /bin/bash
    #sbatch is a command in SLURM


city= "NY LA  DC"

for tr in  ${city};
         do 
              dp[${#tr]=$(sbatch  city${city}.sh );
  done

When I put echo statement within the do loop 

echo  ${dp[${#tr}]} I got
14547
14548
14549


The next job depends on these three job ids.
if I knew these number at the time of job submission then I do rgis
    comb=(sbatch --dependency=afterok:14547,14548,14549  combine.sh )

will work fine.

Since I don't know these numbers at the time of the submission,I have to pass 
it as variable

Let us assume the variable newvar contain the above numbers  = 14547,14548,14549

comb=(sbatch --dependency=afterok:${newvar)  combine.sh )
will do the job


The complete job submission for the two jobs 


#! /bin/bash
#sbatch is a command in SLURM

city= "NY LA DC"

# Job1

for tr in ${city};
     do
         dp[${#tr]=$(sbatch city${city}.sh );
    don
#job2.

comb=(sbatch --dependency=afterok:${newvar) combine.sh )


So my question is how to get those three job ids out of the for loop.
Here is my revised  attempt



#! /bin/bashcity= "NY LA  DC"
count=0

# job1.

for tr in  ${city};
    do 
                 dp[${#tr]=$(sbatch  city${city}.sh );
                 array[$count]=${dp[${#tr}]}
                 ((count++))
          done

H1="address@hidden"
newvar=$(echo "$H1" | sed 's/[[:blank:]]/,/g'); export newvar
echo "$newvar"


# Job2.

comb=(sbatch --dependency=afterok:${newvar) combine.sh )

Is there a better and clean way of doing this?










On Tuesday, June 6, 2017 4:13 PM, Greg Wooledge <address@hidden> wrote:



On Tue, Jun 06, 2017 at 08:48:25PM +0000, Val Krem wrote:
> I have two jobs running one after the other (job2 run after job1). Job1 has 
> three jobs running within for loop.

Background processes?

> val="Four five six"
> 
> Job1.
>    for a in $val

STOP THIS!  Stop putting "lists" in a string variable with spaces between
things.  Use an array.

>             do
>           Within this loop three jobs are submitted 
>         done

Are you running *background jobs*?  Or what?  What do you do inside the
loop?

> Job2. This job should run after the three jobs completed.

Are they background jobs?  Did you capture their PIDs with $! one by
one?  Where did you store the PIDs?  Or do you simply want to call
"wait" to wait for all the background jobs to complete?

> My issue to extract the three PIDs and export them as one variable

EXPORT?!  Why?

> Here is my attempt
>   for tr in ${val}; 

I thought your loop iterator was "a", not "tr".  Why did you change it?
What does "a" mean?  What does "tr" mean?  Why did you choose these
variables?

>          do 
> 
>            job[${#tr}]= some process;

You are creating an array named job and indexing it by the LENGTH of
the string variable tr?  Huh?  What?

What is "some process"?

Your assignment has a god damned SPACE after the = sign so it doesn't
even work.

>                 tt1="${job[${#tr}]}"     #### gets the PIDs for each job
>             echo $tt1
>         done

Can't even bring myself to comment any more.  So tired.

> The echo statement within the for loop  produced the three PID like
> 
> 1009
> 1010
> 1011

"... BY SOME MIRACLE I have an array with three elements."

Let's just assume you got these values somehow.  They are in an array
named "job".  I can't for the life of me figure out HOW you got them,
because your code isn't even code, and even your *fake* code is a
damned disaster, but let's say you have an array.

job=(1009 1010 1011)

> I want this three  PIDs  to be  exported as one variable like, 
> 
> test2=$1009,$1010,$1011

I told you how to do this already.  You use the [*] expansion in
single quotes to join an array into a string variable, using the
first character of IFS.

IFS=,
export test2="${job[*]}"
unset IFS

Are those dollar signs supposed to be part of the string?  Then you
could do a fancypants expansion.

IFS=,
export test2="${job[*]/#/\$}"
unset IFS


WHY are you exporting this crazy string into the environment with a
useless name like "test2" and inscrutable contents?  What are you doing
with it?

If you just want to launch 3 background processes and wait for them all
to finish, you don't need ANY of this crap.

for i in 1 2 3; do
    ./my background job "$i" &

done
wait

That's it!