[Help-bash] How to debug pipeline that seems to stop?

[Peng Yu]

Hi,

I run many pipelines (using |) running in parallel (using &, but ()
encloses the pipeline code and other setup code). And sometimes (about
1/500 chance), a pipeline seems to stop.

The pipeline is made of wget, gawk, sort, a bash script using while
and read to process the input, and sponge. The seemingly stopped
pipeline has processes with either SN+ or S+ states in ps output. It
is weird why they could have a small chance to have problems. Such a
small chance of failure make it difficult to debug the cause of the
problem.

One way I can think of is to gradually remove the commands from the
pipeline to see when the pipeline will not have any problems. Then the
commands removed (and the ones before) from the pipelines could cause
the problem. But I would like a more effective debugging strategy.

Is there a way to interrogate the processes already running in the
pipeline to see why they don't finish? Or there is any other better
solution?

Also, I once typed Enter in the foreground, and the stop process
(stopped running for a few hours) could keep on running. I am not sure
why such weird behavior could occur and I am not sure how to debug it.
Has anybody seen such a situation and the possible debugging
strategies?

Thanks.

-- 
Regards,
Peng