Re: Optimising cat in pipelines

[Martin Schulte]

Hello Dave,

you wrote:
> So I'll use this technique in low bandwidth pipelines where the extra 
> cat doesn't matter but in high bandwidth cases I'll craft a different 
> path for each combination (it can explode a bit for longer pipelines but 
> should be do-able).

Or you could try to optimize away the cats yourself as in the following script 
(but be aware that you might create other problems with this approach):

#!/bin/bash
# shellcheck disable=SC2209

cat-opted-pipe() {
  cmd=$1
  shift
  while [[ $# -gt 0 ]]; do
    [[ $1 != cat ]] && cmd="$cmd | $1"
    shift
  done
  echo "$cmd"
}

TIMEFORMAT=$'real %3lR\tuser %3lU\tsys %3lS'

cmd1='seq 1 500000000'

cmd2=cat cmd3=cat cmd4=md5sum
echo "$cmd1 | $cmd2 | $cmd3 | $cmd4"
time $cmd1 | $cmd2 | $cmd3 | $cmd4
echo
cat-opted-pipe "$cmd1" "$cmd2" "$cmd3" "$cmd4"
time bash -c "$(cat-opted-pipe "$cmd1" "$cmd2" "$cmd3" "$cmd4")"

echo

cmd2=rev cmd3=head cmd4='tail -n 1'
echo "$cmd1 | $cmd2 | $cmd3 | $cmd4"
time $cmd1 | $cmd2 | $cmd3 | $cmd4
echo
cat-opted-pipe "$cmd1" "$cmd2" "$cmd3" "$cmd4"
time bash -c "$(cat-opted-pipe "$cmd1" "$cmd2" "$cmd3" "$cmd4")"

Best regards,

Martin