Re: [Help-bash] Dealing with "Broken pipe: 13" error

[Pierre Gaston]

On Fri, Feb 23, 2018 at 1:06 AM, Peng Yu <address@hidden> wrote:

> Hi,
>
> When I run the following code, it will generate the "Broken pipe: 13"
> error. And the for-loop hangs there at the iteration 1859. How to make
> the awk process persistent so that `read` still can get new data
> beyond iteration 1859?
>
> $ cat ./main.sh
> #!/usr/bin/env bash
> # vim: set noexpandtab tabstop=2:
>
> set -v
> myfifo=$(mktemp -u)
> guard=$(mktemp -u)
> mkfifo "$myfifo" "$guard"
> > "$myfifo" < "$guard" &
> awk -e 'BEGIN { for(i=1;;++i) print i }' > "$myfifo" &
> jobs
> sleep 5
> for i in {1..10000}
> do
>     read -r x < "$myfifo"
>     echo "$x"
>     jobs
> done
>


You are using a guarding process to keep a file descriptor open, but you
keep the writing end opened.
this end doesn't need to be guarded because awk will keep it open anyway.

On the other hand the reading end is not kept opened, read is closing it.
So awk is buffering all it can, then when the first read happens it writes
a full buffer to the pipe, then read close the pipe
and when awk tries to write again it dies with broken pipe.
The guarding process doesn't die because it never writes to the pipe

The loop continues to repoen and read the remaining of the data from the
pipe
(it can do that because you keep the pipe open for writing on the other
end) and block when there is no more data.

opening the  pipe the other way round in the guard process  fixes your
problem

#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

set -v
myfifo=$(mktemp -u)
guard=$(mktemp -u)
mkfifo "$myfifo" "$guard"
< "$myfifo"  < "$guard" &
p=$!
awk  'BEGIN { for(i=1;;++i) print i }' > "$myfifo" &

for i in {1..10000}
do
    read -r x < "$myfifo"
    echo "$x"
done
kill $!
rm "$myfifo" "$guard"

PS: in this case i would rather use another fd, it avoids a process,
reopening the pipe each time
and if you use read -u it probably even avoids calling dup()

#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:

set -v
myfifo=$(mktemp -u)
guard=$(mktemp -u)
mkfifo "$myfifo" "$guard"
awk  'BEGIN { for(i=1;;++i) print i }' > "$myfifo" &
exec 3< "$myfifo"
for i in {1..10000}
do
    read -r -u 3 x
    echo "$x"
done
rm "$myfifo" "$guard"