Re: Possible bug with grep/sed/tail?

[Pádraig Brady]

Paolo Bonzini wrote:
>> A) # tail trace.txt | grep "com"
>> - WORKS: produces output
>> B) # tail trace.txt | grep "com" | cat
>> - WORKS: produces output
>> C) # tail -f trace.txt | grep "com"
>> - WORKS: produces output, then waits and reports new lines
>> D) # tail -f trace.txt | grep "com" | cat
>> - FAILS: no output from existing lines, never gets new data
>>
>> To me, it seems completely counterintuitive that A, B, and C would
>> work, but D does not.  Each line of input read by tail should be
>> passed to STDOUT, which is then read as STDIN by grep/sed, then passed
>> to STDOUT and read by cat.  It should not matter if tail is "done"
>> reading the output or not, as clearly that works fine in case C.
> 
> Buffering occurs line by line in cases A and C, in bigger blocks in
> cases B and D.  So the data is stuck in grep (or sed's) buffers until
> enough of it is produced.  If it is never produced, it is stuck unless
> sed/grep see an end-of-file condition on stdin -- which they do with
> tail, but not with tail -f.

This is a common pain in the neck.
I wrote up a detailed description of the issue:
http://www.pixelbeat.org/programming/stdio_buffering/

Note grep has a --line-buffered option which may help you.

In general if you have a tool in the middle of a pipe line
it will need a way to control the line buffering.
There is a patch at the bottom of the page above which
gives that control to cut.

It's probably useful to add this functionality to all coreutil filters.
I still need to be convinced that glibc is not the correct place for this.
If it was there then any users of stdio could be controlled.

Pádraig.