Re: fifo unlimited buffer size? (possibly tee related)

[Pádraig Brady]

On 12/04/2012 04:26 PM, Peng Yu wrote:
> > I understand the structure, but the concurrent pipelines
> > need separate data sources (process or file copy), or otherwise
> > deadlock may happen as data overflows various buffers.
> > I suppose this could be encapsulated in tee(1) with non-blocking
> > writes and internal buffering, but that would just end up
> > being a data copy anyway, so I'm not sure it's warranted.
>
> So the point to improve shall be tee? If so, I'm glad that we at least
> figure this out.
>
> Let me explain why file copying can be bad. In my pathological example
> based on sort, sort needs to take all the input before generating any
> output. However, there are many applications which just need to see a
> portion (let's call it lookahead) of input before generating any
> output, in which case copying the whole input is a waste, especially
> when the input is very large (say hundres of GB) and lookahead is much
> smaller (say a few MB).
>
> Could the maintainer of tee see if there is anything can be improved?

I notice something similar with:
http://code.dogmap.org/fdtools/multitee/

but it notes issues with non blocking IO,
and it doesn't seem to handle the buffering I mentioned.

So I would say it's a possible enhancement,
though given the awkwardness of implementation,
and more importantly the esoteric nature of
the use case, it would be way down the priority list.

thanks,
Pádraig.