Re: [Monotone-devel] Re: Automate stdio chunk size

[Timothy Brownawell]

On Mon, 2006-07-10 at 16:09 +0200, Thomas Keller wrote:
> Thomas Moschny wrote
> > After thinking a while about it, it is no longer clear to me, why there is 
> > a 
> > need for chunked output *at all* ...
> > 
> > The reading side of a pipe can always read the data in arbitrarily (and 
> > independently of the sender) sized packets, even when using synchronous 
> > I/O, 
> > by simply specifying the size in the read() call. The sender must of course 
> > check how many bytes of it's write() call actually got written.
> 
> Well, maybe there is no need for chunked output, but there is definitely 
> the need for some EOF token which tells the client "hey, I got all the 
> data". Ideally this would be paired with the checksum of the just 
> outputted data so the client can ensure that it got all data correctly.

We can't use an in-stream EOF token, because the stream should be
binary-safe. So this means prefixing each data chunk with the size of
that chunk. A chunk is output when it reaches the maximum size (because
having a known maximum size seems convenient), or when the stream is
flushed (my understanding is that this is the Right Thing to do, plus it
could be nice if we have commands that take a long time to finish).

I think we need to keep the chunked output format, but there's no reason
not to increase the maximum size. Just that when we do we "should" bump
the interface version, since the maximum size is a documented part of
the interface. But, I'm not sure how important this is, since I doubt
anyone is relying on that.

There are changes to inventory in the works, that would require changing
the interface version anyway, perhaps we should increase the chunk size
at the same time we land that?

Tim