Re: Choice of block size for file-sharing

[Christian Grothoff]

Hi pukkamustard,

The rationale was/is using a power of two while also staing below the
~63k block size of GNUnet CORE (so we can fit into one message), which
itself arises from the use of a 16-bit integer for message size, which
itself was used (a) to fit into a single IP packet, (b) to fundamentally
avoid problems with attackers sending us individual messages that don't
fit into the heap on smaller devices, (c) to reduce potential impacts
from (bad code failing to properly guard against) integer arithmetic
overflows (as the 16-bit inputs don't overflow so easily after they are
promoted to (usually) 32-bit 'int's), and (d) to limit the impact of
malformed blocks by allowing ECRS to detect bogus blocks at a fine
granularity level. Also, it was a massive increase from the _original_
(2001) design, where we used 1k blocks to fit into unfragmented UDP
packets on Ethernet ;-).

Note that _today_, if I had time to re-implement FS, I would probably
stick to this block size in terms of verification / CORE-transmission,
but would not issue queries for individual blocks. Instead, it would
probably be better to issue a query for a 'stream' of blocks and require
peers to store all blocks for a file together. That would likely result
in a better privacy/availability trade-off than what we have today.


Happy hacking!

Christian

On 5/13/20 4:26 PM, pukkamustard wrote:
> Hello GNUnet,
> 
> I am interested in the reasons for using a block size of 32kB in the
> file-sharing application [1].
> 
> In the paper "Efficient Sharing of Encrypted Data (2002)" [2] arguments
> are made for using a block size of 1kB:
> 
>> One could argue that a size of 1k is far too small. The rationale
>> behind thisfile size is that UDP, the transport mechanism used
>> bygnunet has (on Ethernet)an optimal packet size that is slightly
>> above 1k. Asgnunet must add headers tothe packets, the size
>> approximates this number. Furthermore, many file systemscan be
>> formatted with a block-size of 1k.
> 
> In "An Encoding for Censorship-Resistant Sharing (2003)" the block size
> 32kB is used without giving concrete arguments (or I missed them).
> 
> Is there a rationale for using 32kB? What are the experiences whit using
> this block size? Would you use a different size today?
> 
> I initially posted this question on IRC. Thanks to @tg and @dvn for the
> input and pointing me here.
> 
> -pukkamustard
> 
> 
> [1] https://git.gnunet.org/gnunet.git/tree/src/fs/fs.h#n37
> [2] https://grothoff.org/christian/esed.pdf
> [3] https://grothoff.org/christian/ecrs.pdf
>