[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Gzz] Raw pools?
From: |
Benja Fallenstein |
Subject: |
Re: [Gzz] Raw pools? |
Date: |
Sun, 10 Nov 2002 14:39:58 +0100 |
User-agent: |
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020913 Debian/1.1-1 |
Tuomas Lukka wrote:
An idea, related to the "canonical blocks":
Maybe we should label some pools "raw", i.e. no header, just a block of
binary data. That way, we could be compatible with other content-based systems
for externally obtained data.
That means we cannot move blocks from there to non-raw pools. Also, we'd
have to guess the type.
Here's another idea I had lately. It would be nice to have support for
tree hashing at the Storm level-- even though it's unfortunate that the
chunk granularity will be determined by the block creator, this way you
could check the id whenever a new chunk arrives, so you could e.g. show
parts of an image while it is loading (like in a Browser-- you only have
to wait until you have a complete chunk, then you can check the hash and
pass it upward). Also, we could more easily download a file from
multiple sources.
Now, how about we implement this in the following way:
- Have a new id type (02 ids).
- Ids of this type contain a hash of the *header* of the block.
- The header would have a "Storm-Body-Parts" field that would list the
hashes of all chunks in the body, plus the chunks' sizes.
- (We could still store the block in a single file on the file system,
since the header is clearly limited and the body chunks are limited by
their size, so we could still check the hashes correctly when reading
the block in, since we know where one chunk ends and another one starts.)
With this, we would solve four things at once:
- We would only have to read the header, then we could read the body
chunk-by-chunk (to stream or to download from multiple sources).
- We could securely read the header without having to read the body as well.
- If there's an existing file on a p2p network we'd like to refer to, we
can simply create a header for it that has as its sole body chunk that
file. Since body chunks are refered to by an SHA-1 hash, the body would
be the existing file, for the purposes of the p2p network.
- We could even securely distribute the header without violating any
copyright by distributing the body.
(Note that the body chunks will *not* be Storm blocks, even though
they're refered to by an SHA-1 hash. It is therefore illegal to refer to
a body chunk through a Storm URI.)
On the downside, there would be more computation in id checking, since
we'd also need to parse the header.
- Benja
- [Gzz] Raw pools?, Tuomas Lukka, 2002/11/10
- Re: [Gzz] Raw pools?,
Benja Fallenstein <=
- Re: [Gzz] Raw pools?, Tuomas Lukka, 2002/11/10
- Re: [Gzz] Raw pools?, Benja Fallenstein, 2002/11/10
- Re: [Gzz] Raw pools?, Tuomas Lukka, 2002/11/11
- Re: [Gzz] Raw pools?, Benja Fallenstein, 2002/11/11
- Re: [Gzz] Raw pools?, Tuomas Lukka, 2002/11/16
- Re: [Gzz] Raw pools?, Benja Fallenstein, 2002/11/16
- Re: [Gzz] Raw pools?, Tuomas Lukka, 2002/11/16
- Re: [Gzz] Raw pools?, Benja Fallenstein, 2002/11/17
- Re: [Gzz] Raw pools?, Tuomas Lukka, 2002/11/18
- Re: [Gzz] Raw pools?, Benja Fallenstein, 2002/11/18