[bug#33899] [PATCH 0/5] Distributing substitutes over IPFS

[Hector Sanjuan]

Hey! Thanks for reviving this discussion!

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, July 12, 2019 10:15 PM, Ludovic Courtès <address@hidden> wrote:

> Hello!
>
> Pierre Neidhardt address@hidden skribis:
>
> > A little update/recap after many months! :)
>
> Thank you, and apologies for the delay!
>
> > -   Extra metadata: IPFS stores files on UnixFSv1 which does not
> >     include the executable bit.
> >     -   Right now we store a s-exp manifest with a list of files and a
> >         list of executable bits. But maybe we don't have to roll out our 
> > own.
> >
> >     -   UnixFSv1 has some metadata field, but Héctor and Alex did not
> >         recommend using it (not sure why though).
> >
> >     -   We could use UnixFSv2 but it's not released yet and it's unclear 
> > when
> >         it's going to be released. So we can't really count on it right now.
> >
>
> UnixFSv1 is not an option because it lacks the executable bit; UnixFSv2
> would be appropriate, though it stores timestamps that we don’t need
> (not necessarily a problem).
>
> > -   Flat storage vs. tree storage: Right now we are storing the files
> >     separately, but this has some shortcomings, namely we need multiple
> >     "get" requests instead of just one, and that IPFS does
> >     not "know" that those files are related. (We lose the web view of
> >     the tree, etc.) Storing them as tree could be better.
> >     I don't understand if that would work with the "IPLD manifest"
> >     suggested above. Héctor?
> >
>
> I don’t consider the web view a strong argument :-) since we could write
> tools to deal with whatever format we use.
>
> Regarding multiple GET requests: we could pipeline them, and it seems
> more like an implementation detail to me. The real question is whether
> making separate GET requests prevents some optimization in IPFS.
>
> > -   Pinning: Pinning all files separately incurs an overhead. It's
> >     enough to just pin the IPLD object since it propagates recursively.
> >     When adding a tree, then it's no problem since pinning is only done 
> > once.
> >
>
> Where’s the overhead exactly?

There are reasons why we are proposing to create a single DAG with an
IPLD object at the root. Pinning has a big overhead because it
involves locking, reading, parsing, and writing an internal pin-DAG. This
is specially relevant when the pinset is very large.

Doing multiple GET requests also has overhead, like being unable to use
a single bitswap session (which, when downloading something new means a
big overhead since every request will have to find providers).

And it's not just the web view, it's the ability to walk/traverse all
the object related to a given root natively, which allows also to compare
multiple trees and to be more efficient for some things ("pin update"
for example). Your original idea is to create a manifest with
references to different parts. I'm just asking you to
create that manifest in a format where those references are understood
not only by you, the file creator, but by IPFS and any tool that can
read IPLD, by making this a IPLD object (which is just a json).

The process of adding "something" to ipfs is as follows.

----
1. Add to IPFS: multipart upload equivalent to "ipfs add -r":

~/ipfstest $ ipfs add -r -q .
QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP

2. Add manifest as IPLD object. dag/put a json file like:

cat <<EOF | ipfs dag put
{
  "executables": ["ipfstest/1.txt"],
  "root": {
    "/": "QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP"
  }
}
EOF
---

That's it "QmXVgwHR2c8KiPPxaoZAj4M4oNGW1qjZSsxMNE8sLWZWTP" is the root
of your package files.

"bafyreievcw5qoowhepwskcxybochrui65bbtsliuy7r6kyail4w5lyqnjm"
is the root of your manifest file with the list of executables
and a pointer to the other root.


Hector