Re: Preservation of Guix report for 2024-01-26

[Timothy Sample]

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Thumbs up on bzip2 support!  We should update Disarchive in Guix but
> perhaps that’s already in your pipeline?

I sent https://issues.guix.gnu.org/68769.  Now I see that I didn’t have
the newest Git hooks installed, so no change ID and no email to the
relevant team.  Sorry!  (I use worktrees so the Makefile didn’t fix this
for me automatically – I should have double checked.)

> We’ll also have to sync the disarchive.guix.gnu.org with ngyro.com.

Hopefully our old system will work again, but I will have to consolidate
my collection of Disarchive specifications, first.

> How did you implement the Subversion check?

The hard way: download, verify Guix hash, compute SWHID, check existence
in SWHID.  (That’s how everything works to date with PoG, but hopefully
the hard way will become obsolete with all the recent support SWH has
been providing us.)

>>   Some of these (I didn’t check them all) are in SWH as content rather
>>   than directories.
>
> Back in the day, they told me that tarballs can sometimes be ingested,
> for instance if they are committed to a VCS repo (that’s why our
> fallback code tries that as well).  Maybe that’s what happened?

Probably, but I don’t quite understand the mechanism.  The “nixguix”
loader uses ExtIDs for deduplication.  My assumption is that it will
only skip unpacking a tarball if there’s an existing ExtID.  It doesn’t
look like there are ExtIDs for these tarballs, so I’m not sure.  (I’ve
been fumbling a bit trying to use the ExtID API, so maybe it’s a mistake
on my end.)

>> The long-term road map is to make it work like an archive.  It will run
>> continuously and store *all* Guix sources.  To make this easy data-wise,
>> it will only store what’s not covered by SWH.
>
> By “it”, you mean the Disarchive DB?

I mean the PoG “project”.  Instead of just testing and reporting, it
will preserve.  For instance, right now if it encounters a tarball that
Disarchive can’t unpack, it shrugs and moves on.  I want it to store
those so that we are guaranteed to be able to revisit it in the future.
Same thing for sources not (yet) in SWH.  I want to store those so that
SWH can ingest them later.  Because we are doing so well working with
SWH, the storage requirements for this will be manageable (10s of
gigabytes).  With that in place, the PoG report will simply explain what
the archive needs to store and why.  Our goal, then, will be for it to
store nothing.


-- Tim