Re: Moving QEMU downloads to GitLab Releases?

[Michael Roth]

Quoting Stefan Hajnoczi (2021-10-04 04:01:22)
> On Fri, Oct 01, 2021 at 09:39:13AM +0200, Philippe Mathieu-Daudé wrote:
> > On 9/30/21 15:40, Stefan Hajnoczi wrote:
> > > Hi Mike,
> > > QEMU downloads are currently hosted on qemu.org's Apache web server.
> > > Paolo and I were discussing ways to reduce qemu.org network traffic to
> > > save money and eventually turn off the qemu.org server since there is no
> > > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to
> > > GitLab Releases.
> > > 
> > > Since you create and sign QEMU releases I wanted to see what you think
> > > about the idea. GitLab Releases has two ways of creating release assets:
> > > archiving a git tree and attaching arbitrary binaries. The
> > > scripts/make-release script fetches submodules and generates version
> > > files, so it may be necessary to treat QEMU tarballs as arbitrary
> > > binaries instead of simply letting GitLab create git tree archives:
> > > https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries
> > > 
> > > Releases can be uploaded via the GitLab API from your local machine or
> > > deployed as a GitLab CI job. Uploading from your local machine would be
> > > the closest to the current workflow.
> > > 
> > > In the long term we could have a CI job that automatically publishes
> > > QEMU releases when a new qemu.git tag is pushed. The release process
> > > could be fully automated so that manual steps are no longer necessary,
> > > although we'd have to trust GitLab with QEMU GPG signing keys.
> > 
> > Before having to trust a SaaS for GPG signing, could this work?
> > 
> > - make-release script should produce a reproducible tarball in a
> >   gitlab job, along with a file containing the tarball hash.
> > 
> > - Mike (or whoever is responsible of releases) keeps doing a local
> >   manual build
> > 
> > - Mike checks the local hash matches the Gitlab artifact hash
> > 
> > - Mike signs its local build/hash and uses the GitLab API to upload
> >   that .sig to job artifacts
> > 
> > - we can have an extra manual job that checks the tarball.sig
> >   (hash and pubkey) and on success deploys updating the download
> >   page, adding the new release
> 
> I wonder what Mike sees as the way forward.

Hi Stefan, Philippe,

In general I like the idea, since we could also have the CI do the full
gamut of testing against the binaries built from said tarball, so the
Release Person can just regenerate the tarball and provide a sig to
attest that it came from the proper sources. Currently I do make check
and make check-acceptance and a few other sanity checks, which I guess
would no longer be needed then.

But I think the more immediate issue is where/how to host those
tarballs. Even moving all the ROMs/capstone out of the source tree still
results in an xz-compressed tarball size ~25MB, which is well above the
10MB limit mentioned earlier. We could break it out into target-specific
tarballs, maybe further into softmmu/user variants, but that sounds
painful for both users and maintainers who need to deal with the
resulting source tree differences.

What I'm wondering is whether we could just use the archive files
generated by gitlab when we tag our releases? E.g.:

  https://gitlab.com/qemu-project/qemu/-/archive/v6.1.0/qemu-v6.1.0.tar.bz2

If we paired that with an in-tree script similar to make-release for
users to download individual ROM sources/subprojects used for a particular
tagged release, would that be sufficient for GPL compliance and verifying
what sources the binaries were built from? Are there any other
considerations WRT ROMs/etc.?

With something like that in place, Release Person could just do a git
checkout, verify the Maintainer's sig/tag (in case we don't necessarily
trust the git host), generate the tarball, verify the hash matches what
gitlab published (or verify/diff individual files if the bz/gz hashes
require a specific environment), then sign the gitlab tarball and add
the sig to qemu.org download page along with a link the gitlab-generated
tarball.

We could also publish the Maintainer and Release Person public keys on
qemu.org download page so users can verify this as well using the same
process.

Users that want the additional sources can then do it locally via
above-mentioned script, which would be part of the now-signed tarball
and so could be 'trusted' assuming the individual project hosts weren't
compromised (which is still an assumption that's needed with the current
process anyway).

I guess the main question is who is using the ROM/BIOS sources in the
tarballs, and would this 2-step process work for those users? If there
are distros relying on them then maybe there are some more logistics to
consider since the make-release downloads are both time-consuming and
prone to network errors/stalls since it relies on the availability of a
good number of different hosts.

-Mike

> 
> Stefan