Guix mirror: sourceware discussion report

[zimoun]

Dear,

Currently "guix pull" from Savannah and issues can arise. As we
recently experimented. Tobias and Ricardo recently discussed how to
mirror the repo. IMHO, it is a good idea to mirror but not a good idea
to locate it on Ricardo infrastructure, again. :-)

Well, I propose to see if we can mirror on sourceware.org which are
GNU friends. ;-)

I took the liberty to contact them on IRC. See below.

What I see is: we still "guix pull" from Savannah as usual.
Then if Savannah is down, we catch the error and we try (transparently
for the user) the sourceware url.

As a first contact, they agree on the principle. Even they ask numbers
about traffic etc. :-)


What do you think?


All the best,
simon

--





[Fri Dec 13 2019]
<zimoun> I am working with Guix and we discussed mirroring the GNU Guix repo
         on sourceware.  [17:56]
<zimoun> The best before doing plan is to ask if it should be possible, on the
         principle.  [17:57]
<fche> is this a source code repo or somethign else?
<fche> (sorry, not familiar)
<cgf> Also, what would be the reason for a mirror on sourceware if you already
      have a repo?  [17:58]
<zimoun> To be more concrete, and if you are not familiar with Guix, there is
         command named "guix pull" which basically "git pull". Currently, the
         main repo is on Savannah.
<zimoun> And everything is fine with Savannah. :-)
<zimoun> However, when Savannah is down; which happens, really rarely, but
         happens, then all Guix is kind-of broken.  [17:59]
<fche> so how big is this repo, what form is it, ?  [18:00]
<zimoun> Broken means that the users cannot update their system. And it is not
         related to development but related on normal usage. Well, "guix pull"
         is kind-of equivalent to "apt-get update".
<fche> accessed via git or what?
<fche> we're not set up to host large binaries e.g.  [18:01]
<zimoun> Yes, via Git. And it is only for source.
<cgf> Maybe hold off on this until new sourceware is online?
<zimoun> The mirror of https://git.savannah.gnu.org/cgit/guix.git/
<zimoun> And if it is too big, maybe even only the master branch  [18:02]
<fche> hm, I can't git-clone that bad boy here
<fche> ah never mind starting  [18:03]
<zimoun> What I have in mind, and I speak for myself and not on the behalf of
         the Guix project.
<fche> not that big apparently, some 170MB of content, that's not a problem
<zimoun> sorry, my browser returns me the wrong URL  [18:04]
<zimoun> https://git.savannah.gnu.org/cgit/guix.git/
<zimoun> https://git.savannah.gnu.org/git/guix.git
<zimoun>
<fche> yeah
<fche> do you have a personal sourceware presence already?
<zimoun> Just to be on the same wavelength, currently, each time an user run
         "guix pull" then they "git pull" from
         https://git.savannah.gnu.org/git/guix.git. However, when Savannah is
         down which rarely happens, everybody is annoyed. So I would like that
         "guix pull" catch the error and then try to use a mirror. Which
         means, that sourceware.org/your-convention-guix will be the default
         mirror. So the traffic pull will be very low (when Savannah is
                                                                        [18:11]
<zimoun> down or unreachable) and the traffic push a bit more.
<mjw> guix is awesome! But savannah often is overloaded :{
<zimoun> No I do not have a personnal sourceware presence already.  [18:12]
* fche has no objection
<fche> how do you plan to make the local git repo a fresh enough mirror?
<zimoun> Then, I can ask to Savannah admin to have a hook to automatically
         mirror to sourceware
<fche> or you can run a cron job on sourceware to auto git-pull periodically
                                                                        [18:13]
<zimoun> yes, I have not think too much about the details. Because I would
         like first if it was even possible. :-)
<fche> do you expect the repo to seriously grow over time ?  [18:14]
<zimoun> No. But I can do some stats with Git to answer you more
         precisely. Currently, Guix is around 13000 commits per
         years. However, I do not know how many addition lines it represents
                                                                        [18:16]
<mjw> It guix.git is ~124MB currently. It normally sees multiple commits a
      day, but most of them are fairly small (a couple of lines to a handful
      of files).  [18:18]
<fche> I mean it's not going to grow from 120MB to 1GB or 10G in the
       forseeable future?
<fche> those are the levels at which we might get concerned
<zimoun> Who knows ;-) But I do not think so. And in case, you can say: stop,
         it is too much for our infrastructure.  [18:20]
<zimoun> Well, if we have a first agreement on an hostage (mirror), can I save
         this conversation and post it on the mailing list guix-devel. Collect
         feedback; especially from the maintainers. Then come back to you with
         more concrete details.  [18:21]
<fche> 'forseeable' doesn't ask for infinite prophecy powers :)
<zimoun> :-)
<fche> yeah, I think we can do it.
<zimoun> thank you for your help.
<segher> but how much load on sourceware will this cause?
<zimoun> do you want to be CC of the email?
<mjw> guix is very much pure GNU, and very much sources should produce
      reproducible builds. You can expect it to stay text based forever and
      never see any binary blobs.
<fche> segher, it's a git repo similar in size to valgrind  [18:22]
<zimoun> Almost no load on sourceware. As I tried to explain above. It would
         be a feedback when Savannah is down -- which happens rarely from my
         experience.
<segher> fche: no...  how many pulls per seconds are asked from it  [18:23]
<segher> zimoun: that doesn't change the worst case at all
<zimoun> segher: I have no idea. I think it is hard to count.
<fche> segher, I expect nothing noticeable, but zimoun will let us know if
       there's reason to worry  [18:24]
<fche> ie a few downloads a day is nothing; a hundred git updates an hour is
       nothing
<segher> and a simple gcc pull is almost 20m
<mjw> zimoun, I can also provide a backup mirror. Not as fancy as sourceware,
      but if you need something to fallback on.  [18:25]
<zimoun> segher: yes, it is just a first contact to see if we can agree. I
         totally understand your limitation.
<segher> a fresh clone, i mean
<fche> the gcc git repo is weird/svny and much larger
<segher> yes
<segher> my point is that we already are overloaded  [18:26]
<fche> not with git clones
<segher> the server, in general
<mjw> zimoun, it would go under https://code.wildebeest.org/git/mirror/ (I
      cannot guarantee an SLA though, but I think neither can sourceware)
<fche> and definitely not with git updates
<segher> if it is actually light weight, then who cares ;-)  [18:27]
<cgf> overloaded sourceware is why I suggested waiting until the new server
      comes online.
<segher> right
<cgf> There's also the issue of outdated git.  [18:28]
<segher> it already is super annoying (to GCC) that half of the time (or
         whatever; it feels very often) the problem is caused by cygwin, not
         by us
<cgf> Probably more like 70% of the time.  :-)  [18:29]
<zimoun> mjw: thank you. And thank you for the hard work you did on the gnu
         list moderation.
<segher> cgf: it feels that way, yes
<mjw> zimoun, haha, yeah about that... o boy.
<cgf> I may be wrong but I don't think there are any other projects like
      cygwin on sourceware.  cygwin is a source code repository (lightweight)
      and a binary distribution mechanism (imo, heavy weight)  [18:30]
<cgf> but I digress...
<zimoun> mjw: as you are also on #guix maybe you can suggest the idea of
         wildebeest. Did you subscribe to the guix-devel mailing list?  [18:31]
<segher> cgf: right, and sharing resources with a project with different usage
         patterns of those resources is a bit problematic
<fche> segher, let us know of problems when they appear, and we'll keep
       tweaking things  [18:32]
<fche> there are a lot of knobs to turn
<segher> fche: i can tell you things were bad yesterday  [18:33]
<segher> not too useful, eh :-)
<zimoun> My point coming here and asking is: establish a first contact to see
         if we can go further together. Or if it is already a no go. Because I
         understand you need numbers and details. And I do not have them,
         right now. But I would like to open the discussion on this topic of
         mirroring.
<fche> segher, exactly :)
<mjw> And on Monday gcc will switch to git too!  [18:34]
<fche> zimoun, we can put something like that up in a matter of minutes
<segher> (gcc bugzilla gave ISEs most of the time)
<mjw> well as a mirror and then Dec 31! finally git all the way!
<fche> mjw, that should help our loads, svn is much less efficient than git
<zimoun> fche: thank you for the offer but I am not maintainer and I speak on
         my own voice. I would like to hear the civodul/Ludo's opinion :-)
                                                                        [18:35]
<mjw> civodul did suggest sourceware too, I did tell him to join here. But
      does he listen to me...  [18:36]
<zimoun> mjw: yeah! I know... that's why I am doing ;-)
<zimoun> but you know, french diplomacy :-)  [18:37]