Re: High system load due to bug mbox downloads

[Ricardo Wurmus]

Ricardo Wurmus <address@hidden> writes:

> Hi Glenn,
>
> Ricardo Wurmus <address@hidden> writes:
>
>> The first big mistake here is that using GET gets the full mbox, even if
>> I’m only looking at the headers.  I don’t know if HEAD requests are
>> supported, but they should have been a much cheaper way to request the
>> modification times.  Since I made GET requests the server had to send
>> the full mbox even though I mostly didn’t even read it.  Embarrassing!
>
> Would it be acceptable to perform repeated HEAD requests for the mbox
> file of all active bugs for “guix” and “guix-patches”?  Or is this as
> expensive as GET requests for debbugs.gnu.org?

I looked at my code again and see that I did pass the If-Modified-Since
header.  This means that the vast majority of the requests coming from
issues.guix.gnu.org should have been pretty cheap as no body would need
to be sent, even when using GET requests.

So I wonder how it can be that fetching all messages for a bug via the
SOAP service has not caused any problems, yet making a GET request for
a *changed* mbox per bug number would have a much higher cost.

I would very much like to restore issues.guix.gnu.org for the Guix
project soon, so I would like to know what option would be acceptable
for the maintainers of debbugs.gnu.org.  I would be very happy for rsync
access to the guix and guix-patches mboxes (or even all mboxes) from
issues.guix.gnu.org.  But I’d also be fine with fetching mboxes over the
web.

I just don’t want to this to bring debbugs.gnu.org to its knees, so your
advice is very welcome.

--
Ricardo