[Monotone-devel] Re: [PATCH 1/2] git_export: avoid multiple sql queries

[Felipe Contreras]

On Sun, Mar 8, 2009 at 8:39 PM, Derek Scherger <address@hidden> wrote:
>
>
> On Sat, Mar 7, 2009 at 11:31 PM, Felipe Contreras
> <address@hidden> wrote:
>>
>> This improves performance while exporting. In my system I see an
>> improvement from 52 minutes to 6 seconds.
>
> I do see quite a big performance improvement with this patch as well,
> although not nearly as dramatic as your 52 minutes down to 6 seconds.
> Exporting my monotone database takes ~26 minutes without this patch and ~17
> minutes with it, so there is about 9 minutes of cert loading time which
> certainly needs fixing.

Yeah, I think there must be something fishy in my system, that's why
you were getting better performance all the time, but hopefully with
this patch my system (and other ones) will perform better.

> I've also tested against pidgin, xaraya and OE databases and it shaves about
> 30 minutes off of each of those as well. For pidgin and xaraya this cuts the
> export time almost in half. OE takes *forever* to export though (12 hours or
> so) so cutting 30 minutes off doesn't make such a big difference. I think OE
> has lots more active files and much larger rosters but I haven't really
> looked into why it's so slow.
>
> So, I'm *really* curious as to exactly what you're testing to get this
> speedup. Can you provide some more details?

I'm interested in the Pidgin repo, I've been using my mtn2git script
that is also doing only a few sql queries but unfortunately that
generates different results than git_export. So I made the changes to
do the queries exactly as you do, and *bang*... slow as hell. My bet
is my sqlite (3.5.9).

BTW, my script is in Ruby, so sqlite is being used through the
bindings. You can find it here:
http://github.com/felipec/mtn2git

I still need to push some stuff, like an option to find out missing
authors from the map.

After finding this I decided to profile the 'loading' step, since IMO
it's taking too much time. I used gprof and gprof2dot and it looks
like the biggest offender is get_change->roster_t::get_name taking 60%
of the time. That is after my modifications, which I guess can't be
applied upstream but maybe you would like to take a look?

> I don't have any objections to this patch and I'll probably commit it later
> today, after poking around a bit more to see why in the heck loading certs
> by name is as slow as it is.

Cool, it would be interesting to find out what caused this.

> I did have to make a few adjustments to the patch as I've merged the
> fast-export branch back to net.venge.monotone and a few minor things have
> changed there. patch(1) also didn't like this patch and I had to apply it by
> hand. Is there some know problem with git-format-patch (assuming that's what
> you're using) that produces patches that patch(1) doesn't like?

I don't know... it's a normal patch with some extra info. It looks
like the patches came from mtn revision
'44683b999fa8092a1e7111728cf72e429b0abd0d'.

-- 
Felipe Contreras