[bug#39258] [PATCH v2 0/3] Xapian for Guix package search

[Ludovic Courtès]

Hello!

Arun Isaac <address@hidden> skribis:

>>> This could be accomplished even with pre-rendering. Xapian provides
>>> "slots" to store arbitrary strings with a document. Instead of storing
>>> the pre-rendered document as a whole, we could store pre-rendered fields
>>> in separate slots. Then, during `guix search` time, we can assemble the
>>> result from these pre-rendered fields.
>>
>> I’m not sure I understand.  The index wouldn’t store pre-rendered
>> strings for every possible terminal width, right?
>
> No, it wouldn't. It would store a partially pre-rendered string, that is
> without fill-paragraph. We run fill-paragraph at `guix search` time to
> complete the rendering.

Note that Texinfo rendering doesn’t use (@ (guix ui) fill-paragraph).
It has its own paragraph-filling code.  We cannot use ‘fill-paragraph’
after Texinfo rendering anyway, since Texinfo knows where things can be
filled and where they cannot—e.g., @example.

>> I think we need to take the whole user experience into account, not
>> just ‘guix search’.  ‘guix pull’ already feels very slow, and it’s a
>> fairly common operation.  Conversely, ‘guix search’ takes roughly
>> between 0.5 and 2 seconds and is an uncommon operation on a “slow
>> path” (in the sense that when you’re searching for software, you’ll
>> probably have to spend more than a couple of seconds to find what
>> you’re looking for.)
>
> I agree we can't compromise too much on `guix pull` performance.
>
>> To me, adding 20–50 seconds on ‘guix pull’ would be undesirable.  :-/
>
> Maybe I'm missing something here. guix pull takes around 40 minutes on
> my machine. In comparison to that, is another 20-50 seconds (roughly 1
> minute) a big deal? How much time would it be acceptable to spend on
> building the Xapian index?

On my laptop, in the best case, when all the substitutes are available
(not uncommon), it takes 2 minutes.  Sometimes, when some substitutes
are missing, it takes 15 minutes.

So of course, the 20–50 seconds matter only in the best case.  But they
matter primarily because that index build may not be substitutable: it’s
possibly unique to each profile (see below).  That means we know we’re
often going to pay for it.

> Also, is it possible to somehow provide substitutes for the Xapian index
> so that the user does not have to actually build it locally during `guix
> pull` time?

We could provide a substitute for users who use only the official 'guix
channel.  However, as soon as users combine multiple channels, they’ll
have to build the index locally.

>> I’m not sufficiently familiar with Xapian’s query language.  The
>> examples I had in mind were:
>> It’s not so much about regexps than it is about selecting individual
>> fields.
>
> I have totally not tested this, but I imagine that equivalent Xapian
> queries might look something like:
>
>>   guix search | recsel -p name -e 'license ~ "LGPL 3"'
>
> guix search license:LGPL3

Nice.

>>   guix search crypto library | \
>>     recsel -e '! (name ~ "^(ghc|perl|python|ruby)")' -p name,synopsis
>
> guix search crypto library AND (NOT ghc) AND (NOT perl) AND (NOT python)
> AND (NOT ruby)

This one is not quite equivalent I guess, but yeah.  :-)

>> What I meant was that we could use (statprof) to see whether/how Texinfo
>> rendering/parsing can be optimized.
>
> Oh, ok. I'll try this if we decide not to pre-render.

It’d be beneficial anyways.

Thank you!

Ludo’.