Re: Faster “guix search” (was Re: How many bytes do we add (closure of guix) when adding one new package?)

[Csepp]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Tue, 30 May 2023 at 21:10, Csepp <raingloom@riseup.net> wrote:
>
>>              It makes zero sense to load full package definitions from
>> disk for most queries, such as guix search, with an SoA representation
>> we could load only the fields that we care about.
>
> That’s already the case; see
> ~/.config/guix/current/lib/guix/package.cache.
>
> For instance, “guix package -A” exploits it and the performances are
> acceptable.  Two past summers, wow already! I tried to augment it and
> exploit it for “guix search”.  The implementation and benchmark is in
> #39258 [1].  Well, the whole thread of #39258 appears to me worth to
> consider because it spots various bottleneck specific to “guix search”
> and explains why the improvement is not straightforward.

That's a good improvement, but it's in addition to the ELF files, so it
doesn't save any space, and as far as I know it doesn't speed up
non-textual queries, like searching for packages that use a specific
build system.

> Well, I have started months ago to write a Guix extension using
> guile-xapian.  My aim is to tackle two annoyances: 1. the speed and
> 2. the relevance.
>
> About the relevance #2, the issue is that the current scoring considers
> only the local information of one package without considering the global
> information of all the others.  Well, see [2,3,4] for some details. :-)
>
> 1: https://issues.guix.gnu.org/39258#119
> 2: 
> https://yhetil.org/guix/CAJ3okZ3E3bhZ5pROZS68wEKdKOcZ8SpXsvdi-bnB=9Jz3mPahA@mail.gmail.com
> 3: 
> https://yhetil.org/guix/CAJ3okZ3+hn0nJP98OhnZYLWJvhLGpdTUK+jB0hoM5JArQxO=zw@mail.gmail.com
> 4: 
> https://yhetil.org/guix/CAJ3okZ0LaJzWDBA7bjqZew_jAmtt1rj9PJhevwrtBiA_COXENg@mail.gmail.com

Thanks for the links, gonna read them in more detail later.

>> ps.: Now I'm even more glad that I'm using a file system with
>> transparent compression on all my Guix systems.
>
> Did you benchmarked the performances for some Guix operations on these
> compressed vs uncompressed file system?

I haven't, but I have recently tried to move to a larger drive and
accidentally did a btrfs send without compression and the system didn't
fit.