Re: How to search on emacs-devel?

[Kyle Meyer]

Eric Abrahamsen writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>
>>> public-inbox is git under the hood (see
>>> https://yhetil.org/emacs-devel/_/text/mirror/).
>>
>> Maybe local indexing with notmuch or what have you is still the easiest
>> solution, to.

For those that want to go this route, the main thing is getting the
Git-stored messages [1] into a format that the indexer understands.
There are some existing tools (not part of public-inbox) that can
convert a public-inbox archive into a Maildir.  l2md [2] and impibe [3]
are the ones listed at
<https://public-inbox.org/public-inbox.git/plain/Documentation/clients.txt>,
though I haven't used either of them myself.  (There's also
public-inbox's lei, which I'll get to below.)

In the case of Notmuch, the idea of making the Maildir-based Notmuch
understand public-inbox archives was mentioned briefly on the Notmuch
mailing list, but that hasn't gone anywhere yet:

  https://yhetil.org/notmuch/20210501051612.GB6409@dcvr/


[1] Documentation on public-inbox's archive format:
    
https://public-inbox.org/public-inbox.git/plain/Documentation/public-inbox-v2-format.pod
[2] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git
[3] https://leahneukirchen.org/dotfiles/bin/impibe

> Oh I see, if you have the public-inbox package installed locally, then
> it can be used to index the git repo and provide a search interface.
> Installing public-inbox on Arch has a dependency on approximately 9,000
> perl packages, so I won't be doing that, but it looks like that would be
> the easiest solution.

I suspect you're referring to public-inbox's lei (local email interface)
that was introduced in the most recent release (1.7.0).  One way to use
it is to wire up some number of archives as "externals".  These can be
remote, local, or some combination.

For example, below adds emacs-devel as a local external.  This clones
the corresponding Git repo and indexes it with public-inbox-index.  It
unsurprisingly takes some time (maybe around 20 minutes on my machine).

  $ mkdir -p inboxes
  $ lei add-external --mirror https://yhetil.org/emacs-devel inboxes/emacs-devel

Later, you can update the Git repo and index new messages with

  $ make -C inboxes/emacs-devel update

And here's an example of adding a remote external:

  $ lei add-external https://yhetil.org/emacs-bugs

The lei-q command is the main interface for searching:

  $ lei q f:eric d:2.days.ago.. | wc -l
  8
  $ lei q f:eric d:2.days.ago.. | head -1
  
[{"blob":"91ec6216f8daa3b731c832f5c5de722e891b49b7","dt":"2022-03-09T22:23:54Z","f":[["Eric
 Abrahamsen" ...

If there are any local externals configured, lei-q won't reach out to
the remote externals unless explicitly requested with --remote:

  $ lei q --remote f:eric d:2.days.ago..  | wc -l
  9

You can also limit a search to only a set of externals:

  $ lei q --only emacs-bugs f:eric d:2.days.ago.. | jq '.[0].s'
  "bug#54324: 29.0.50; Don't error during debbugs-gnu-rescan"

Or specify a one-off remote URL that wasn't added via add-external:

  $ lei q --only https://lore.kernel.org/git userdiff NEAR emacs

All of the above commands show JSON output, but there are several output
formats available, including various mbox flavors and Maildir.

Anyway, I've already written too much, but, for those interested,
https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
along with lei-overview(7) and the other lei- manpages may be worth
checking out.