bug#63842: 30.0.50; Slow 'gnus-summary-refer-thread'

[Andrew Cohen]

OK, I think I understand the problem.

Before the change that Manuel identified as the culprit of the slowdown,
 thread referral resulted in the creation of a new ephemeral group to
 hold the search results. One of the major features of the changes was
 to try to add the articles from the search to the existing summary
 buffer rather than create a new one (this is an important improvement
 for a variety of reasons; for example, changes made in the additional
 ephemeral buffer would be overriden when exiting the originating
 buffer).

So what Manuel is seeing: previously the new ephemeral group parses only
the headers for the articles in the thread, while in the current
implementation where the newly found articles are simply added to the
existing summary buffer, all the headers in the original buffer are
parsed.

I need to figure out the right way to fix this. I think parsing the full
set of headers is the right thing, but since it is slow that creates a
problem. In the meantime, Manuel can you try setting
'gnus-refer-thread-use-search to t and see if this resolves your
problem? (This should effectively restore the old behavior of creating a
new ephemeral group to hold the search result).

Best,
Andy



>>>>> "MG" == Manuel Giraud <manuel@ledu-giraud.fr> writes:

    MG> Andrew Cohen <cohen@bu.edu> writes:
    >> Sorry, I have gotten busy with other things at the moment.

    MG> Hi Andrew and you don't need to be sorry for this :-)

    >>>>>>> "MG" == Manuel Giraud <manuel@ledu-giraud.fr> writes:
    >> 
    MG> Hi, So here is the crux of this issue.  When using
    MG> 'gnus-summary-refer-thread' in a nnml group, Emacs ends up
    MG> calling 'gnus-get-newsgroup-headers-xover' (via
    MG> 'gnus-fetch-headers').  AFAIU in this function when
    MG> 'gnus-read-all-available-headers' is t, Emacs will parse *all*
    MG> of the " *nntp*" buffer content.  In my case, this buffer is
    MG> quite big (about 50k lines and 23MiB) hence the slowness.
    >> 
    >> Thanks for continuing to debug this. I am confused---why is the
    >> nntp buffer so full?

    MG> I think in a nnml group the nntp buffer is populated with the
    MG> content of the ".overview" file.  In this particular group, I
    MG> have thousands of messages and I think that explains the size of
    MG> this file.

    >> The search routine should populate the buffer only with the
    >> headers of the articles found in the search (I am assuming that
    >> this list of found articles is not 50K lines long).  Maybe the
    >> search is not working properly?

    MG> As we are talking about 'gnus-summary-refer-thread', I guess
    MG> that it is expected that the nntp buffer is filled with this
    MG> content.  A regular query (with 'G G') is still fast, so I think
    MG> my search engine is set up properly.

    >> Can you step through gnus-summary-refer-thread and in the
    >> conditional that retrieves the new headers can you tell me which
    >> branch of the conditional is chosen (there are three
    >> possibilities: 'gnus-request-thread, 'gnus-search-thread, and the
    >> clause with the comment "Otherwise just retrieve some headers").

    MG> In my case, Emacs is using the 'gnus-search-thread' branch and
    MG> ends up calling 'gnus-get-newsgroup-headers-xover' which is the
    MG> function that parses all the nntp buffer content.

    MG> BTW, I also have examples where 'gnus-summary-refer-thread'
    MG> gives me some false positives (eg., not the same thread but part
    MG> of the subject matching)
    >> 
    >> This is probably by design: in the olden days many mailers were
    >> broken and didn't handle the references header properly (I don't
    >> know if this is still the case). So by default gnus tries to use
    >> information from the subject header to help gather loose threads,
    >> which can result in articles not actually part of the thread
    >> being included. You can check if this is the reason for what you
    >> are seeing by setting
    >> 
    >> (setq gnus-summary-thread-gathering-function
    >> 'gnus-gather-threads-by-references)
    >> 
    >> and seeing if this makes a difference.

    MG> Ok, thanks for the explanation and FWIW, my
    MG> 'gnus-summary-thread-gathering-function' is already set to
    MG> 'gnus-gather-threads-by-references.

    MG> Best regards, -- Manuel Giraud

-- 
Andrew Cohen
Director, HKUST Jockey Club Institute for Advanced Study
Lam Woo Foundation Professor and Chair Professor of Physics
The Hong Kong University of Science and Technology