bug#34525: replace-regexp missing some matches

[Alan Mackenzie]

Hello again, Eli.

On Mon, Feb 18, 2019 at 17:50:16 +0200, Eli Zaretskii wrote:
> > From: Daniel Lopez <daniel.lopez999@gmail.com>
> > Date: Mon, 18 Feb 2019 08:28:35 +0000

> > - Start "emacs -Q" and open the file BitmapFontFace.h
> > - Evaluate the expression (replace-regexp "\\<Bitmap\\>" "SharedBitmap")
> > - The text "Replaced 8 occurrences" appears in the echo area.

> > Problem:

> > There were actually 12 occurrences (ie. of the word "Bitmap" surrounded 
> > by word boundaries) in the file that should have been replaced. If I now 
> > move point back to the start of the buffer and evaluate the expression 
> > again, it says "Replaced 4 occurrences".

> > The exact number of incorrect replacements perhaps varies over time. 
> > That is, I can test it five times in a row and get 8 initial replacments 
> > each time, but after trying some other search terms, messing with the 
> > file, restarting Emacs etc, I try my initial test again and then maybe 
> > it consistently replaces 10 the first time, for a while. So your exact 
> > numbers may vary.

> C++ Mode plays some funky games with the syntax of '<', so maybe this
> is the consequence.  CC'ing Alan, in the hope that he might have
> something interesting to say about this.

I have a lot interesting to say about this.

Firstly, it is a difficult bug.

I see the bug in master and Emacs 26.1 but not on Emacs 25.3.  For the
latter two, I tested with both -Q and --no-desktop in the command line.
The bug would thus appear to be independent of the CC Mode version in
use.

The bug is definitely some sort of interaction between the regexp
"\\<Bitmap\\>" and the opening template delimiter < in code lines such
as:
    
    std::vector< Bitmap<PixelType> > m_bitmaps;
                       ^

.  If a space is inserted before the marked <, the bug doesn't happen.
Neither does it happen if the < is removed altogether.  That < has a
category property whose symbol contains the syntax-table property "(>
" (i.e. "open paren" syntax with matching paren being ">").

I have checked, with a throwaway testing command, that all the <s
following "Bitmap" do indeed have this syntax-table property.  I also
checked the arguments supplied to each call of re-search-forward, and
they are also always correct.

Here is a list of things which DON'T affect the bug:
(i) Building CC Mode so as not to use category text properties.
(ii) Disabling query-replace-lazy-highlight and/or
isearch-lazy-highlight.
(iii) Disabling font-locking.
(iv) Building replace.el and isearch.el without lexical binding.
(v) syntax-propertize-function, which is nil.

However, on instrumenting the two crucial functions replace-search and
isearch-search-fun-default for edebug, the bug was no longer seen.
Setting Edebug's "default mode" to "continue" (with C-x C-a C-m c) to
prevent edebug breaking at the beginning of a function, caused the bug to
be seen again.  So edebug is not of much help for this bug.

I put an invocation of `redisplay' at the top of replace-search.  This
increased the number of matches found from 6, 7, or 8 to about 10 (from a
total of 12).  On adding, additionally, an invocation of
(recenter-top-bottom 0) at the end of replace-search, all 12 matches were
found and replaced.  On removing the `redisplay' call, leaving the
recenter-top-bottom, the number of matches was only 7 or 8.

Thus, if the next occurence of Bitmap is visible on the screen when Emacs
searches for it, it seems to get found.  Sort of...  Most of the time.

Gradually increasing the argument to the above recenter-top-bottom
reduced the number of replaced matches.  With the `redisplay' there, with
(recenter-top-bottom 60), there were just 7 matches.

I think there's some unwanted interaction between redisplay, the syntax
functionality (in particular, the syntax-table text property), and the
regexp machinery.

I don't have any great ideas as how to make further progress here.
Suggestions would be welcome.

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).