bug#45054: 27.1; Can get point into the middle of a run of characters with a replacing display spec

[Zack Weinberg]

I'm implementing a minor mode that "folds" certain strings, causing them
to be displayed as the actual single characters that they are notation
for.  For purposes of this bug report, I will use the running example of
displaying C-style backslash escapes for Unicode characters as their
actual characters, e.g. \u00B6 is displayed as ¶ when the mode is
active.

The mode does this by adding a font-lock keywords rule to the buffer,
that applies display properties that make each string look like it's a
single character.  The documentation says that Emacs will not let point
get into the middle of a run of text that's hidden this way, and that's
mostly true, but I found a way to do it.

Starting from ‘emacs -Q’:

In the *scratch* buffer, evaluate the following minor mode definition;
this is a cut-down version of the real mode, that only replaces \u00B6.

(define-minor-mode fold-demo-mode
  "Folding bug demo"
  :init-value nil
  :lighter " FD"
  :keymap nil
  (make-variable-buffer-local 'font-lock-extra-managed-props)
  (cl-flet ((refontify ()
             (save-excursion
               (goto-char (point-min))
               (while (re-search-forward "\\\\u00[bB]6" nil t)
                 (save-excursion
                   (font-lock-fontify-region
                    (match-beginning 0) (match-end 0)))))))
    (let ((case-fold-search nil)
          (demo-fl-rule '(("\\\\u00[bB]6" 0
                           '(face font-lock-string-face
                                  display "¶")))))
      (if fold-demo-mode
          (progn
            (push 'display font-lock-extra-managed-props)
            (font-lock-add-keywords nil demo-fl-rule)
            (refontify))
        (font-lock-remove-keywords nil demo-fl-rule)
        (refontify)
        (setq font-lock-extra-managed-props
              (cl-remove 'display font-lock-extra-managed-props
                         :count 1))))))

Then, in any convenient buffer, on an initially blank line, type the
following.  I’ve written the recipe out as it might appear in a
keyboard-macro editing buffer, but instead of command names, the
annotations show what you see at each stage.  The *visible cursor* is
indicated with a |; all other characters are real.

\                    ;; \|
u00B6                ;; \u00B6|
C-a                  ;; |\u00B6
M-x fold-demo-mode   ;; |¶
C-d                  ;; |u00B6
\                    ;; ¶|
x                    ;; \x|u00B6

The bug happens when you type \ for the second time.  The font-lock rule
matches again, “\u00B6” is converted to “¶”, and the *visible cursor* is
displaced to after the “¶”, but *point* is still where it was, between
the invisible \ and u characters.  Any self-inserting character, like
the ‘x’ shown in the recipe, will be inserted at the actual location of
point, instead of the position of the visible cursor.  In the demo, this
causes the font-lock rule to stop matching again.

Any cursor-motion command while Emacs is in this state, will put point
back in sync with the visible cursor; this can lead to odd but harmless
phenomena, for instance a single C-f doesn’t appear to do anything,
instead of moving the cursor to the next line like it normally would.
(Because point moves within the hidden run of characters, and then gets
displaced to the end of the run, where the visible cursor already is.)

Questions:

1. Is this indeed a bug in Emacs?
2. Assuming it is, can you suggest a workaround?  I’m hoping for a
   viable solution within my code that’s compatible at least as far back
   as Emacs 24.

3. I think it would be better UI if the C-d in the recipe deleted the
   entire run of text that’s getting hidden (more generally, deletion
   commands should behave as if the ¶ is really what’s in the buffer).
   I can’t figure out how to implement that, can you suggest anything?

4. Is there a better way to do this sort of visual replacement of runs
   of text?  This is the first time I’ve done any serious elisp
   programming, I might have missed something.

Thanks for your attention,
zw

In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.23,
cairo version 1.16.0) of 2020-11-07, modified by Debian built on
x86-ubc-01 Windowing system distributor 'The X.Org Foundation', version
11.0.12009000 System Description: Debian GNU/Linux bullseye/sid

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
 
--enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --build
 x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
 --with-pop=yes
 
--enable-locallisppath=/etc/emacs:/usr/local/share/emacs/27.1/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/27.1/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --with-cairo
 --with-x=yes --with-x-toolkit=gtk3 --with-toolkit-scroll-bars
 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs-6jKC2B/emacs-27.1+1=.
-fstack-protector-strong -Wformat -Werror=format-security -Wall'
'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD
JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Features:
(shadow sort mail-extr emacsbug message rmc puny rfc822 mml mml-sec epa
derived epg epg-config gnus-util rmail rmail-loaddefs
text-property-search mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils help-mode server visual-regexp-steroids
visual-regexp tramp tramp-loaddefs trampver tramp-integration files-x
tramp-compat parse-time iso8601 time-date ls-lisp dired dired-loaddefs
dbus xml solarized-light-theme solarized color dash typo paren grep
cus-start cus-load erlang-start ess-site ess-toolbar ess-mouse mouseme
ess-swv ess-noweb ess-noweb-font-lock-mode ess-jags-d ess-bugs-l
essd-els ess-xls-d ess-vst-d ess-stata-mode ess-stata-lang cc-vars
cc-defs make-regexp ess-sp6w-d ess-sp5-d ess-sp4-d ess-sas-d ess-sas-l
ess-sas-a ess-s4-d ess-s3-d ess-omg-d ess-omg-l ess-arc-d ess-lsp-l
ess-sp6-d ess-dde ess-sp3-d ess-julia julia-mode cl ess-r-mode
ess-r-flymake flymake-proc flymake warnings thingatpt ess-r-xref xref
project ess-trns ess-r-package shell pcomplete ess-r-syntax
ess-r-completion ess-roxy ess-rd essddr noutline outline easy-mmode
hideshow ess-s-lang ess-help info ess-mode ess ess-noweb-mode ess-inf
ess-tracebug advice format-spec ess-generics compile ess-utils ido
ess-custom executable comint ansi-color ring go-mode-autoloads package
easymenu browse-url url-handlers url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
cairo move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)