bug#20703: BUG 20703 further evidence

[Dmitry Gutov]

Hi Sam,

On 01/13/2016 08:54 PM, Sam Halliday wrote:

> I have been seeing a problem that is described in this bug report
>
>    https://debbugs.gnu.org/db/20/20703.html
>
> I have applied the suggested patch to etags-tags-completion-table (copied below 
> in completeness for your convenience) and trapped an error case.

You should try the current version in emacs-25, it's smaller and faster 
than previously, although it also probably fails at long-enough lines.

> I'm triggering the error in an extremely long line of code (46,000 characters!). I 
> presume somebody programmatically generated the line and pasted it into the source. A 
> workaround could be to simply filter such lines at the ctag building or loading stage, 
> just something that deletes "long" lines, whatever that may mean. Probably 500 
> characters is long enough!
>
> I could also look at adding maximum sizes to my regexes in ctags, but that 
> really isn't a general solution because many ctags patterns do not have such 
> limits.

I can think of some other possible solutions:

- External pre-processor that removes lines that are too long.

- Extra step, together with a custom variable, in visit-tags-table, that 
goes through the opened files and does the same.

- re-search-forward with limit, as implemented in the patch below 
(against emacs-25), that might work against problematic files like that 
(I haven't tested it).

I don't really know if we should install it, though, because it adds a 
performance overhead of ~10%. And I don't know if this problem is common 
enough.

Because another way to combat it is at the source: through judicious 
application of --exclude argument. As a bonus, the generation phase will 
become faster as well (sometimes dramatically).

Should we add a validation phase to visit-tags-table instead? Like, one 
that would say "your TAGS files contains obviously malformed entries 
from file XXX.min.js, go back and ignore it"?

diff --git a/lisp/progmodes/etags.el b/lisp/progmodes/etags.el
index 2db7220..9a663d4 100644
--- a/lisp/progmodes/etags.el
+++ b/lisp/progmodes/etags.el
@@ -1252,8 +1252,9 @@ etags-file-of-tag
          str
        (expand-file-name str (file-truename default-directory))))))

+(defvar etags--table-line-limit 500)

-(defun etags-tags-completion-table () ; Doc string?
+(defun etags-tags-completion-table ()   ; Doc string?
   (let (table
        (progress-reporter
         (make-progress-reporter
@@ -1263,10 +1264,13 @@ etags-tags-completion-table
       (goto-char (point-min))
       ;; This regexp matches an explicit tag name or the place where
       ;; it would start.
-      (while (re-search-forward
-              "[\f\t\n\r()=,; ]?\177\\\(?:\\([^\n\001]+\\)\001\\)?"
-             nil t)
-       (push   (prog1 (if (match-beginning 1)
+      (while (not (eobp))
+        (if (not (re-search-forward
+                  "[\f\t\n\r()=,; ]?\177\\\(?:\\([^\n\001]+\\)\001\\)?"
+                  ;; Avoid lines that are too long (bug#20703).
+                  (+ (point) etags--table-line-limit) t))
+            (forward-line 1)
+          (push (prog1 (if (match-beginning 1)
                           ;; There is an explicit tag name.
                           (buffer-substring (match-beginning 1) (match-end 1))
                         ;; No explicit tag name.  Backtrack a little,
@@ -1277,7 +1281,7 @@ etags-tags-completion-table
                              (buffer-substring (point) 
(match-beginning 0))
                            (goto-char (match-end 0))))
                  (progress-reporter-update progress-reporter (point)))
-               table)))
+               table))))
     table))

 (defun etags-snarf-tag (&optional use-explicit) ; Doc string?