Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Alex Harsanyi]

On Thu, Jun 21, 2018 at 10:32 PM, Eli Zaretskii <address@hidden> wrote:
>
> Can you step with Edebug through the vc-git.el function where this
> command is invoked, and see if it gets invoked in Emacs 25?

I set `vc-command-messages`, which prints out the subprocesses run by
`vc-do-command` and also updated `vc-do-command` to show the time it took for
each process.  It turns out that Emacs 26.1 does run one extra command:

In Emacs 25.2, a "vc-dir" command runs the following:

Running git --no-pager update-index --refresh . in background... done
0.48 seconds
Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.42 seconds
Running git --no-pager ls-files -z -o --directory --no-empty-directory
--exclude-standard -- . in background... done 0.10 seconds

In Emacs 26.2, a "vc-dir" command runs the following:

Running git --no-pager update-index --refresh . in background... done
0.46 seconds
Running git --no-pager diff-index --relative -z -M HEAD -- . in
background... done 0.46 seconds
Running git --no-pager ls-files -z -c -s -- . in background... done 4.49 seconds
Running git --no-pager ls-files -z -o --directory --no-empty-directory
--exclude-standard -- . in background... done 0.09 seconds

The extra command is "git --no-pager ls-files -z -c -s -- ." and it takes 4.5
seconds to run (I ran each test multiple times and the times are consistent).
However, when I run that command in a terminal, it finishes in less than a
second.

The only thing special about that command is that it produce a lot of output,
so I wrote the following test:

A script to generate lots of output (written in Perl):

    $line = "a" x 1024;
    $nlines = 1024;
    while ($nlines-- > 0) {
        print $line;
    }

Running the perl script in the terminal takes 0.4 seconds if I let it print
all output to the terminal and takes 0.047 seconds if I redirect it to a file:

    $ time perl gen-output.pl
    [... lots of output omitted ...]
    real    0m0.388s
    user    0m0.000s
    sys     0m0.030s

    $ time perl gen-output.pl > x

    real    0m0.047s
    user    0m0.015s
    sys     0m0.015s

An Emacs-Lisp program to run this script, but ignore all output, it takes 16
seconds, in both 25.2 and 26.1:

    ;;;  -*- lexical-binding: t -*-

    (defun time-delta (start end)
      (destructuring-bind (_shigh slow susec spsec) start
        (destructuring-bind (_ehigh elow eusec epsec) end
          (let ((s (+ (* 1.0 slow) (* 1e-6 susec) (* 1e-12 spsec)))
                (e (+ (* 1.0 elow) (* 1e-6 eusec) (* 1e-12 epsec))))
            (- e s)))))

    (defun time-process ()
      (interactive)
      (let* ((start (current-time))
             (buf (get-buffer-create " *AH-TMP*"))
             (proc (start-file-process "AH-TMP" buf "perl" "gen-output.pl")))
        (set-process-sentinel
         proc
         (lambda (p _code)
           (when (eq (process-status p) 'exit)
             (let ((delta (time-delta start (current-time))))
               (message "time-process: it took %.2f seconds" delta)))))
        (set-process-filter proc #'ignore)))

It seems reading output from a process in Emacs is somehow limited to 64 Kb /
second -- I verified this with procmon.  It looks like the behavior is the
same in 25.2 and 26.1, it is just that in vc-dir 25.2 does not run a git
command that produces a lot of output.

>
>> Both versions were downloaded as compiled binaries from the
>> gnu.org website and not modified in any way -- this happens when
>> I start Emacs with "-Q -no-site-file".
>
> Are they both 64-bit builds (or both 32-bit)?

Both are 64 bit builds, `emacs-version` shows:

GNU Emacs 26.1 (build 1, x86_64-w64-mingw32) of 2018-05-30
GNU Emacs 25.2.1 (x86_64-w64-mingw32) of 2017-04-25