bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks

[Michael Albinus]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

Hi Paul,

>> Combining stat and readlink into one remote command have even slowed
>> down the machinery. Likely, because it isn't needed always, but it
>> always consumes more time to compute both on the remote side, and to
>> read all the results.
>
> Well, that's hard to believe. I mean, it is understandable that
> `readlink' (whatever Elisp call results in that shell command) is not
> often needed, but how long can that take on a modern computer, 1 ms?
> It might be that we evaluate performance changes differently. I care
> more about worst cases (since they affect me directly, yeah) and not
> so much if Tramp becomes 1 ms slower "per user-visible operation" on
> average.

As said, I live in a pretty good connected LAN. remote commands do not
result in heavy performance degradations, compared with the rest of
Elisp computing.

> I tested it a bit now. Initially mentioned usecase
> (`insert-file-contents', but only a file part, not the whole file)
> appears to be considerably faster after the changes (hard to say,
> feels like 30-50% faster).

Oh, good. In my environment it was something about 5%.

> The connection log says that number of
> commands is 10 now, with the originally mentioned list heavily
> changed. However, there must still be space for improvement, e.g. `M-x
> occur tramp-send RET' suggests there are outright duplicated commands
> now:

Pretty sure there's still place for improvements!

> On the other hand, refreshing a remote-tracking Magit (Emacs Git
> interface) buffer doesn't feel any faster than before. Logs also
> suggest that number of commands that tramp send stays as before (43).
> Some commands are certainly excessive, e.g. it repeats `test -d [...]
> 2>/dev/null; echo tramp_exit_status $?' 4 times for the same directory
> (intermixed with other commands), suggesting that Tramp doesn't cache
> the result. Three times it issues `test -r ...' for the same
> directory, the source checkout root. Most other commands cannot be
> generically skipped by Tramp though, this would rather depend on
> Magit.

Caches are problematic. They are limited to 10 seconds by default, did
you try to increase that number? VC related commands are also
problematic, Emacs tests different VC's for availability. It is always a
good idea to reduce the number of VC backends. If you completely trust
in magit, you shall deactivate Emacs' vc.el. Something like (setq
vc-handled-backends nil) should help.

And directories are also problematic for caches. As soon as something
changes there (creation or deletion of a file, for example), the cached
values of the directory must be flushed.

I propose we try step by step. That is, we investigate typical code
snippets, and see what Tramp does. Case by case.

The Tramp debug buffer is a little bit noisy. So you could do

(progn
  (trace-function-background 'tramp-send-command)
  (trace-function-background 'tramp-file-name-handler))

By this, only the calls of tramp-send-command (go remote) and
tramp-file-name-handler (any invocation of Tramp) are traced. The latter
is important to know, what has triggered a given remote command.

Starting with your initial example

(with-temp-buffer
  (insert-file-contents "/ssh:detlef:.emacs" nil 100 nil))

I get the *trace-output* buffer as appended. Let's study this first, and
see, what could be done in order to improve this case. Obviously, most
of the commands are about the temporary file /tmp/tramp.qgnQ01, which is
superfluous. I'll check whether handling temporary files could be
handled better.

When we're happy with this example, let's go to the next example.

> Paul

Best regards, Michael.

PS: Next 2 or 3 days I'm kind of absent: my granddaughter will visit
us. Sorry, but you know: priorities :-)

txtkIKzbFJ2e3.txt
Description: Text document