bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks

[Paul Pogonyshev]

Some more thoughts. Why does it even need `echo are you awake'? It's a network connection, it can still fail even if it worked fine 1 ms before when you checked. So, why not just let the first command fail if the connection is dead and restart the connection if it fails in such a way as to suspect that it is dead (i.e. no output)? Maybe limit this to read commands.

A way to let higher-level code avoid certain `file-exists-p' calls: add a dynamic variable that tells TRAMP to skip certain commands if the result is not available from a cache. Something similar to `process-file-side-effects'. Calling code could then do sth. like this:

    (when (let ((tramp-may-skip-if-not-cached `((file-exists-p unknown ,file))))

            (file-exists-p file))  ; TRAMP will return t or nil if it knows or 'unknown if not cached; for local files there is no effect

      ...)

Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED ARGUMENT...). Any element of the list with unknown function name etc. would be simply ignored.

Code that doesn't let-bind this variable will behave as before. Code that cares can be optimized.

Paul

On Sat, 2 Jul 2022 at 17:58, Michael Albinus <michael.albinus@gmx.de> wrote:

Paul Pogonyshev <pogonyshev@gmail.com> writes:

Hi Paul,

> 1) check if connection is alive (`echo are you awake');
> 2) test if the file exists;
> 3) creating a temporary file for the chunk to be inserted; I guess it
> tries until it finds an unused filename, e.g. here it seems to be done
> after `test -e /tmp/tramp.OD3cCu', which doesn't exist;
> 4) 'touch' on the temporary file, presumably to create it;
> 5) 'chmod' on the temporary, presumably so that other users cannot
> read it;
> 6) copying the requested chunk from the full file into the temporary
> (using `dd');
> 7) finding the real name of the temporary with `readlink';
> 8) finding attributes of the temporary with `stat';
> 9) gzipping the temporary for transmition from the remote to the local
> machine;
> 10) testing if the temporary is a directory (WTF?);
> 11) removing the temporary.
>
> I guess it should be obvious that this is a bit too much for one
> `insert-file-contents' call.

In general, I agree. However, some of the commands are caused by
primitive file operations, like file-exists-p. Tramp cannot know what
will be the next call, and it doesn't have all the opportunities to
optimize, compared with the overall picture you see in the eleven steps.

> Suggested improvements:
>
> * TRAMP should issue just one `stat' command to find out most of the
> things about a file: whether it exists, if it is a directory, its real
> name when dereferencing links and whatever stats it is used to find
> now; from `$ stat --help' this seems to be possible. In other words,
> TRAMP shouldn't use simple commands like `test -e': any ping, even
> nominal, will negate any gains from using a tad faster command.
> Instead, if it needs to find anything about a file, it should ask the
> remote about as many things as possible in one go: it is very likely
> that the additional information will be needed soon and even if not,
> this is basically free compared to ping anyway.

Not all remote hosts carry a stat command, and not all existing stat's
are GNU compatible. But yes, if possible, Tramp shall gather as much
information in one run, and cache the results for further use.

I will see what could be done. Will come back with a proposal next days
(note that this will be for Emacs 29, ie git master).

> * TRAMP code should prefer the approach "try do something and handle
> resulting errors" where possible. For example, don't check if the file
> exists, try to read it right away and handle failures properly. Code
> like `(when (file-exists-p ...) do-something)' adds an unnecessary
> command call and creates a racing condition anyway. Also, error-free
> requests should be more frequent, so they should be the main
> optimization goal. I'm not sure if it is applicable to TRAMP itself
> and doesn't come from a higher level, though.

Indeed, this is not Tramp's responsibility. Tramp is a stupid
library. If there is a call for file-exists-p, it must return the
answer. It doesn't know what will be the next request. So I'm rather
pesimistic that Tramp can improve here.

Best regards, Michael.