bug#50988: 26.3; gnus-cloud gets wrong chunk byte count writing sync from windows

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> Does the file read by that code have DOS-style CRLF end-of-line
> format?

Yes.

> And what does that code have to do with your .gnus.el?  (I don't use
> Gnus, so apologies for asking questions whose answers are trivial.)

The purpose of gnus-cloud is to create IMAP like functionality for NNTP
groups.  Ie. what's read and followed up on on one computer, is marked
as read and followed up on a different computer (I use gnus on 3
different computers and sync state between them).

gnus-cloud also syncs setup in the .gnus.el and the current state of the
score files. The .gnus.el contents is the first chunk in the sync file.

The mechanism of the sync is an IMAP folder with new syncs written as
messages, with the body of the message consisting chunks that have a
lisp-ish header, like this:
(:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" 
:length 16241)
and then the contents of the chunk, until the next header

An example:
 https://gist.github.com/steinarb/155c43fd102c1a929ea441b0703492a3

When the byte count in "length" is wrong, the contents of bytes past the
end, are added to ~/.gnus.el in the computer synced to.

> In general, if that function is supposed to read user-written files
> (as opposed to files Gnus itself writes), then it should not assume
> the EOL format is Unix.  But I'm not sure insert-file-contents is the
> right solution here, because I don't know what will Gnus do with the
> file.

Yes, I'm pretty sure insert-file-contents isn't the right fix either,
but now it sort of works, as opposed to not working at all.  So I'll
limp along on that as a workaround until a real fix is.

> Note that the function reads the file into a unibyte buffer, while
> insert-file-contents performs decoding, and these two don't mix well,
> usually.

> Do you understand why the byte count "should have been 15754"?

Yes.

The .gnus.el file became unparsable on the synched to machine after
sync, because of stuff that was added at the end (which sort of looked
like gnus score files).

So I looked into the contents of the file and used head to return the
first 16241 bytes of the file following the header
(:type :file :file-name "~/.gnus.el" :timestamp "2021-09-19T15:04:33+0200" 
:length 16241)
and saw that this included the stuff that broke the parsing of
.gnus.el:
 https://gist.github.com/steinarb/4ed3a4b716aab22f855df9af387cce78
(search for :type and you'll find the point where .gnus.el parsing
breaks after sync).

At this point I had concluded that the byte count was wrong, and the
only thing I could think about as the cause, was CRLF vs LF.

My .gnus.el has 487 lines and (- 16241 487) is 15754, so I tried using
head to extract the first 15754 bytes of the chunk and ended up with the
correct results:
 https://gist.github.com/steinarb/650839cec8b9961bb614b95be9c026b2

So the evidence is circumstantial but compelling (compelling to me, at
least). 

> If gnus-cloud synchronizes your files with a remote repository, then
> the files on the remote should have the same EOL format as on your
> original disk, so where does Emacs strip or ignore the CR characters
> to come up with a smaller number of bytes in the actual transfer?

I'm not sure where the conversion happens and have been unable to figure
it out.

But I'm guessing the buffer inserted into here, has eol lf-only and that
this is where/when the conversion happens:
 https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/gnus/gnus-cloud.el#n104

FWIW I didn't really want gnus-cloud to sync .gnus.el, since that file
is handled as a config file checked into git.  But excluding it is not a
part of gnus-cloud config (at least not documented).