Very basic question regarding encoding and `open-network-stream'

[Eric Abrahamsen]

I have an embarrassingly basic question regarding how encoding works
with relationship to `open-network-stream'. I don't have a strong grasp
of encoding issues, particularly regarding processes.

Long story short, I'm trying to get Gnus to internally use decoded group
names as much as possible. I'm investigating whether that means that the
process Gnus uses to talk to remote servers should be encoded differently.

Presently, when Gnus talks to an nntp server, it does so in a process
buffer in which multibyte has been disabled. It wraps the call to
`open-network-stream' in a let which sets `coding-system-for-read/write'
to 'binary. So far so clear.

Gnus also mostly leaves group names as unibyte internally, so reading
group names as bytes works out okay -- but this is what I'm looking at
changing.

The NNTP RFC notes that the default character set for the protocol has
changed from ascii to utf-8.
(https://tools.ietf.org/html/rfc3977#section-1)

What I don't grasp is: if the process buffers are left multibyte, and
the `coding-system-for-read/write' variables are changed to 'utf-8 (or
'undecided?), will this simply do the right thing?

More specifically, is the remote nntp server expected to send along some
information about the encoding it is using for its data? Or is the data
always binary, and we simply know via convention that it can be safely
decoded as 'utf-8?

Or maybe I should just be leaving the process buffer as-is, but doing
the decoding immediately after the `accept-process-output'?

I would very much appreciate it if someone could explain step-by-step,
using small words if possible, how the process encoding is negotiated,
and what might be a reasonable approach to this problem.

Thanks,
Eric