Re: [PATCH] binary-io: do not treat set_binary_mode() on stdin/out/err as an error on OS/2

[Bruno Haible]

KO Myung-Hun wrote:
> setmode(O_BINARY) on tty works well instead
> of returning -1 as previous set_binary_mode() does.

"works well", but isn't this the case which produces staircase-shaped
output?

> In addition, set_binary_mode() breaks the compatibility with
> SET_BINARY() before set_binary_mode(). Because SET_BINARY() does nothing
> on tty. It does not cause any error at all. My patch is compatible with
> SET_BINARY().

So you have a problem with the patches by Paul in
<https://lists.gnu.org/archive/html/bug-gnulib/2017-02/msg00058.html>
<https://lists.gnu.org/archive/html/bug-gnulib/2017-02/msg00062.html>
and then in coreutils 
https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=75aababed45d0120d44baa76c5107d0ceb71fc59
https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=1c27b56095b4a82be7d072baabc09262cb4227e5

Paul, what was the problem that motivated these patches?
Was it <https://lists.gnu.org/archive/html/bug-gnulib/2009-11/msg00293.html> ?

There are comments, for example in tr.c, that explain why these programs
want to avoid conversion between LF and CR/LF:

  /* Use binary I/O, since 'tr' is sometimes used to transliterate
     non-printable characters, or characters which are stripped away
     by text-mode reads (like CR and ^Z).  */
  xset_binary_mode (STDIN_FILENO, O_BINARY);
  xset_binary_mode (STDOUT_FILENO, O_BINARY);

Error checking has been added because we want reliable software.

> the 'abort' is the result of xset_binary_mode() using the result of
> set_binary_mode() which affected by __gl_setmode_check().
> 
> No.many programs of coreutils including tee are using
> xset_binary_mode(). Maybe more programs using xset_binary_mode() of
> gnulib. And they will abort as soon as trying to use xset_binary_mode()
> at startup.

OK, then maybe the fix is in gnulib. Your justification sounded like only
the 'tee' program was affected.

You will need to argue what is the desired behaviour, before proposing
a patch.

* What is the default mode of stdin/out/err when it is not a tty?

* Is this default desirable? If not, why not?

* What is the default mode of stdin/out/err when it is a tty?

* Is this default desirable? If not, why not?

* When does staircase-shaped output occur?

* What is the result of isatty(1) 
  a) for console output?
  b) for pipe output?
  c) for regular file output?

* What is the result of isatty(0) 
  a) for console input?
  b) for pipe input?
  c) for regular file input?

* Can setmode fail, other than when the fd argument is invalid (-> EBADF) or
  the mode argument is invalid (-> EINVAL)?

When you have provided the answers to these questions, maybe we'll get closer.

Bruno