Fwd: read command sometimes misses newline on timeout

[alex xmb sw ratchev]

---------- Forwarded message ----------
From: *Thomas Oettli* <thomas.oettli@sfs.com>
Date: Monday, October 7, 2024
Subject: read command sometimes misses newline on timeout
To: alex xmb sw ratchev <fxmbsw7@gmail.com>


Sorry, English is not my mother tongue, so let me try to clear this.

I have a script that asynchronously reads from a pipe by using read with
timeout (basically the "reader" function in the example). The separator
could be any character, I just assume that it is "\n" in this case. I do
not control the writer to the pipe, lags may occur at any moment.

Now this happens:


   1. the script calls "read" on pipe
   2. "read" hits timeout just after it has read "\n" from the pipe
   3. "read" returns the full line, but exits with rc > 128
   4. the script assumes a partial line, appends received data to a buffer
   and calls "read" again, but without timeout to get the rest of the line
   5. because "\n" was already read at step 2, "read" returns the next line
   6. script appends received data to the buffer to complete the line (due
   to wrong rc from step 3)

The result is two lines in the buffer (without "\n" in between). Do you
understand now what I mean?
------------------------------
*Von:* alex xmb sw ratchev <fxmbsw7@gmail.com>
*Gesendet:* Montag, 7. Oktober 2024 20:02
*An:* Thomas Oettli <thomas.oettli@sfs.com>
*Cc:* bug-bash@gnu.org <bug-bash@gnu.org>; chet.ramey@case.edu <
chet.ramey@case.edu>
*Betreff:* [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not
follow guidance, click links or open attachments unless you recognize the
sender and know the content is safe.
MAIL FROM:  <@fxmbsw7@gmail.com> Report as Spam / Check Mail
<https://mail1.sfs.biz/pyquarantine.php?quarantine=orig&action=report_or_check&mailfrom=fxmbsw7%40gmail.com&id=20241007200237_2002029C296CC>

to know if read returned anything , even if its exit code is not 0 , check
using [[ $reply ]]

i really didnt get ur english or issue

On Monday, October 7, 2024, Thomas Oettli <thomas.oettli@sfs.com> wrote:


I know that it works in that case.
This is just an example that tries to force read into the same situatiation
that I hit in a real world example in which I don't control the input.
------------------------------
*Von:* alex xmb sw ratchev <fxmbsw7@gmail.com>
*Gesendet:* Montag, 7. Oktober 2024 19:17
*An:* Thomas Oettli <thomas.oettli@sfs.com>
*Cc:* bug-bash@gnu.org <bug-bash@gnu.org>; chet.ramey@case.edu <
chet.ramey@case.edu>
*Betreff:* [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not
follow guidance, click links or open attachments unless you recognize the
sender and know the content is safe.
MAIL FROM:  <@fxmbsw7@gmail.com> Report as Spam / Check Mail
<https://mail2.sfs.biz/pyquarantine.php?quarantine=orig&action=report_or_check&mailfrom=fxmbsw7%40gmail.com&id=20241007191742_006EB2A9EAD22>

well try remove the -n in echo

On Monday, October 7, 2024, Thomas Oettli <thomas.oettli@sfs.com> wrote:

Sorry Alex, I don't understand exactly what you mean.
Here is the Test-Script again with some comments:

    function reader() {
      local buf line
      while :; do
        read -t .01 buf                  # try to read line to $buf with timeout
        rc=$?
        if (( rc == 0 )); then           # got a full line or the rest
of a partial line, append $buf to $line
          line+=$buf
        elif (( rc > 128 )); then        # ran into timeout
          line+=$buf                     # maybe received partial
line, append $buf to $line and continue reading
          continue
        fi

        # at this point, the content of $line should always be "TEST"

        [[ $line != TEST ]] && echo Invalid line: $line && exit
        echo OK
        line=""                          # prepare to read next line,
set $line to empty string
      done
    }
    reader < <(
      while :; do
        echo -n TEST
        sleep .00$(($RANDOM%10))
        echo
      done
    )


Could you please explain to me, where exactly the mistake is? I don't know
if you ran the script by yourself, but on all of my machines it exits after
just a few seconds
with the following output:

Invalid line: TESTTEST
------------------------------
*Von:* alex xmb sw ratchev <fxmbsw7@gmail.com>
*Gesendet:* Montag, 7. Oktober 2024 16:41
*An:* Thomas Oettli <thomas.oettli@sfs.com>
*Cc:* bug-bash@gnu.org <bug-bash@gnu.org>; chet.ramey@case.edu <
chet.ramey@case.edu>
*Betreff:* [EXT] Re: read command sometimes misses newline on timeout

CAUTION: This email originated from outside the SFS organization. Do not
follow guidance, click links or open attachments unless you recognize the
sender and know the content is safe.
MAIL FROM:  <@fxmbsw7@gmail.com> Report as Spam / Check Mail
<https://mail1.sfs.biz/pyquarantine.php?quarantine=orig&action=report_or_check&mailfrom=fxmbsw7%40gmail.com&id=20241007164140_2F94A29C1A462>

there is a case , u [[ $readreply ]] after read

On Monday, October 7, 2024, Thomas Oettli via Bug reports for the GNU
Bourne Again SHell <bug-bash@gnu.org> wrote:

I agree with you, but it should never happen that read returns timeout,
also returns the full line and has already read the newline character.
If that happens, there is no way for the script to decide what to do.
Please see the provided test script, it showcases the error.

If I did a mistake there, I would gladly change it. But I currently don't
see any way how to handle this properly from the script side.

Please also see the answer from Martin D Kealey, I think he is on to
something:
https://lists.gnu.org/archive/html/bug-bash/2024-10/msg00007.html




On 10/4/24 8:18 AM, Thomas Oettli via Bug reports for the GNU Bourne Again
SHell wrote:

> Bash Version: 5.2
> Patch Level: 26
> Release Status: release
>
> Description:
> I have tried to write a bash script that asynchronously reads from a pipe
(line by line) with the help of "read -t".
> If the timeout occurs in just the right moment, read returns the full
line, but the return code says timeout (rc > 128).

If the read command times out, it always returns > 128, so if you have a
return code in that range, you can assume read timed out and react
accordingly.

> Therefor it is not possible to know if a full line was returned or not.

When read times out, it always returns what it read before the timeout in
the buffer, so you don't lose any data. Whether or not that's a `full line'
is up to timing, and it's up to the script to decide how to cope with it.

--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU chet@case.edu http://tiswww.cwru.edu/~chet/