bug#34524: wc: word count incorrect when words separated only by no-break space

[Bruno Haible]

[Ccing bug-libunistring, because this is about Unicode handling in GNU. The
 original thread is in <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=34524>.]

> > The man page for wc states: "A word is a... sequence of characters 
> > delimited by white space."
> > 
> > But its concept of white space only seems to include ASCII white space.  
> > U+00A0 NO-BREAK SPACE, for instance, is not recognized.
> > 
> > If your terminal displays UTF-8 encoding:
> > 
> > printf 'how are\xC2\xA0you\n'
> > 
> > or if your terminal displays ISO 8859-1 encoding:
> > 
> > printf 'how are\xA0you\n'
> > 
> > the visible output of this printf is "how are you".  In either case, wc 
> > does not recognize the second space as white space, resulting in an 
> > incorrect word count:

It is a complicated issue.

I) Relax. Don't be religious about it.
II) POSIX char classes
III) User expectations
IV) The Unicode standard
V) Implementation issues


I) Relax. Don't be religious about it.
======================================

Unicode is an effort to make programs work *reasonably well* with as many
kinds of text as possible.

For example, Unicode 23.2
<http://www.unicode.org/versions/Unicode11.0.0/ch23.pdf> page 859
says:
  "The effect of layout controls is specific to particular text processes.
   As much as possible, layout controls are transparent to those text processes
   for which they were not intended."

Or, Unicode TR 29 <https://www.unicode.org/reports/tr29/tr29-33.html> says:
  "The precise determination of text elements may vary according to
   orthographic conventions for a given script or language. The goal of
   matching user perceptions cannot always be met exactly because the text
   alone does not always contain enough information to unambiguously decide
   boundaries. For example, the period (U+002E FULL STOP) is used ambiguously,
   sometimes for end-of-sentence purposes, sometimes for abbreviations, and
   sometimes for numbers. In most cases, however, programmatic text boundaries
   can match user perceptions quite closely, although sometimes the best that
   can be done is not to surprise the user."

Or, there is criticism: <http://jkorpela.fi/unicode/linebr.html>

Therefore, this is a reminder that sometimes no optimal solution can be found.
Relax.


II) POSIX char classes
======================

> There is some discussion of POSIX and unicode classes at:
> http://unicode.org/L2/L2003/03139-posix-classes.htm
> 
> I guess POSIX is defining lower level functionality
> and has to be compat with all uses of iswspace()
> which might be used for line reformatting etc.
> but wc(1) being higher level, perhaps should consider
> the non breaking variants as word separators?

Exactly, that's the right approach. The POSIX char classes are defined in
glibc/localedata/unicode-gen/unicode_utils.py; in this case what matters is
the is_space function, and it has a comment:
    # Don’t make U+00A0 a space. Non-breaking space means that all programs
    # should treat it like a punctuation character, not like a space.
If U+00A0 was made a space, most programs would treat NO-BREAK SPACE like
SPACE, which is against the purpose of NO-BREAK SPACE. So, in general,
users should be aware that NO-BREAK SPACE is not a space. (And likewise,
the SOFT HYPHEN is not to be treated like HYPHEN, because that would be
against the purpose of the SOFT HYPHEN.)

But 'wc' is a specific program, with a specific purpose, and that might
warrant exceptions.

> I pasted `printf '=\u00A0=\u2007=\u202F=\u2060=\n'`
> into libreoffice writer and it treated all but the last
> as a word separator in its word count tool.

This is a good approach, because text processors usually deal with Unicode
in more detail and with more thought than we usually do in the command-line
/ monospaced world.


III) User expectations
======================

On one hand, user expectation that a no-break space separates words is
justified: In "Dr.\u00A0Pinkwart" a user sees two words.

On the other hand, the opposite user expectation is justified as well.
The English sentence "Look: here he is" is translated into French as
"Regarde\u00A0: le voilà". (It is customary to put a space before colon,
question mark, and exclamation mark in French. And to avoid line breaking
at these points, it must be a NO-BREAK space.) When a translator counts
the words they have translated, "Regarde : le voilà" should count as
3 words, not 4 words. OTOH, it could be argued that in this case, the
problem is that a word (":") consisting only of punctuation characters
should not be counted as a word.

But again: relax. Translators are being paid according to word counts,
but a word count that is 1 too high or 1 too low is not dramatic.


IV) The Unicode standard
========================

On one hand, the Unicode standard makes it clear in several places that
  1) NO-BREAK SPACE prohibits line breaking,
  2) line breaking and words are related.

See for example, the Unicode standard section 5.12
<http://www.unicode.org/versions/Unicode11.0.0/ch05.pdf>
page 219:
  "Line breaking algorithms generally use state machines for determining
   word breaks."

Or the Unicode standard section 23.2
<http://www.unicode.org/versions/Unicode11.0.0/ch23.pdf> page 859
  "Word Joiner. U+2060 word joiner behaves like U+00A0 no-break space
   in that it indicates the absence of line breaks; ..."

On the other hand, in the same section 23.2 it says
  "Line breaking and word breaking are distinct text processes.
   Although a candidate position for a line break in text often coincides
   with a candidate position for a word break, there are also many
   situations where candidate break positions of different types do not
   coincide."

And in the Unicode TR 29 section 4 "Word boundaries"
<https://www.unicode.org/reports/tr29/tr29-33.html#Word_Boundaries>
it treats NO-BREAK SPACE as a word boundary by default - this can be
verified through the program below - but also says that SPACE and
NO-BREAK SPACE "may be tailored to be in MidNum, depending on the environment".

Here's an example program, that uses GNU libunistring:
==============================================================
#include <stdio.h>
#include <uniwbrk.h>

int main ()
{
  printf ("%d\n", uc_wordbreak_property (0x00A0));
  {
    uint8_t string[] = "Regarde : le voilà";
    char p[19];
    u8_wordbreaks (string, 19, p);
    puts ((char *) string);
    for (int i = 0; i < 19; i++)
      if (p[i])
        printf ("word break at position %d\n", i);
  }
  {
    uint8_t string[] = "Regarde\u00A0: le voilà";
    char p[20];
    u8_wordbreaks (string, 20, p);
    puts ((char *) string);
    for (int i = 0; i < 20; i++)
      if (p[i])
        printf ("word break at position %d\n", i);
  }
}
==============================================================
and its output:
0                                (means: WBP_OTHER)
Regarde : le voilà
word break at position 7
word break at position 8
word break at position 9
word break at position 10
word break at position 12
word break at position 13
Regarde : le voilà
word break at position 7
word break at position 9
word break at position 10
word break at position 11
word break at position 13
word break at position 14


V) Implementation issues
========================

> The following change would do that:
> 
> diff --git a/src/wc.c b/src/wc.c
> index 179abbe..ca990b4 100644
> --- a/src/wc.c
> +++ b/src/wc.c
> @@ -147,6 +147,13 @@ the following order: newline, word, character, byte, 
> maximum line length.\n\
>    exit (status);
>  }
> 
> +static int _GL_ATTRIBUTE_PURE
> +iswnbspace (wint_t wc)
> +{
> +  return  wc == L'\u00A0' || wc == L'\u2007' \
> +       || wc == L'\u202F' || wc == L'\u2060';
> +}
> +
>  /* FILE is the name of the file (or NULL for standard input)
>     associated with the specified counters.  */
>  static void
> @@ -455,7 +462,7 @@ wc (int fd, char const *file_x, struct fstatus *fstatus, 
> off_t current_pos)
>                            if (width > 0)
>                              linepos += width;
>                          }
> -                      if (iswspace (wide_char))
> +                      if (iswspace (wide_char) || iswnbspace (wide_char))
>                          goto mb_word_separator;
>                        in_word = true;
>                      }
> 
> ...
> For more sophisticated contextual processing we would need
> to use some of the word break functionality from libunistring.

I don't think you will be able to satisfactorily blend POSIX behaviour
with Unicode behaviour without introducing a command-line option.

On the POSIX side: POSIX says
<https://pubs.opengroup.org/onlinepubs/9699919799/utilities/wc.html>
  "The wc utility shall consider a word to be a non-zero-length string
   of characters delimited by white space."
and
<https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap07.html>
  "space
   Define characters to be classified as white-space characters."
So, when operates according to POSIX expectations, it MUST use
  iswspace (wide_char)
not
  iswspace (wide_char) || iswnbspace (wide_char)

On the Unicode side: It is reasonable to see two "words" in
"Regardez\u00A0:", and the GNU libunistring library implement it like
this. It is also reasonable to expect that 'wc' counts words in the Thai
language, which does not use spaces to delimit words. GNU libunistring
may implement this in the future as well.

For this reason, I would find it best to introduce an option '--unicode'
to 'wc', that would produce Unicode compliant results, at the cost of
  - not following POSIX to the letter,
  - being slower.

Bruno