Re: [pre-lexer-2 1/4] command: Factor command name matching out of command.c.

[Ben Pfaff]

John Darrington <address@hidden> writes:

> I've no doubt this patch is an improvement.
> However, I'm worried about how this is going to work with non-ascii encodings.
> For example some recent syntax files that I've seen have UTF-8 "hard" spaces 
> (0xc2 0x0a) instead of the normal ' '.
> I think the lines : 
>  /* Skip whitespace. */
>   ss_ltrim (s, ss_cstr (" \t\n\r\f\v"));
>
> will fail in this case.

Hmm, that's a good point.  It is not an issue for this patch as
such, because the only user in this patch always uses ' ' as
separator, but certainly it will be a problem later in the
series.

I think you might be right that we need a wide-character
[sub]string or at least a UTF-8 [sub]string.

I'll think about this before I proceed.
-- 
Ben Pfaff 
http://benpfaff.org