Re: [Help-bash] Posix: 2.3 Token Recognition & 2.10 Shell Grammar

[Michael Convey]

As a follow up to this old thread, here's my write-up that summarizes token
recognition and shell grammar.

https://docs.google.com/document/d/14JO-CZBlcv3WQFh1998gGwD_XOz5kyhyAiqKGSJlFJI/edit?usp=sharing

Mike


On Tue, Jul 14, 2015 at 7:48 PM Michael Convey <address@hidden> wrote:

> On Tue, Jul 14, 2015 at 6:01 AM, Eric Blake <address@hidden> wrote:
>
>>
>> Not that I'm aware of.  But I can at least give a layman's shot at
>> trying to explain the intent:
>>
>> The shell allows:
>>
>> case in in in ) echo yes;; esac
>>
>> which means that the tokenizer cannot blindly treat 'in' as a keyword
>> everywhere, but only in the places where the keyword is expected (the
>> third token after seeing 'case' as the first token).  So, reading the
>> grammar, we see (among others):
>>
>>
>> case_clause      : Case WORD linebreak in linebreak case_list    Esac
>>
>> in               : In                       /* Apply rule 6 */
>>
>> %token  In
>> /*      'in'   */
>>
>> 6. [Third word of for and case]
>>
>>    a. [ case only]
>>
>>     When the TOKEN is exactly the reserved word in, the token identifier
>> for in shall result. Otherwise, the token WORD shall be returned.
>>
>>
>> So the parser has seen 'case' as Case, the first 'in' as WORD, and is
>> trying to determine whether the second 'in' fits the rules for
>> "case_clause".  Initially, 'in' is classified as TOKEN, and we are at
>> the rule for the "in" production, which says to use rule 6 to
>> disambiguate the token.  Rule 6 says that the string "in" is recognized
>> as a reserved word at this point of context, so the tokenizer
>> reclassifies from TOKEN to In, and the grammar then accepts the clause
>> as a valid sequence of tokens.  If you do anything else, like:
>>
>> case in \in in ) echo yes;; esac
>>
>> you'll get "bash: syntax error near unexpected token `\in'".  Or,
>> applying the same analysis as above, the "in" production applies Rule 6
>> to the TOKEN of '\in', but since it is not the literal string 'in', it
>> is not recognized as a reserved word, and is not reclassified, and
>> therefore the "case_clause" rule is not satisfied and you have a syntax
>> error.
>
>
> Eric, very helpful, thank you. I'm working on a write-up that summarizes
> the token recognition and shell grammar sections at a slightly higher level
> and in more understandable terms than is provided in the POSIX standard
> ​. Your specific example has helped me better understand the
> interpretation and syntax of shell grammar.
>