Re: tokenize honoring quotes

[Koichi Murase]

2022年8月6日(土) 2:44 Robert E. Griffith <bobg@junga.com>:
> Is there an efficient native bash way to tokenize a string into an array
> honoring potentially nested double and single quotes?

I think you may use history expansions to safely cut tokens. For example,

function tokenize {
  eval "tokens=($(
    local str=$1
    while
      history -s "$str" &&
        word=$(history -p '!:0' 2>/dev/null) &&
        [[ $word && $str == *"$word"* ]]
    do
      printf '%q\n' "$word"
      str=${str#*"$word"}
    done
  ))"
}

tokenize 'echo hello $(ls >/dev/tty) >World'; declare -p tokens
tokenize 'echo "hello world"'; declare -p tokens
tokenize '"hello world" "$(date 1>&2)"'; declare -p tokens
tokenize '"hello world" ${a[x]}'; declare -p tokens
tokenize '"hello world" *.txt'; declare -p tokens

---- Result ----

declare -a tokens=([0]="echo" [1]="hello" [2]="\$(ls >/dev/tty)"
[3]=">" [4]="World")
declare -a tokens=([0]="echo" [1]="\"hello world\"")
declare -a tokens=([0]="\"hello world\"" [1]="\"\$(date 1>&2)\"")
declare -a tokens=([0]="\"hello world\"" [1]="\${a[x]}")
declare -a tokens=([0]="\"hello world\"" [1]="*.txt")

--
Koichi