Re: Enable compgen even when programmable completions are not available?

[Fabien Orjollet]

On 28/06/2023 00:40, Kerin Millar wrote:
> On Tue, 27 Jun 2023 21:52:53 +0200
> of1 <of1@disroot.org> wrote:
>
> > On 27/06/2023 21:05, Kerin Millar wrote:
> > > It doesn't work at all for >=5.2. The reason for this is interesting and I may 
> > > make a separate post about it.
> > >
> > > Prior to 5.2, it can easily be tricked into printing names that do not exist.
> > >
> > > $ VAR=$'\nNONEXISTENT=' ./declare-P | grep ^NONEXISTENT
> > > NONEXISTENT
> > Thank you. I was just reading the discussion in Gentoo forum and
> > realizing that I've been to quickly: it doesn't pass the
> > FOO=$'\nBAR BAZ QUUX=' test. But what about this? (I also quickly
> > tested with Bash 5.2.15).
> >
> >
> > FOO=$'\nBAR BAZ QUUX='
> > VAR=$'\nNONEXISTENT='
> >
> > declare-P() {
> >      local curVar
> >      declare -a curVars
> >
> >      readarray -t curVars <<<"$1"
> >      curVars=( "${curVars[@]%%=*}" )
> >      curVars=( "${curVars[@]##* }" )
> >      for curVar in "${curVars[@]}"; do
> >         ### we can use [[ -v "$curVar" ]] at some point!
> >         [[ "${curVar//[a-zA-Z0-9_]}" || \
> >            "${curVar:0:1}" == [0-9] || \
> >            ! -v "$curVar" || \
> >            ! "$curVar" =~ $2 ]] || printf '%s\n' "$curVar"
> >      done
> > }
> >
> > declare-P "$(declare -p)"
> > echo "##################"
> > declare-P "$(declare -p)" "QU|^NON|VAR"r
>
> This use of test -v is probably sufficient to address that particular issue. It 
> is just as well that the test occurs after determining that the string is a 
> valid identifier because test -v has the power to facilitate arbitrary code 
> execution.
>
> My assertion that your code is broken in 5.2 was incorrect. Rather, the problem 
> seems to be that regcomp(3) behaviour differs across platforms, where given an 
> empty expression to compile. To address that, you could determine whether a 
> second positional parameter was given and is non-empty before proceeding to use 
> it.

Thank you, good to know. I think that ${2:-.} should be correct.

> Curiously, repeated invocation may lead to obvious issues of resource 
> consumption.
>
> # This is excruciatingly expensive
> time for i in {1..100}; do
>         declare-P "$(declare -p)"     
> done
>
> # This is not so expensive (!)
> time for i in {1..100}; do
>         declare-P "$(declare -p)"
>         :
> done

I made some testing by adding
echo -n "$(( ++COUNT ))>$(wc -c <<<"$1") "
echo "$1" >"/tmp/declare-P$COUNT"
to the function:

1>4651 2>9785 3>15283 4>21504 5>29173 6>39739 7>56095 8>84036
9>135145 10>232592 11>422710 12>798174 13>1544326 14>3031854
15>6002134 16>11937918 ^C

ls -rtnh /tmp/declare-P*
-rw-r--r-- 1 1000 1000 4,6K 28 juin  11:09 /tmp/declare-P1
-rw-r--r-- 1 1000 1000 9,6K 28 juin  11:09 /tmp/declare-P2
-rw-r--r-- 1 1000 1000  15K 28 juin  11:09 /tmp/declare-P3
-rw-r--r-- 1 1000 1000  21K 28 juin  11:09 /tmp/declare-P4
-rw-r--r-- 1 1000 1000  29K 28 juin  11:09 /tmp/declare-P5
-rw-r--r-- 1 1000 1000  39K 28 juin  11:09 /tmp/declare-P6
-rw-r--r-- 1 1000 1000  55K 28 juin  11:09 /tmp/declare-P7
-rw-r--r-- 1 1000 1000  83K 28 juin  11:09 /tmp/declare-P8
-rw-r--r-- 1 1000 1000 132K 28 juin  11:09 /tmp/declare-P9
-rw-r--r-- 1 1000 1000 228K 28 juin  11:09 /tmp/declare-P10
-rw-r--r-- 1 1000 1000 413K 28 juin  11:09 /tmp/declare-P11
-rw-r--r-- 1 1000 1000 780K 28 juin  11:09 /tmp/declare-P12
-rw-r--r-- 1 1000 1000 1,5M 28 juin  11:09 /tmp/declare-P13
-rw-r--r-- 1 1000 1000 2,9M 28 juin  11:09 /tmp/declare-P14
-rw-r--r-- 1 1000 1000 5,8M 28 juin  11:09 /tmp/declare-P15
-rw-r--r-- 1 1000 1000  12M 28 juin  11:09 /tmp/declare-P16

But by adding set -p to the subshell:


declare-P() {
   local curVar
   declare -a curVars

   echo -n "$(( ++COUNT ))>$(wc -c <<<"$1") "
   echo "$1" >"/tmp/declare-Pb$COUNT"

   readarray -t curVars <<<"$1"
   curVars=( "${curVars[@]%%=*}" )
   curVars=( "${curVars[@]##* }" )

   for curVar in "${curVars[@]}"; do
      [[ -z "${curVar//[a-zA-Z0-9_]}" && \
         "${curVar:0:1}" != [0-9] && \
         -v "$curVar" && \
         "$curVar" =~ ${2:-.} ]] && printf '%s\n' "$curVar" 1>/dev/null
   done
}

time for i in {1..100}; do
   declare-P "$(set -p; declare -p)"
done

1>4660 2>5010 3>5013 4>5012 5>5012 6>5012 7>5013 8>5013 9>5013 10>5014
11>5014 12>5014 13>5014 14>5015 15>5015 16>5015 17>5015 18>5014 19>5014
20>5014 21>5014 22>5015 23>5015 24>5015 25>5014 26>5015 27>5015 28>5014
29>5015 30>5014 31>5015 32>5014 33>5015 34>5014 35>5015 36>5015 37>5015
38>5015 39>5014 40>5013 41>5015 42>5012 43>5015 44>5015 45>5014 46>5015
47>5015 48>5015 49>5011 50>5015 51>5015 52>5014 53>5013 54>5014 55>5015
56>5015 57>5015 58>5014 59>5015 60>5013 61>5014 62>5014 63>5014 64>5013
65>5015 66>5014 67>5014 68>5015 69>5015 70>5015 71>5015 72>5015 73>5015
74>5015 75>5015 76>5014 77>5014 78>5014 79>5015 80>5015 81>5015 82>5015
83>5015 84>5014 85>5015 86>5014 87>5015 88>5015 89>5015 90>5013 91>5015
92>5014 93>5014 94>5014 95>5014 96>5013 97>5014 98>5015 99>5015 100>5016
real    0m0,854s
user    0m0,669s
sys     0m0,214s

ls -rtnh /tmp/declare-Pb*
-rw-r--r-- 1 1000 1000 4,6K 28 juin  12:07 /tmp/declare-Pb1
-rw-r--r-- 1 1000 1000 4,9K 28 juin  12:07 /tmp/declare-Pb2
[...]
-rw-r--r-- 1 1000 1000 4,9K 28 juin  12:07 /tmp/declare-Pb99
-rw-r--r-- 1 1000 1000 4,9K 28 juin  12:07 /tmp/declare-Pb100

grep -m1 "model name" /proc/cpuinfo
model name      : Intel(R) Core(TM) i7-3770S CPU @ 3.10GHz
uname -srv
Linux 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12)
echo "${BASH_VERSINFO[*]}"
5 1 4 1 release x86_64-pc-linux-gnu


> At present, I do not have a concrete explanation as to why this is, though it 
> may well have something to do with the value of the _ parameter. Suffice to say 
> that I'll be sticking to multiple ${!prefix*} expansions until such time as 
> bash offers a better way of going about it that doesn't also require readline.
A not-homemade declare -P :)