Re: Counting words, fast!

[Jesse Hathaway]

On Tue, Mar 16, 2021 at 10:30 PM Dennis Williamson
<dennistwilliamson@gmail.com> wrote:
> I've been playing with your optimized code changing the read to grab data in 
> chunks like some of the other optimized code does - thus extending your move 
> from by-word to by-line reading to reading a specified larger number of 
> characters.
>
> IFS= read -r -N 4096 var
>
> And appending the result of a regular read to end at a newline. This seemed 
> to cut about 20% off the time. But I get different counts than your code. 
> I've tried using read without specifying a variable and using the resulting 
> $REPLY to preserve whitespace but the counts still didn't match.
>
> In any case this points to larger chunks being more efficient.

Oh! That is a clever idea, I wanted to try reading in larger chunks, but
I wasn't sure how to ensure I had read whole words until you gave
this idea. Using 64K chunks I was able to shave off about 7s in my
testing:

declare -iA words_to_freq
eof='false'
set -o noglob
while [[ "${eof}" == 'false' ]]; do
  if ! LANG='C' IFS='' read -N 65536 -r block; then
    eof='true'
  fi
  if ! IFS='' read -r line; then
    eof='true'
  fi
  for word in ${block@L}${line@L}; do
    words_to_freq["${word}"]+=1
  done
done
set +o noglob