Re: [Help-bash] Uniq identical lines not matching

[konsolebox]

On Thu, Feb 18, 2016 at 10:20 AM, Matthew Cengia <address@hidden> wrote:
> On 2016-02-17 23:48, Christopher Maier wrote:
>> Hi,
>>
>> I've run into an odd problem with uniq, which I've simplified to the below 
>> example. When trying to remove duplicates from identical consecutive text 
>> lines where there is a special byte like \xFF, it does not consider most of 
>> the lines to be identical and passes them through. What could be causing 
>> this?
>>
>> address@hidden:~> echo $BASH_VERSION
>> 3.2.51(1)-release
>>
>> address@hidden:~> echo 'aabaaa' | sed 's|b|\xFF|g' | awk '{print $1; print 
>> $1; print $1; print $1; print $1;}' | uniq -c
>>       2 aaÿaaa
>>       1 aaÿaaa
>>       1 aaÿaaa
>>       1 aaÿaaa
>
> Quite sure this is a problem within uniq, not bash:

Confirmed:

# echo 'aabaaa' | sed 's|b|\xFF|g' | awk '{print $1; print $1; print
$1; print $1; print $1;}' | uniq -c
      5 aa�aaa

It could be a problem with uniq itself, or maybe the locale.

-- 
konsolebox