Re: [Help-bash] bash suitable for parsing big files?

[Matthew Cengia]

On 2013-09-13 04:33, adrelanos wrote:
> Matthew Cengia:
> >>> > > Ultimately, it comes down to "What are you really trying to do?"
> >> > 
> >> > Imagine you are using $linux-distribution on hdd and you want to check
> >> > the integrity of your system. You're booting from USB or DVD and which
> >> > you assume the clean of backdoors while you're not so sure your hdd
> >> > contains a backdoor.
> >> > 
> >> > The script I am writing looks what files are installed, downloads the
> >> > package from $linux-distribution's repositories and compares them with
> >> > the ones on the disk. Finally reports which were modified and which ones
> >> > could not be verified (because they are not in a package, auto generated
> >> > files, etc.). [And more.] I am doing such a thing, just not to verify a
> >> > hdd, but to verify a virtual machine image.
> >> > 
> >> > Code:
> >> > https://github.com/Whonix/Whonix/blob/master/release/verify_build#L187
> >> > 
> >> > Function:
> >> > parse_dpkg_status_file
> >> > 
> > This is what debsums is for: 
> > http://packages.debian.org/search?keywords=debsums
> 
> debsums is fine tool, but not the right tool for this job. It's fine for
> checking integrity, but not for checking malware. Uses weak hash sums
> and its manpage acknowledges "[...] is of limited use as a security
> tool. [...]" It only works within the system and can not in another root
> folder. (Well, you could use chroot, but then you would use untrusted
> binaries.) Debsums lacks also a feature to make a list with files it
> hasn't verified, because they are not owned by any packages.

OK, fine. In that case, looking at your script, it needs lots of work.
It has lots of bad practices in it. Below is a summary of what I can see
at first glance, with things to ask greybot (a bot who lives in
Freenode/#bash; accepts commands starting with '!') about if applicable,
and/or pages with more info:

* Capitalised variable names: !varcaps

* 'cd' without test to confirm success: !cd

* Use "$PWD" instead of "$(pwd)": !pwd

* 'export unset http_proxy' won't do what you want; 'unset http_proxy'
  will

* for i in $(find...) is bad. Use find -exec or a while loop with
  null-separated entries instead: !drlwf !for`

* basename and dirname are unnecessary if you're using Bash. Use
  parameter-expansion instead: !basename !dirname

* Use [[ instead of [: ![[

All this in addition to my previous hints on parsing the
/var/lib/dpkg/status file using either Bash or Awk.

-- 
Regards,
Matthew Cengia

signature.asc
Description: Digital signature