Re: [Sks-devel] more database corruption

[Dan Egli]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yaron M. Minsky wrote:
| Two quick thoughts:
|
| 1) if nothing else pans out, it might be time to start thinking about
| the possibility of hardware problems.  I don't know of anyone else who
| has seen routine database corruption anything like what you've seen, or
| really even close.  It makes me think it might be hardware related.
|
| 2) The last thing in your your backup script looks like a run of "sks
| dump".  I'm not sure what directory you're in when you do that (it looks
| like ${TEST}/PTree, but that seems impossible, since "sks dump" should
| simply fail when run from that directory), but if it's run from the main
| database dump directory, then that's definitely a potential source of
| corruption.  As I've mentioned before, running sks dump on a live
| database could corrupt it or do god-knows-what.
|
| y


Thats what I get for typing the script from memoy vs pasting it. I
missed a couple of lines, including a line that changes back to $TEST
before running sks dump.

I'm running a badblock check on the partition that the database resides
on right now. If that comes up empty I'll run a memory test. Beyond that
I don't know what hardware could be bad.

- --- Dan

P.s. the complete script is: (pasted this time, so I know it's accurate)

#!/bin/bash

function errorabort {

~  echo "NON-ZERO exit status! Aborting Keydump! Deleting failed backup!
Alerting SysAdmin"
~  echo "The SKS Keyserver automated database backup and keyring dump
sequence encountered a fatal" > msg
~  echo "error on "`date`". This should be investigated immediately.
Until then, no further automated" >> msg
~  echo "backups or keydumps will take place. A file called BAD_DB was
created in the sks home" >> msg
~  echo "directory. The automated script will not run while this file
exists. When the database" >> msg
~  echo "problem has been corrected, remove this file to re-enable the
automatic backup and dumps." >> msg
~  mail dan -s "SKS Backup routine failure!!" < msg
~  rm -f msg
~  rm -f ${TEST}/PTree/*
~  rm -f ${TEST}/KDB/*
~  touch ${HOME}/BAD_DB
~  exit;
}


PATH=$PATH:/usr/local/bin
# before we do anything, check to see if BAD_DB exists. If so consider
the database unusable. Abort.
if [ -f ${HOME}/BAD_DB ] ; then
~  echo BAD_DB found. Database considered corrupt. Aborting backup/dump.
~  exit 2;
fi


# step 1 - define environment variables
TEST=${HOME}/test_backup
DB=${HOME}/backup_db
NEW=${HOME}/newdump
OLD=${HOME}/olddump
WORK=${HOME}/workdump

# step 2 - backup existing databases

cd $HOME/KDB
# step 2.1 - remove old KBD logs
rm -f `db_archive`
[ $? -ne 0 ] && errorabort
# step 2.2 - copy database files across
cp `db_archive -s` ${TEST}/KDB
[ $? -ne 0 ] && errorabort
# step 2.3 - copy KDB logs across
cp log.* ${TEST}/KDB



# step 2.4 - remove old PTree logs
cd ../PTree
rm -f `db_archive`
# step 2.5 - copy PTree databases across
cp `db_archive -s` ${TEST}/PTree
[ $? -ne 0 ] && errorabort
# step 2.6 - copy PTree logs across
cp log.* ${TEST}/PTree

# step 3 - validate DB files
cd ${TEST}/KDB
for DB in `db_archive -s` ; do
~  db_verify $DB 2>&1 | grep -i VERIFY_BAD >& /dev/null
~  if [ $? -eq 0 ]; then
~    errorabort
~  fi;
done

cd ${TEST}/PTree
for DB in `db_archive -s` ; do
~  db_verify $DB 2>&1 | grep -i VERIFY_BAD >& /dev/null
~  if [ $? -eq 0 ]; then
~    errorabort
~  fi;
done

cd ${TEST}


# step 4 - make keydump

rm -f ${WORK}/*
sks dump 50000 $WORK
[ $? -ne 0 ] && errorabort

cd ${WORK}
for FILE in *.pgp; do
~  mv $FILE dungeon${FILE##sks-dump};
done
cd ..

rm -f ${OLD}/*
mv ${NEW}/* ${OLD}
mv ${WORK}/* ${NEW}

rm -f ${DB}/PTree/*
rm -f ${DB}/KDB/*
mv ${TEST}/KDB/* ${DB}/KDB
mv ${TEST}/PTree/* ${DB}/PTree

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQE/pXodtwT22Jak4/4RAhYDAKCRmxsXX8P6KAcx/kW4aN22m+PgMACgpGyJ
jdoaacy3c1y8Z9YuxjVQwjA=
=6SzG
-----END PGP SIGNATURE-----