tar - `extract dir' removes files, a workaround

[Roland Huebner]

In May I wrote in a mail to address@hidden:
  " `extract dir'  removes all files `newer than' the incremental
  backup"

The problem to be solved:
  I tried to repair the following quite popular damage:
   -  a working dir with a large number of files (e.g. 500 MB full dump)
   -  having done an incremental backup (e.g. 5 - 10 MB) some times ago
   -  continued to work in that wd / creating not a small number
      of new files
   -  removed by a mistake not a small number of the older files, saved
      in the incremental backup
   -  now my hope was, that I could restore the old files by extracting
      the dir, without reconstructing all old filenames one by one.

Paul Eggert said in his very quick answer:
  "... I suppose it might be useful to have yet another option to have
  tar behave the way that you want it to."

Since that time I am using some quick-an-dirty scripts as a workaround
(an example is included below) together with daily backups on CDs and
it works fine.

Not only overwriting files can be omitted but also a very fast restore
procedure is possible, doing a "reverse restore" which avoids
examination of the large full dump in most cases.

To demonstrate the ideas of such a "reverse restore script" a simple
version is shown:

------------------------------------------------------------------------

#! /bin/sh
#
#   Roland Huebner <address@hidden> 
#
#   rev-restore  restore_subdirs Lmax [Lmin]
#
#   A restore script using GNU tar 1.13.17 / 1.13.19
#       restore_subdirs =   a `--files-from=FILE', a list of parent-dirs
#                           which require a restore of files
#                           (searching for `FILE' in BACKUPDIR) or
#                           a single pathname only;
#       Lmax ( 1 ... )  =   the level of the newest incremental backup
#                           which should be used;
#       Lmin (Lmax > Lmin >= 0) =   the level of the oldest incr. backup
#                           which should be used;
#                           or the full backup Lmin=0 (default);
#
#   Adapt: 
#   RESTOREDIR="use the --create working dir for --extract, too"
#   BACKUPDIR="the dir storing all the tar backup / restore stuff:
#              LV.tar, LV.snar and temporary LV.lst, LV.files, LV.reduc"
#   --------------------------------------------------------------------
#
#    *  `reverse restore'  accepts any number of consecutive
#       incremental backups (backup level LV):                  LV.tar
#
#   No `extract dir' will be used; each file will be restored
#   by its own `extract file' to avoid removing of newer file
#   versions.
#
#    *  A SNAPSHOT-Archive must exist for each backup level:    LV.snar 
#       To save a  LV.snar  per level is the additional cost
#       we need with a `reverse restore' compared with an
#       restore which starts with level `0'. 
#
#   Normally the full backup (level 0) creates a SNAPSHOT-File
#   and each new incremental backup updates that only File.
#   The SNAPSHOT-Archives (`reverse restore' needs) document the
#   changes between the last level LV-1 and the actual level LV.
#   Creating a new level LV now takes two steps:
#       cp -p LV-1.snar LV.snar
#
#       tar --create --verbose \
#           --file "${BACKUPDIR}/LV.tar" \
#           --listed-incremental="${BACKUPDIR}/LV.snar" \
#           dir-to-archive 1>"${BACKUPDIR}/LV.out" 2>&1
#
#    *  This script creates reduced list-files of all `new' /
#       `new-dated' Files of `restore_subdirs' which might be 
#       restored:                                               LV.reduc
#
#   The files  LV.reduc  are used as  `--files-from=' with the
#   --extract option.
#
#
#   `Reverse Restore':
#   ------------------
#   1st Step:
#   ---------
#    *  For each backup level  (Lmax <= LV <= Lmin)  the
#       archive members of the `restore_subdirs' (see above
#       `FILE') will be listed in the tar tf-lists:             LV.lst
#   LV.tar, LV.snar and restore_subdirs  -> LV.lst
#
#   `restore_subdirs' only should comprise those parent-dirs which call
#   for a restore of some files. So the following extract will be
#   speeded up.
#   A reverse restore often does not need the full dump level L0. In
#   most cases only newer file versions will be looked up. That's what
#   results in the best saving of time compared with a normal restore
#   starting with L0 and passing through increasing levels LV.
#
#   Each file will be restored with the `newest' version found in the
#   LV-window  (Lmax <= LV <= Lmin). `newest' means: found in the
#   decreasing LV-loop for the first time as `new file' or `new-dated
#   file'.
#   Older changes of a file version are all eliminated from the LV.reduc
#   files. Each file may be restored only once. No overwrite of an
#   already existing file version happens.
#   So, if you have to guess the level where to find the newest version
#   of a file, with a smaller Lmin you will always be on the safe site.
#   But avoid Lmin = 0 as long as possible; the procedure can be much
#   faster when the large full dump does not take part.
#   All files not changed in that window will not be listed in the
#   LV.reduc files and therefor will not be restored.
#   (see  LV.files -> LV.reduc).
#
#   2nd Step:
#   ----------
#    *  Eliminate in the tf-lists per Level LV the first part which 
#       shows per subdir two lines:
#           line 1 = the dir-name
#           line 2 = with `^@' separated file-names and subdir-names
#                    (the contents of the dir-name);
#       The second part of the tf-lists show the pathnames of all `mew'
#       / `new-dated' files with level LV:                      LV.files
#   LV.files may be empty.
#   LV.lst  ->  LV.files
#
#   3rd Step:
#   ---------
#    *  Eliminate in the  LV.files  all lines with pathnames 
#       existing in any higher level (Lmax to LV+1),
#       i.e. all pathnames for which a newer version exist:     LV.reduc
#   LV.files  ->  LV.reduc
#
#   4th Step:
#   ---------
#    *  Change to the working dir of the --create of the incremental
#       backups and start the `reverse restore'
#   LV.tar, LV.snar and LV.reduc  ->  restoring the newest file versions
#
#   An open point (not very weighty):
#   All new made empty sub-dirs will not be restored with that `reverse
#   restore'. Have a look at the first part of the tf-lists.


# RESTOREDIR="use the --create working dir for --extract, too"
RESTOREDIR="/"                                  # adapt it
BACKUPDIR="/work2/CD-BACKUP/rh-suck-tst"        # adapt it

#
#   create tar tf-lists with those (parent-) dirs which should be
#   restored.
#   LV.tar, LV.snar and to_restore  ->  LV.lst
#

RESTORE_SUBDIRS="$1"
if [ -e "${BACKUPDIR}"/"$RESTORE_SUBDIRS" ]
then
    to_rest="${BACKUPDIR}/$RESTORE_SUBDIRS"
    to_restore="--files-from=$to_rest"
else
    to_rest="$RESTORE_SUBDIRS"
    to_restore=$to_rest
fi

Lmax=$2
Lmin=$3
Lmin=${Lmin:-0}
LV=$Lmin

echo "rev-restore: create tf-lists for  $to_rest"
echo "             levels Lmin=$Lmin to Lmax=$Lmax"

while [ $LV -le $Lmax ]
do

    tar --list --file="${BACKUPDIR}/L${LV}.tar" \
        --listed-incremental="${BACKUPDIR}/L${LV}.snar" \
        $to_restore \
        1>"${BACKUPDIR}/L${LV}.lst" 2>&1
        
    let LV+=1

done

#
#   Eliminate first part of the tf-lists  LV.lst;
#   Create the lists of all `mew' / `new-dated' files in each backup
#   level  LV.files :
#   LV.lst  ->  LV.files
#

LV=$Lmax

echo "rev-restore: Create lists of \`mew' / \`new-dated' files 
LV.files"

while [ $LV -ge $Lmin ]         # empty LV.files may be created
do
    FILE0="${BACKUPDIR}/L${LV}.lst"
    FILE1="${BACKUPDIR}/L${LV}.files"
    let LV-=1
    [ -e $FILE0 ] ||  { touch $FILE0 &>/dev/null
                        touch $FILE1 &>/dev/null
                        continue; }
    
    cat -v $FILE0 | sed -e '{ N
        /@/d
        }' > $FILE1
done

#
#   Eliminate all pathnames in  LV.files  which are included in all
#   higher levels (Lmax to LV+1);
#   Create the reduced lists of all `mew' / `new-dated' files in each
#   backup level  LV.reduc :
#   LV.files  ->  LV.reduc
#

LV=$Lmax
FILE1="${BACKUPDIR}/L${LV}.files"
FILE0="${BACKUPDIR}/L${LV}.reduc"
if [ -s $FILE1 ]
then
    cp -p $FILE1 $FILE0
else
    touch $FILE0 &>/dev/null    # leer
fi
let LV-=1

REDUC2="${BACKUPDIR}/reduc2"
touch $REDUC2

echo "rev-restore: Create reduced lists  LV.reduc"

while [ $LV -ge $Lmin ]
do
    FILE1="${BACKUPDIR}/L${LV}.files"
    [ ! -s "$FILE1" ] && { touch "${BACKUPDIR}/L${LV}.reduc"    # empty
                           let LV-=1
                           continue; }
    LV1=$Lmax
    REDUC1="${BACKUPDIR}/L${LV}.reduc"
    cp $FILE1 $REDUC1

    while [ $LV1 -gt $LV ]
    do
        FILE2="${BACKUPDIR}/L${LV1}.files"
        comm -2 -3 $REDUC1 $FILE2 > $REDUC2
        cp $REDUC2 $REDUC1
        let LV1-=1
    done    
    
    cp -p $REDUC2 "${BACKUPDIR}/L${LV}.reduc"
    let LV-=1
done

rm $REDUC2

#
#   Reverse incremental restore:
#   LV.tar, LV.snar and LV.reduc  ->  restoring the newest file version

(cd $RESTOREDIR
echo "restore: PWD=$PWD"
LV=$Lmax
while [ $LV -ge $Lmin ]         # works with empty LV.reduc, too
do

    echo "rev-restore: restore level $LV:"
    tar --extract --verbose \
        --file "${BACKUPDIR}/L${LV}.tar" \
        --listed-incremental="${BACKUPDIR}/L${LV}.snar" \
        --files-from="${BACKUPDIR}/L${LV}.reduc"

    let LV-=1

done
)

#   The files  L*.lst, L*.files and L*.reduc  are specific to 
#   RESTORE_SUBDIRS. They may be removed.
#   rm L*.lst L*.files L*.reduc