Re: [Duplicity-talk] Delete recent incremental updates, not all?

[Chris Poole]

My cache shows a number of files with the same timestamp,

duplicity-full-signatures
duplicity-full
duplicity-inc
duplicity-new-signatures

All of these are appended with .part. Am I ok to erase these .part
files, leaving the rest, provided I do the same to the encrypted
difftars with the same timestamp on the remote?

As for errors, at first I got this:

######

    Giving up trying to execute 'sftp  -oServerAliveInterval=15
-oServerAliveCountMax=2 address@hidden' after 5 attempts
    Backend error detail: Traceback (most recent call last):
      File "/usr/local/bin/duplicity", line 1245, in <module>
        with_tempdir(main)
      File "/usr/local/bin/duplicity", line 1238, in with_tempdir
        fn()
      File "/usr/local/bin/duplicity", line 1220, in main
        incremental_backup(sig_chain)
      File "/usr/local/bin/duplicity", line 488, in incremental_backup
        globals.backend)
      File "/usr/local/bin/duplicity", line 316, in write_multivol
        (tdp, dest_filename)))
      File "/usr/local/lib/python2.7/site-packages/duplicity/asyncscheduler.py",
line 145, in schedule_task
        return self.__run_synchronously(fn, params)
      File "/usr/local/lib/python2.7/site-packages/duplicity/asyncscheduler.py",
line 171, in __run_synchronously
        ret = fn(*params)
      File "/usr/local/bin/duplicity", line 315, in <lambda>
        async_waiters.append(io_scheduler.schedule_task(lambda tdp,
dest_filename: put(tdp, dest_filename),
      File "/usr/local/bin/duplicity", line 241, in put
        backend.put(tdp, dest_filename)
      File 
"/usr/local/lib/python2.7/site-packages/duplicity/backends/sshbackend.py",
line 207, in put
        self.put_sftp(source_path, remote_filename = remote_filename)
      File 
"/usr/local/lib/python2.7/site-packages/duplicity/backends/sshbackend.py",
line 218, in put_sftp
        self.run_sftp_command(commandline, commands)
      File 
"/usr/local/lib/python2.7/site-packages/duplicity/backends/sshbackend.py",
line 201, in run_sftp_command
        raise BackendException("Error running '%s'" % commandline)
    BackendException: Error running 'sftp  -oServerAliveInterval=15
-oServerAliveCountMax=2 address@hidden'

I tried again, and got this:

    Last inc backup left a partial set, restarting.
    Last full backup date: Fri Dec  3 11:18:38 2010
    Restarting after volume 1, file Documents/Inbox/Video.flv, block 383
    A .
    M .duplicity-log-server
    A Documents/Inbox/Video.flv
    AsyncScheduler: instantiating at concurrency 0
    A Documents/Inbox/Desk_Exercise_Instructions-DB8eVaMgJgs.mp4
    AsyncScheduler: running task synchronously (asynchronicity disabled)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' (attempt #1)
    sftp command: 'put
"/var/folders/ZD/ZDDgBR3jGKOxftJAGo+jM++++TI/-Tmp-/duplicity-EYwZUN-tempdir/mktemp-9r2LzN-2"
"/server/username/home/Backup/Duplicity//duplicity-inc.20101224T223616Z.to.20101224T224905Z.vol2.difftar.gpg"'
    Timeout waiting for response
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' failed (attempt #1)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' (attempt #2)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' failed (attempt #2)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' (attempt #3)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' failed (attempt #3)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' (attempt #4)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' failed (attempt #4)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' (attempt #5)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' failed (attempt #5)
    Giving up trying to execute 'sftp  -oServerAliveInterval=15
-oServerAliveCountMax=2 address@hidden' after 5 attempts
    Backend error detail: Traceback (most recent call last):
      File "/usr/local/bin/duplicity", line 1245, in <module>
        with_tempdir(main)
      File "/usr/local/bin/duplicity", line 1238, in with_tempdir
        fn()
      File "/usr/local/bin/duplicity", line 1220, in main
        incremental_backup(sig_chain)
      File "/usr/local/bin/duplicity", line 488, in incremental_backup
        globals.backend)
      File "/usr/local/bin/duplicity", line 316, in write_multivol
        (tdp, dest_filename)))
      File "/usr/local/lib/python2.7/site-packages/duplicity/asyncscheduler.py",
line 145, in schedule_task
        return self.__run_synchronously(fn, params)
      File "/usr/local/lib/python2.7/site-packages/duplicity/asyncscheduler.py",
line 171, in __run_synchronously
        ret = fn(*params)
      File "/usr/local/bin/duplicity", line 315, in <lambda>
        async_waiters.append(io_scheduler.schedule_task(lambda tdp,
dest_filename: put(tdp, dest_filename),
      File "/usr/local/bin/duplicity", line 241, in put
        backend.put(tdp, dest_filename)
      File 
"/usr/local/lib/python2.7/site-packages/duplicity/backends/sshbackend.py",
line 207, in put
        self.put_sftp(source_path, remote_filename = remote_filename)
      File 
"/usr/local/lib/python2.7/site-packages/duplicity/backends/sshbackend.py",
line 218, in put_sftp
        self.run_sftp_command(commandline, commands)
      File 
"/usr/local/lib/python2.7/site-packages/duplicity/backends/sshbackend.py",
line 201, in run_sftp_command
        raise BackendException("Error running '%s'" % commandline)
    BackendException: Error running 'sftp  -oServerAliveInterval=15
-oServerAliveCountMax=2 address@hidden'

    BackendException: Error running 'sftp  -oServerAliveInterval=15
-oServerAliveCountMax=2 address@hidden'

(Network probably went down again...) I tried it again, and got this:

    Last inc backup left a partial set, restarting.
    Last full backup date: Fri Dec  3 11:18:38 2010
    Traceback (most recent call last):
      File "/usr/local/bin/duplicity", line 1245, in <module>
        with_tempdir(main)
      File "/usr/local/bin/duplicity", line 1238, in with_tempdir
        fn()
      File "/usr/local/bin/duplicity", line 1220, in main
        incremental_backup(sig_chain)
      File "/usr/local/bin/duplicity", line 488, in incremental_backup
        globals.backend)
      File "/usr/local/bin/duplicity", line 255, in write_multivol
        globals.restart.setLastSaved(mf)
      File "/usr/local/bin/duplicity", line 1094, in setLastSaved
        vi = mf.volume_info_dict[self.start_vol]
    KeyError: 1

Once more,

    Last inc backup left a partial set, restarting.
    Last full backup date: Fri Dec  3 11:18:38 2010
    RESTART: The first volume failed to upload before termination.
             Restart is impossible...starting backup from beginning.
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' (attempt #1)
    sftp command: 'cd "/server/username/home/Backup/Duplicity/"'
    sftp command: 'rm
"duplicity-inc.20101224T224905Z.to.20101224T233326Z.manifest.part"'
    Remote file or directory does not exist in command='sftp
-oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden'
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' failed (attempt #1)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' (attempt #2)
    sftp command: 'cd "/server/username/home/Backup/Duplicity/"'
    sftp command: 'rm
"duplicity-inc.20101224T224905Z.to.20101224T233326Z.manifest.part"'
    Remote file or directory does not exist in command='sftp
-oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden'
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' failed (attempt #2)
    Running 'sftp  -oServerAliveInterval=15 -oServerAliveCountMax=2
address@hidden' (attempt #3)

Once more:

    Warning, found signatures but no corresponding backup files
    Warning, found the following orphaned backup file:
    [duplicity-inc.20101224T233326Z.to.20101224T233428Z.manifest.part]
    Last inc backup left a partial set, restarting.
    Last full backup date: Fri Dec  3 11:18:38 2010
    No signatures found, switching to full backup.
    Traceback (most recent call last):
      File "/usr/local/bin/duplicity", line 1245, in <module>
        with_tempdir(main)
      File "/usr/local/bin/duplicity", line 1238, in with_tempdir
        fn()
      File "/usr/local/bin/duplicity", line 1216, in main
        full_backup(col_stats)
      File "/usr/local/bin/duplicity", line 417, in full_backup
        globals.backend)
      File "/usr/local/bin/duplicity", line 255, in write_multivol
        globals.restart.setLastSaved(mf)
      File "/usr/local/bin/duplicity", line 1094, in setLastSaved
        vi = mf.volume_info_dict[self.start_vol]
    KeyError: 1

I stopped it myself if I recall correctly (this was a few days ago),
since I didn't want a full backup (I had run it on the 'automatic'
setting where it usually guesses to do an incremental backup). Just
tried again:

    Traceback (most recent call last):
      File "/usr/local/bin/duplicity", line 1245, in <module>
        with_tempdir(main)
      File "/usr/local/bin/duplicity", line 1238, in with_tempdir
        fn()
      File "/usr/local/bin/duplicity", line 1143, in main
        globals.archive_dir).set_values()
      File "/usr/local/lib/python2.7/site-packages/duplicity/collections.py",
line 682, in set_values
        backup_chains = self.get_sorted_chains(backup_chains)
      File "/usr/local/lib/python2.7/site-packages/duplicity/collections.py",
line 920, in get_sorted_chains
        assert chain_list[1].archive_dir # other is local
    AttributeError: BackupChain instance has no attribute 'archive_dir'

######

TL;DR: I keep blindly trying to do a backup, which chooses
incremental, and then decides on full backup:

    Last inc backup left a partial set, restarting.
    Last full backup date: Fri Dec  3 11:18:38 2010
    RESTART: The first volume failed to upload before termination.
             Restart is impossible...starting backup from beginning.
    ....
    Warning, found signatures but no corresponding backup files
    Warning, found the following orphaned backup file:
    [duplicity-inc.20101224T233326Z.to.20101224T233428Z.manifest.part]
    Last inc backup left a partial set, restarting.
    Last full backup date: Fri Dec  3 11:18:38 2010
    No signatures found, switching to full backup.

Not sure if all this is of any use, but it may be...

(I only had verbosity set to 5.)