[Duplicity-talk] scp/sftp backup fails (but not consistently)

[Robin Smidsrød]

Can any of you make some sense of this error message? Everything in the 
script seems ok. I use the same script on several machines. On some 
machines it fails, but on others it works. It is run via anacron.

No signatures found, switching to full backup.
Running 'scp  /tmp/duplicity-YqMZsi-tempdir/mktemp-YI6IFp-557 
address@hidden:dev/root/duplicity-full.2008-09-30T02:00:49+02:00.vol556.difftar.gz' 
failed (attempt #1)
Running 'scp  /tmp/duplicity-YqMZsi-tempdir/mktemp-YI6IFp-557 
address@hidden:dev/root/duplicity-full.2008-09-30T02:00:49+02:00.vol556.difftar.gz' 
failed (attempt #2)
Running 'scp  /tmp/duplicity-YqMZsi-tempdir/mktemp-YI6IFp-557 
address@hidden:dev/root/duplicity-full.2008-09-30T02:00:49+02:00.vol556.difftar.gz' 
failed (attempt #3)
Running 'scp  /tmp/duplicity-YqMZsi-tempdir/mktemp-YI6IFp-557 
address@hidden:dev/root/duplicity-full.2008-09-30T02:00:49+02:00.vol556.difftar.gz' 
failed (attempt #4)
Running 'scp  /tmp/duplicity-YqMZsi-tempdir/mktemp-YI6IFp-557 
address@hidden:dev/root/duplicity-full.2008-09-30T02:00:49+02:00.vol556.difftar.gz' 
failed (attempt #5)
Giving up trying to execute 'scp  
/tmp/duplicity-YqMZsi-tempdir/mktemp-YI6IFp-557 
address@hidden:dev/root/duplicity-full.2008-09-30T02:00:49+02:00.vol556.difftar.gz' 
after 5 attempts
Running 'scp  /tmp/duplicity-YqMZsi-tempdir/mktemp-9STcvQ-558 
address@hidden:dev/root/duplicity-full.2008-09-30T02:00:49+02:00.vol557.difftar.gz' 
failed (attempt #1)
Traceback (most recent call last):
 File "/usr/bin/duplicity", line 522, in ?
   with_tempdir(main)
 File "/usr/bin/duplicity", line 516, in with_tempdir
   fn()
 File "/usr/bin/duplicity", line 507, in main
   full_backup(col_stats)
 File "/usr/bin/duplicity", line 209, in full_backup
   bytes_written = write_multivol("full", tarblock_iter, globals.backend)
 File "/usr/bin/duplicity", line 149, in write_multivol
   bytes_written += waiter()
 File "/usr/lib/python2.4/site-packages/duplicity/asyncscheduler.py", 
line 154, in __run_synchronously
   ret = fn(*params)
 File "/usr/bin/duplicity", line 143, in <lambda>
   async_waiters.append(io_scheduler.schedule_task(lambda tdp, 
dest_filename: put(tdp, dest_filename),
 File "/usr/bin/duplicity", line 97, in put
   backend.put(tdp, dest_filename)
 File 
"/usr/lib/python2.4/site-packages/duplicity/backends/sshbackend.py", 
line 211, in put
   self.run_scp_command(commandline)
 File 
"/usr/lib/python2.4/site-packages/duplicity/backends/sshbackend.py", 
line 154, in run_scp_command
   raise BackendException("Error running '%s'" % commandline)
duplicity.errors.BackendException: Error running 'scp  
/tmp/duplicity-YqMZsi-tempdir/mktemp-YI6IFp-557 
address@hidden:dev/root/duplicity-full.2008-09-30T02:00:49+02:00.vol556.difftar.gz'
Running 'sftp  address@hidden' failed (attempt #1)
Running 'sftp  address@hidden' failed (attempt #2)
Running 'sftp  address@hidden' failed (attempt #3)
Running 'sftp  address@hidden' failed (attempt #4)
Running 'sftp  address@hidden' failed (attempt #5)
Giving up trying to execute 'sftp  address@hidden' after 5 
attempts
Traceback (most recent call last):
 File "/usr/bin/duplicity", line 522, in ?
   with_tempdir(main)
 File "/usr/bin/duplicity", line 516, in with_tempdir
   fn()
 File "/usr/bin/duplicity", line 446, in main
   globals.archive_dir).set_values()
 File "/usr/lib/python2.4/site-packages/duplicity/collections.py", line 
476, in set_values
   backend_filename_list = self.backend.list()
 File 
"/usr/lib/python2.4/site-packages/duplicity/backends/sshbackend.py", 
line 239, in list
   l = self.run_sftp_command(commandline, commands).split('\n')[1:]
 File 
"/usr/lib/python2.4/site-packages/duplicity/backends/sshbackend.py", 
line 203, in run_sftp_command
   raise BackendException("Error running '%s'" % commandline)
duplicity.errors.BackendException: Error running 'sftp  
address@hidden'
run-parts: /etc/cron.daily/backup_db1 exited with return code 1

Before you ask if ssh is setup not to ask for password correctly:

# echo test >test
# scp test address@hidden:dev/test
test                                                   100%    5     
0.0KB/s   00:00
# ssh address@hidden
$ cd dev
$ cat test
test
$

As you can see, no password is asked for.

The script in question is like this:

#!/bin/bash

# Destination user/host
export DST_USER=safe
export DST_HOST=backup-host.example.com
export DST_PATH=dev

# Backup details (boot)
export BK_LABEL=boot
export BK_PATH=/boot
export BK_RET=2W

ssh address@hidden -C "mkdir -p $DST_PATH/$BK_LABEL"
duplicity --no-encryption --exclude-other-filesystems $BK_PATH \ 
scp://address@hidden/$DST_PATH/$BK_LABEL
duplicity remove-older-than $BK_RET --force --no-encryption \ 
scp://address@hidden/$DST_PATH/$BK_LABEL

# Backup details (root)
export BK_LABEL=root
export BK_PATH=/
export BK_RET=2W

ssh address@hidden -C "mkdir -p $DST_PATH/$BK_LABEL"
duplicity --no-encryption --exclude-other-filesystems $BK_PATH \ 
scp://address@hidden/$DST_PATH/$BK_LABEL
duplicity remove-older-than $BK_RET --force --no-encryption \ 
scp://address@hidden/$DST_PATH/$BK_LABEL

I'm using duplicity 0.5.02 on Debian stable, which is compiled with my 
own build script (I call it build_duplicity.sh), see below:

#!/bin/sh

export BUILD_DIR=build_duplicity
export BOTO_VER=1.4c
export PEXPECT_VER=2.3
export DUPLICITY_VER=0.5.02

# Create build dir
mkdir -p $BUILD_DIR
cd $BUILD_DIR

# Install prerequisits from distribution repository
# Add more or less here depending on the age of your distribution
aptitude install python python-dev librsync1 librsync-dev gnupg 
python-gnupginterface ncftp

# Install boto (for Amazon S3 support)
wget "http://boto.googlecode.com/files/boto-$BOTO_VER.tar.gz";
tar xzvf boto-$BOTO_VER.tar.gz
cd boto-$BOTO_VER
python setup.py install
cd ..
rm -r boto-$BOTO_VER
rm boto-$BOTO_VER.tar.gz

# Install pexpect >2.1 (for proper ssh/scp support)
wget "http://pexpect.sourceforge.net/pexpect-$PEXPECT_VER.tar.gz";
tar xzvf pexpect-$PEXPECT_VER.tar.gz
cd pexpect-$PEXPECT_VER
python setup.py install
cd ..
rm -r pexpect-$PEXPECT_VER
rm pexpect-$PEXPECT_VER.tar.gz

# Build duplicity itself
wget 
"http://savannah.nongnu.org/download/duplicity/duplicity-$DUPLICITY_VER.tar.gz";
tar xzvf duplicity-$DUPLICITY_VER.tar.gz
cd duplicity-$DUPLICITY_VER
python setup.py install
cd ..
rm -r duplicity-$DUPLICITY_VER
rm duplicity-$DUPLICITY_VER.tar.gz

# Remove build dir
cd ..
rmdir $BUILD_DIR

-- Robin