[Duplicity-talk] timeouts on S3 backups

[Russell Clemings]

I'm getting pretty frequent errors on cron'd backups to S3, although
sometimes it succeeds. Running the script from SSH sometimes works,
sometimes doesn't. I can't see a pattern, or even whether the problem
is duplicity, python, boto or S3.

Suggestions?

Here's what I get, followed by the relevant section of httplib.py:

Using temporary directory /s3backuptmp/duplicity-SDBARO-tempdir
Traceback (most recent call last):
 File "/usr/local/bin/duplicity", line 522, in ?
   with_tempdir(main)
 File "/usr/local/bin/duplicity", line 516, in with_tempdir
   fn()
 File "/usr/local/bin/duplicity", line 446, in main
   globals.archive_dir).set_values()
 File "/usr/local/lib/python2.4/site-packages/duplicity/collections.py",
line 476, in set_values
   backend_filename_list = self.backend.list()
 File 
"/usr/local/lib/python2.4/site-packages/duplicity/backends/botobackend.py",
line 193, in list
   for k in self.bucket.list(prefix = self.key_prefix, delimiter = '/'):
 File "/usr/local/lib/python2.4/site-packages/boto/s3/bucketlistresultset.py",
line 31, in bucket_lister
   delimiter=delimiter)
 File "/usr/local/lib/python2.4/site-packages/boto/s3/bucket.py", line
203, in get_all_keys
   body = response.read()
 File "/usr/local/lib/python2.4/httplib.py", line 460, in read
   return self._read_chunked(amt)
 File "/usr/local/lib/python2.4/httplib.py", line 503, in _read_chunked
   value += self._safe_read(chunk_left)
 File "/usr/local/lib/python2.4/httplib.py", line 549, in _safe_read
   chunk = self.fp.read(min(amt, MAXAMOUNT))
 File "/usr/local/lib/python2.4/httplib.py", line 965, in read
   s = self._read()
 File "/usr/local/lib/python2.4/httplib.py", line 941, in _read
   buf = self._ssl.read(self._bufsize)
socket.sslerror: The read operation timed out

-----

My python skills are almost non-existent, but this looks like the
relevant part of httplib.py. I suppose I could just comment out the
part of the "excepts" but that doesn't seem right:

    def _read(self):
        buf = ''
        # put in a loop so that we retry on transient errors
        while True:
            try:
                buf = self._ssl.read(self._bufsize)
            except socket.sslerror, err:
                if (err[0] == socket.SSL_ERROR_WANT_READ
                    or err[0] == socket.SSL_ERROR_WANT_WRITE):
                    continue
                if (err[0] == socket.SSL_ERROR_ZERO_RETURN
                    or err[0] == socket.SSL_ERROR_EOF):
                    break
                raise
            except socket.error, err:
                if err[0] == errno.EINTR:
                    continue
                if err[0] == errno.EBADF:
                    # XXX socket was closed?
                    break
                raise
            else:
                break
        return buf

rac