[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] performance issue
From: |
Peter Schuller |
Subject: |
Re: [Duplicity-talk] performance issue |
Date: |
Thu, 31 Jul 2008 19:09:32 +0200 |
User-agent: |
Mutt/1.5.18 (2008-05-17) |
> I've been using duplicity to back up to amazon S3 for about 3 months.
> I'm backing up nearly an entire RHEL5 server, excluding a few things. It
> used to take 70 minutes or so, and suddenly one day it jumped to taking
> 4 or 5 hours. The size of the files has only gone up from 32GB to about
> 36. Is there a way to see which part is taking long (ie. is it the
> amazon interaction taking so long, or is it collecting the information
> on the files, or is it the taring that's taking so long)? Now that I
> look at the output and back in my logs, it looks like it catches a
> socket error 3 times each run. Any other ideas what the problem might be?
Unfortunately there is no direct timing of "local cpu/disk"
vs. "transfer", etc to give an immediate indication of where the
bottleneck is. However, if you are trying to figure it out I sugest
running at -v5 or above and observation the time it takes to upload a
file - also in conjunction with e.g. 'nload' or some other traffic
monitoring tool to show you the bandwidth usage.
In the case of S3 though, I can definitely say that I have had some
inconsistent performance in the past. There was one particular period
of a few weeks where I would get *lots* of socket errors,
nececcitating sending each volume several times. I would also see very
bursty bandwidth utilization. This problem, when I had it, was not
specific to duplicity or the boto S3 library that duplicity uses
(there is much traffic on this from a few months back in the ML
archives).
I think it's pretty likely that the increase in time has to do with
the backend - that is, either your connectivity, an S3 issue, or
something in between. Try timing a backup of something reasonable -
say 10 MB. Extrapolate the time it took to upload the file (just time
it manually when running with -v5) and extrapolate.
How long has the performance been subpar? If only for a few days you
might just try waiting a week ;)
--
/ Peter Schuller
PGP userID: 0xE9758B7D or 'Peter Schuller <address@hidden>'
Key retrieval: Send an E-Mail to address@hidden
E-Mail: address@hidden Web: http://www.scode.org
pgptjeHECFnpu.pgp
Description: PGP signature