[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] A duplicity backend for Amazon S3
From: |
Brian Sutherland |
Subject: |
Re: [Duplicity-talk] A duplicity backend for Amazon S3 |
Date: |
Mon, 1 May 2006 04:42:31 +0200 |
User-agent: |
Mutt/1.5.9i |
On Sun, Apr 30, 2006 at 06:45:27PM -0500, Ben Escoto wrote:
> >>>>> Brian Sutherland <address@hidden>
> >>>>> wrote the following on Thu, 13 Apr 2006 20:28:38 +0200
> > Hi,
> >
> > I've written a backend for duplicity that writes to the Amazon S3
> > service [1]. It uses the bitbucket.py module [2].
> >
> > It allows you to do something like this:
> >
> > duplicity / s3+http://${access_key}:${secret_key}/${bucket}
> >
> > and so on to store you backups on S3. Which, well is very cheap. If
> > others start using this, it would be nice for the code to be included in
> > duplicity.
> >
> > (Beware, bitbucket.py is a fairly young module and may have some bugs)
> >
> > [1] http://www.amazon.com/gp/browse.html/103-1203077-2066220?node=16427261
> > [2] http://www.other10percent.com/?p=18
>
> Thanks, this sounds like a great idea, and maybe even something I
> might use once my current remote-storage contract expires. I couldn't
> download the bitbucket module though at the link you provided.
Perhaps here is better:
http://cheeseshop.python.org/pypi/BitBucket
> I added the code in your message and changed the web page to mentioned
> the Amazon S3 link.
After testing the code for some time, I found it didn't work as well as
I hoped. It would fail to write to the connection often. Also the
BitBucket module changed it's interface in a backwards incompatible way.
Below is a new bitbucket backend [1] that I am running the first test on
right now (It takes a while). It will try to re-connect and re-try an
operation if one fails. Also included is a patch [2] to fix a bug in
bitbucket 0.3b when running without a config file.
> I'd like to add an S3 example to the
> documentation, but don't know what ${access_key} et al look like. If
> you give me an S3 URL with some realistic looking values I'll add it
> to the web and man pages.
Here is the example from the amazon web pages:
access key: 44CF9590006BF252F707
secret key: OtxrzxIsfpFjA7SwPzILwy8Bw21TLhquhboDYROV
resulting url:
s3+http://44CF9590006BF252F707:OtxrzxIsfpFjA7SwPzILwy8Bw21TLhquhboDYROV/bucket_name
A big problem with putting the keys in the url seems to be that the keys
can contain almost any character, my secret key contains a + and /. I am
not sure what other characters can appear or what to do about it.
[1]
class BitBucketBackend(Backend):
def __init__(self, parsed_url):
import bitbucket
self.module = bitbucket
parts = parsed_url.suffix.split('/')
self.bucket_name = parts[-1]
self.access_key, self.secret_key =
'/'.join(parts[:-1]).split(':')
self._connect()
self.debug = False
def _connect(self):
self.connection =
self.module.connect(access_key=self.access_key,
secret_key=self.secret_key)
self.bucket = self.connection.get_bucket(self.bucket_name)
def put(self, source_path, remote_filename = None):
"""Transfer source_path (Path object) to remote_filename
(string)
If remote_filename is None, get the filename from the last
path component of pathname.
"""
if not remote_filename:
remote_filename = source_path.get_filename()
bits = self.module.Bits(filename=source_path.name)
if self.debug:
print 'Putting %s' % source_path.name
try:
self.bucket[remote_filename] = bits
except:
self._connect()
self.bucket[remote_filename] = bits
def get(self, remote_filename, local_path):
"""Retrieve remote_filename and place in local_path"""
local_path.setdata()
bits = self.bucket[remote_filename]
if self.debug:
print 'Getting %s' % local_path.name
try:
bits.to_file(local_path.name)
except:
self._connect()
bits.to_file(local_path.name)
local_path.setdata()
def list(self):
"""Return list of filenames (strings) present in backend"""
try:
self.bucket.fetch_all_keys()
keys = self.bucket.keys()
except:
self._connect()
self.bucket.fetch_all_keys()
keys = self.bucket.keys()
if self.debug:
print 'Getting Keys: %s' % keys
return keys
def delete(self, filename_list):
"""Delete each filename in filename_list, in order if
possible"""
for file in filename_list:
if self.debug:
print 'Deleting: %s' % file
try:
del self.bucket[file]
except:
self._connect()
del self.bucket[file]
[2]
--- src/BitBucket-0.3b/bitbucket.py 2006-04-23 16:02:33.000000000 +0200
+++ pylib/bitbucket.py 2006-05-01 03:07:21.000000000 +0200
@@ -558,11 +558,15 @@
# SecretAccessKey: YourSecretAccessKeyHere
# Debug: 1
#-------------------------------------------------------
-BB_DEFAULTS = {'AccessKeyID': '',
- 'SecretAccessKey': '',
- 'BucketNamePrefix': '',
- 'PageSize': 100,
- 'Debug': 1}
+_BB_DEFAULTS = {'AccessKeyID': '',
+ 'SecretAccessKey': '',
+ 'BucketNamePrefix': '',
+ 'PageSize': '100',
+ 'Debug': '1'}
+
+BB_DEFAULTS = {}
+for i in _BB_DEFAULTS:
+ BB_DEFAULTS[i.lower()] = _BB_DEFAULTS[i]
#
# Just wanted to subclass to include an implementation of HEAD
--
Brian Sutherland
Metropolis - "it's the first movie with a robot. And she's a woman.
And she's EVIL!!"