[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Backend for Amazon S3
From: |
Kenneth Loafman |
Subject: |
Re: [Duplicity-talk] Backend for Amazon S3 |
Date: |
Sat, 07 Jul 2007 20:04:58 -0500 |
User-agent: |
Thunderbird 1.5.0.12 (X11/20070604) |
I'm working on the RC9 patches at the moment. This looks a bit bare
bones. Don't we need to address retries for network errors, timeouts,
etc., or is that handled in boto itself?
...Thanks,
...Ken
Eric Evans wrote:
> I filed a bug[1] for this a couple of days ago, but thought it might
> warrant a post here in case others are interested in testing.
>
> The current duplicity backend for S3 uses the bitbucket module from
> Mitch Garnaat which has been deprecated in favor of boto[2].
>
> I've worked up a patch for a boto-using-backend which is attached to the
> bug report and this email. Use of this backend is identical to the
> current one with one exception, you need to export the AWS_ACCESS_KEY_ID
> and AWS_SECRET_ACCESS_KEY environment variables with your Amazon key id
> and secret as opposed to the current S3KEY/S3SECRET combo.
>
> I'm also working to get boto into the Debian archive and currently have a
> package waiting in the NEW queue. Any Debian users interested in testing
> this patch can grab the package using the following sources.
>
> deb http://people.debian.org/~eevans/python-boto binary/
> deb-src http://people.debian.org/~eevans/python-boto source/
>
> Any feedback would be much appreciated.
>
> Regards,
>
>
> [1] http://savannah.nongnu.org/bugs/?20339
> [2] http://code.google.com/p/boto/
>
>
>
> ------------------------------------------------------------------------
>
> --- a/duplicity/backends.py Fri Jun 29 20:09:26 2007 +0000
> +++ b/duplicity/backends.py Tue Jul 03 16:27:13 2007 -0500
> @@ -680,6 +680,62 @@ class BitBucketBackend(Backend):
> % file)
> self._connect()
> del self.bucket[file]
> +
> +class BotoBackend(Backend):
> + """
> + Backend for Amazon's Simple Storage System, (aka Amazon S3), though
> + the use of the boto module, (http://code.google.com/p/boto/).
> +
> + To make use of this backend you must export the environment variables
> + AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your Amazon Web
> + Services key id and secret respectively.
> + """
> +
> + def __init__(self, parsed_url):
> + try:
> + from boto.s3.connection import S3Connection
> + from boto.s3.key import Key
> + except ImportError:
> + raise BackendException("This backend requires the boto
> library, " \
> + "(http://code.google.com/p/boto/).")
> +
> + self.key_class = Key
> +
> + self.conn = S3Connection()
> + self.bucket_name = parsed_url.suffix
> + self.bucket = self.conn.create_bucket(self.bucket_name)
> +
> + if not (os.environ.has_key('AWS_ACCESS_KEY_ID') and
> + os.environ.has_key('AWS_SECRET_ACCESS_KEY')):
> + raise BackendException("The AWS_ACCESS_KEY_ID and " \
> + "AWS_SECRET_ACCESS_KEY environment variables
> are not set.")
> +
> + if '/' in self.bucket_name:
> + raise BackendException("Invalid bucket specification.")
> +
> + def put(self, source_path, remote_filename=None):
> + if not remote_filename:
> + remote_filename = source_path.get_filename()
> + log.Log("Uploading %s to Amazon S3" % remote_filename, 5)
> + key = self.key_class(self.bucket)
> + key.key = remote_filename
> + key.set_contents_from_filename(source_path.name,
> + {'Content-Type': 'application/octet-stream'})
> +
> + def get(self, remote_filename, local_path):
> + log.Log("Downloading %s from Amazon S3" % remote_filename, 5)
> + key = self.key_class(self.bucket)
> + key.key = remote_filename
> + key.get_contents_to_filename(local_path.name)
> + local_path.setdata()
> +
> + def list(self):
> + filename_list = [k.key for k in self.bucket.get_all_keys()]
> + return filename_list
> +
> + def delete(self, filename_list):
> + for filename in filename_list:
> + self.bucket.delete_key(filename)
>
> class webdavBackend(Backend):
> """Backend for accessing a WebDAV repository.
> @@ -838,5 +894,5 @@ protocol_class_dict = {"scp": scpBackend
> "ftp": ftpBackend,
> "hsi": hsiBackend,
> "rsync": rsyncBackend,
> - "s3+http": BitBucketBackend,
> + "s3+http": BotoBackend,
> "webdav": webdavBackend}
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk
- [Duplicity-talk] Backend for Amazon S3, Eric Evans, 2007/07/07
- Re: [Duplicity-talk] Backend for Amazon S3,
Kenneth Loafman <=
- Re: [Duplicity-talk] Backend for Amazon S3, Eric Evans, 2007/07/07
- Re: [Duplicity-talk] Backend for Amazon S3, Charles Knowlton, 2007/07/08
- Re: [Duplicity-talk] Backend for Amazon S3, daacyy302, 2007/07/08
- Re: [Duplicity-talk] Backend for Amazon S3, Charles Knowlton, 2007/07/08
- Re: [Duplicity-talk] Backend for Amazon S3, Kenneth Loafman, 2007/07/08
- Re: [Duplicity-talk] Backend for Amazon S3, daacyy302, 2007/07/08
- Re: [Duplicity-talk] Backend for Amazon S3, Eric Evans, 2007/07/08
- Re: [Duplicity-talk] Backend for Amazon S3, Kenneth Loafman, 2007/07/08