Re: [Duplicity-talk] Show volume Duplicity is working on

[Remy van Elst]

On 10/18/14 15:26, address@hidden wrote:
> On 18.10.2014 15:13, Remy van Elst wrote:
>>
>>
>> On 10/18/14 14:49, address@hidden wrote:
>>> 1. exactly what are you trying to achieve? 
>>
>> I have a server with 1.6 TB of data that takes almost a week to do a
>> full backup over an umetered gbit connection. It backs up to Rackspace
>> Openstack Swift, with a default 25MB volume size.
>>
>> The backup becomes inconsistent because stuff changes a lot on that
>> server. It has multiple +100 GB databases running. I sent an email
>> earlier to this list regarding --max-blocksize, but did not receive a
>> response yet.
> 
> you are doing snapshots or dumps of those right?

For the databases? Yes, this is one of the slaves, before Duplicity runs
pg_dump does it's trick. The actual data folders are excluded from the
duplicity run, the dump is made every 4 hours.

> 
>> (http://lists.nongnu.org/archive/html/duplicity-talk/2014-10/msg00002.html)
>>
>> With a volume size of 250 MB, or with 2048 MB the upload still takes
>> more than 3 days sadly.
>>
>> What I want to achieve is solving the long running backup issue.
> 
> do the backup to a local file:// target. that'd should be fast, at least as 
> fast as possible. then sync this with a swift client to your backend.
>  

This seems to be a good idea, I'm going to try that. Thanks :)

>>> 2. why do you need to know which volume is uploaded?
>>
>> I was troubleshooting with strace and saw the volume number. I know the
>> volume size, so with the current volume number I know which one is being
>> uploaded (async upload enabled). With that data I should be able to
>> estimate how far the backup is.
> 
> for that you can enable verbose logging or progress meter, although the 
> latter is sometimes not working properly.
>  

The progress option results in Duplicity segfaulting after +16 hours
without uploading anything or earlier when the OOM killer comes along.
This particular machine has 256G RAM.


>>> 3. duplicity is written in python script language. you can easily patch 
>>> your local duplicity to print out the current volume name.
>>>
>>
>> Is also an option, however I would use that as a last resort. I'm using
>> Duplicity 0.6.24, compiled from source on Debian 7.
> 
> try the solution, separate upload, above. that way the backup will not have 
> to wait for the upload top finish.
> 

Going to do that, thanks again :).



> ..ede/duply.net
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

0x1B7F88DC.asc
Description: application/pgp-keys

signature.asc
Description: OpenPGP digital signature