Re: [Duplicity-talk] duplicity collection status slowness

[edgar . soldin]

just a heads up: duplicity switch --no-compression is currently broken and 
leads to not restorable, verifyable backups.

i suggest to not zip your data to be backed up, as it make it difficult for 
librsync's rolling checksum mechanism to find only changed parts. use 
duplicity's compression instead.

..ede

On 04.09.2014 10:56, Erik Romijn wrote:
> Hello all,
> 
> Thanks for all your suggestions. I've disabled GPG compression to exclude 
> that as a factor (seems rather unnecessary anyways as we do gzip), but due to 
> other reasons I started a new backup set which means it'll be a little while 
> before the problem likely comes back. Once it does, I now have some better 
> ideas of where to look  :)
> 
> Erik
> 
> On 26 Aug 2014, at 10:54, address@hidden wrote:
> 
>> i suggest you dig away.. 19 x 92 backups sounds like a lot of backups to me. 
>>
>> on the other hand, just checked: my 3 monthly fulls with daily incrementals 
>> (roughly 3x30) do a collection status in 6 seconds.
>>
>> 1. maybe you sftp backend is just very slow. what happens when you delete 
>> the archived cache?
>> 2. does your system start swapping during listing, maybe because your vm has 
>> very little ram?
>>
>> ..ede/duply.net
>>
>>
>> On 25.08.2014 18:19, Erik Romijn wrote:
>>> Hello all,
>>>
>>> Is there anyone who might be able to provide any insights regarding this 
>>> issue? I'm happy to dig into the code myself, but I don't really know what 
>>> would be the best place to start.
>>>
>>> cheers,
>>> Erik
>>>
>>> On 20 Jul 2014, at 19:39, Erik Romijn <address@hidden> wrote:
>>>
>>>> Hello all,
>>>>
>>>> I'm using duplicity to run a few backups for my servers, and generally 
>>>> found it to work very well. However, although my data is incredibly tiny, 
>>>> duplicity has become incredibly slow, which I think I've narrowed down to 
>>>> the collection status process.
>>>>
>>>> My source file size is only 20MB, but running this backup takes about 7 
>>>> minutes, and is almost completely cpu bound. Running the collection status 
>>>> takes nearly the same amount, so it would seem that this is where the 
>>>> slowness comes from.
>>>>
>>>> I make incremental backups every 15 minutes, with a full backup after 23 
>>>> hours, so 92 sets per day. I currently have 19 backup chains, according to 
>>>> collection status, and there are no orphaned or incomplete sets.  The 
>>>> source file size is about 20MB. In total the destination volume is 154MB 
>>>> now. Running verify confirms that the backups are correct.
>>>>
>>>> These numbers are for the backups of my /var/log, but I have another 
>>>> backup of an unrelated directory of about 300MB on the same backup schema, 
>>>> which shows similar numbers for collection status.
>>>>
>>>> One workaround would be for me to move the files away from the duplicity 
>>>> destination, so that the total collection appears smaller. But that leaves 
>>>> me to wonder: why does collection status take so much time, particularly 
>>>> considering it's cpu bound?
>>>>
>>>> I'm running duplicity 0.6.23 with python 2.7.6 on an Ubuntu 14.04 VPS.
>>>>
>>>> The full duplicity command line I use is:
>>>> /usr/bin/duplicity --full-if-older-than 23h --encrypt-sign-key [...] 
>>>> --verbosity info --ssh-options=-oIdentityFile=/root/.ssh/backup_rsa 
>>>> --exclude-globbing-filelist /root/duplicity_log_exclude_filelist.txt 
>>>> /var/log sftp://address@hidden/[...]/backups/log
>>>>
>>>> Can anyone here provide insights into what might be the issue, and what 
>>>> would be the best approach to tackle this?
>>>>
>>>> cheers,
>>>> Erik
>>>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>