Re: [Duplicity-talk] Duplicity full backup has become slow

[Matthew Harrison]

Hi,

sorry, I realised just after sending that I'd left that out.

Currently, duplicity is on 0.7.12. (I'm now at Ubuntu 17.10) 

In Ubuntu 16.10 and 17.04 it was (as far as I can tell) 0.7.06

(It possibly went from 0.7.06-2ubuntu2 to 0.7.06-2ubuntu3, looking at launchpad.net, but I haven't found where to find more detail in /var/log/apt/history.log

I'll try to see what version changes happened with the dependencies.

I'm running with no-encryption, and Volsize set to 4096.

(I've also now added -v9 now to get more debug.)

Many thanks,

Matthew.

On 12 November 2017 at 18:25, edgar.soldin--- via Duplicity-talk <address@hidden> wrote:

On 11/12/2017 19:19, Matthew Harrison via Duplicity-talk wrote:
> Hi,
>
> I've been happy using duplicity to backup my Ubuntu PC to my NAS for around
> 18 months now, using rsync.
>
> I do a full back up every 2 weeks, and and incremental daily.
>
> I noticed in August that the full backup started taking around twice as
> long, from
>
> --------------[ Backup Statistics ]--------------
> StartTime 1502355166.95 (Thu Aug 10 09:52:46 2017)
> EndTime 1502373067.73 (Thu Aug 10 14:51:07 2017)
> ElapsedTime 17900.78 (4 hours 58 minutes 20.78 seconds)
> SourceFiles 1361618
> SourceFileSize 212181777813 (198 GB)
> NewFiles 1361618
> NewFileSize 212181777813 (198 GB)
> DeletedFiles 0
> ChangedFiles 0
> ChangedFileSize 0 (0 bytes)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 1361618
> RawDeltaSize 211041882531 (197 GB)
> TotalDestinationSizeChange 166597720163 (155 GB)
> Errors 0
> -------------------------------------------------
>
> to
>
> --------------[ Backup Statistics ]--------------
> StartTime 1503684459.99 (Fri Aug 25 19:07:39 2017)
> EndTime 1503718307.60 (Sat Aug 26 04:31:47 2017)
> ElapsedTime 33847.61 (9 hours 24 minutes 7.61 seconds)
> SourceFiles 1361798
> SourceFileSize 212042512113 (197 GB)
> NewFiles 1361798
> NewFileSize 212042512113 (197 GB)
> DeletedFiles 0
> ChangedFiles 0
> ChangedFileSize 0 (0 bytes)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 1361798
> RawDeltaSize 210902807446 (196 GB)
> TotalDestinationSizeChange 166567853868 (155 GB)
> Errors 0
> -------------------------------------------------
>
> As you can see, pretty much the same statistics except for the time taken.
>
> Now in between these 2 full backups, probably the most significant change
> to my system was an upgrade from Ubuntu 16.10 to 17.04.
>
> I don't think network speed is an issue because some of the (4GB) vol files
> can take as little as 5 minutes (looking at the timestamps) which I assume
> includes creation and transfer time.
> I've noticed that the slowest archive files to be produced took around 1h
> 25m, and either consisted of lots and lots of small files, or very large
> binary files.
>
> Is anyone aware of any change in the versions of the Ubuntu packages that
> duplicity relies on, between Ubuntu 16.10 and 17.04 that might have caused
> this slowdown?
>
> I haven't looked yet at adding extra logging to help see if my thoughts on
> creation vs transfer time are correct.
>
> On the plus side, this has nudged me to look more closely at what
> unnecessary cruft I'm backing up :D
>
> Many thanks, and best regards,

Matthew,

what are the respective duplicity versions that were/are used?

..ede/duply.net

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk