Re: [Duplicity-talk] webdavs and large sync volume failing

[address@hidden]

Hi,
noone an idea what this problem could be or where I could continue trouble 
shooting?

Thanks for help.

BR,
Wolle


> Am 06.02.2016 um 09:09 schrieb address@hidden:
> 
> Hi,
> 
> I have reproduced the error with verbosity 9 and sent the logfiles to 
> ede/duply.net.
> As stated in that mail it looks like this is a memory issue 
> (/var/log/messages shows that oom-killer kicks in)
> 
> In parallel I have now upgraded my system to 
> duplicity 0.7.06
> duply 1.11.1
> 
> and re-run the backup with backend lftp+webdavs as proposed below. This works 
> fine without errors
> 
> but
> 
> If I check the status of this backup via the same backend lftp+webdavs it 
> shows no backup present.
> If I change back to webdavs backend for the status command I can see the 1 
> successfull full backup I created with the lftp+webdavs backend. 
> I tested the same with a small backup repository, here the backend 
> lftp+webdavs works fine both for the backup and the status command
> 
> A quick investigation I did shows that the status command for the big backup 
> (roughly 14GB of photos) has an issue when retrieving the file info from the 
> webdav server. There is no data returned - in the log below there should be a 
> long file list after STDOUT: but this is empty.
> 
> ——— snip from log —— 
> CMD: lftp -c 'source 
> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd 
> 'backup/photo/1971-2005/' || exit 0; ls'
> Reading results of 'lftp -c 'source 
> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd 
> 'backup/photo/1971-2005/' || exit 0; ls''
> STDERR:
> ---- Resolving host address...
> ---- 1 address found: xxxxxxx
> ---- Connecting to sd2dav. xxxxxxx port 443
> ---- Sending request...
> ---> PROPFIND / HTTP/1.1
> ---> Host: sd2dav. xxxxxxx
> ---> User-Agent: lftp/4.6.0
> ---> Accept: */*
> ---> Depth: 0
> ---> Authorization: Basic xxxxxxx
> ---> Connection: keep-alive
> ---> 
> Certificate: xxxxxxx
>  Trusted
> <--- HTTP/1.1 207 Multi-Status
> <--- Date: Sat, 06 Feb 2016 06:55:09 GMT
> <--- Server: Apache
> <--- ETag: "xxxxxxx ="
> <--- Content-Length: 1857
> <--- Content-Type: text/xml; charset="utf-8"
> <--- Vary: Accept-Encoding
> <--- Keep-Alive: timeout=3, max=100
> <--- Connection: Keep-Alive
> <--- 
> ---- Receiving body...
> ---- Hit EOF
> ---- Closing HTTP connection
> ---- Connecting to sd2dav. xxxxxxx port 443
> ---- Sending request...
> ---> PROPFIND /backup/photo/1971-2005/ HTTP/1.1
> ---> Host: sd2dav. xxxxxxx
> ---> User-Agent: lftp/4.6.0
> ---> Accept: */*
> ---> Depth: 0
> ---> Authorization: Basic xxxxxxx
> ---> Connection: keep-alive
> ---> 
> Certificate: xxxxxxx
>  Trusted
> <--- HTTP/1.1 207 Multi-Status
> <--- Date: Sat, 06 Feb 2016 06:55:19 GMT
> <--- Server: Apache
> <--- ETag: "xxxxxxx ="
> <--- Content-Length: 1368
> <--- Content-Type: text/xml; charset="utf-8"
> <--- Vary: Accept-Encoding
> <--- Keep-Alive: timeout=3, max=100
> <--- Connection: Keep-Alive
> <--- 
> ---- Receiving body...
> ---- Hit EOF
> ---- Closing HTTP connection
> ---- Connecting to sd2dav. xxxxxxx port 443
> ---- Sending request...
> ---> GET /backup/photo/1971-2005/ HTTP/1.1
> ---> Host: sd2dav. xxxxxxx
> ---> User-Agent: lftp/4.6.0
> ---> Accept: */*
> ---> Authorization: Basic xxxxxxx
> ---> Connection: keep-alive
> ---> 
> Certificate: xxxxxxx
>  Trusted
> <--- HTTP/1.1 200 OK
> <--- Date: Sat, 06 Feb 2016 06:55:20 GMT
> <--- Server: Apache
> <--- Last-Modified: Mon, 01 Feb 2016 19:44:00 GMT
> <--- ETag: "xxxxxxx ="
> <--- Accept-Ranges: bytes
> <--- Content-Length: 0
> <--- Content-Type: text/html; charset="UTF-8"
> <--- Vary: Accept-Encoding
> <--- Keep-Alive: timeout=3, max=100
> <--- Connection: Keep-Alive
> <--- 
> ---- Receiving body...
> ---- Received all
> ---- Closing HTTP connection
> 
> STDOUT:
> 
> Local and Remote metadata are synchronized, no sync needed.
> 
> ——— snip end from log ——  
> 
>> Am 02.02.2016 um 21:27 schrieb address@hidden:
>> 
>> On 02.02.2016 21:18, address@hidden wrote:
>>> Hi,
>>> 
>>> I am trying to use Duplicity and duply to backup appr 60GB data to webdavs 
>>> storage.
>>> 
>>> The script is running on a raspberrypi with raspbian jessie
>>> 
>>> Small backups are working.
>>> I have tried to run this backup in one go, it failed; I then split into 
>>> 15GB chunks it failed again with exit code 137
>>> 
>>> Any ideas what to do / where to look for?
>> 
>> can you run duplicity in max. verbosity '-v9' and send me the _complete_ 
>> output privately?
>> 
>>> Is the webdavs access not stable enough for such backup?
>> 
>> should make no difference. the backup is split into volumes of the same size 
>> anyway.
>> 
>>> Is it maybe a performance issue with the raspberry
>>> 
>>> btw I started the backup script via a cron job, it was then running several 
>>> hours before it failed.
>>> 
>> 
>> what's your duplicity version? with the latest duplicity you can install 
>> lftp and use that as alernative webdav backend via lftp+webdav://
>> 
>> ..ede/duply.net
>> 
>> 
>