Re: [Duplicity-talk] webdavs and large sync volume failing

[edgar . soldin]

hey Wolle,

any news on this issue? i'd assume a timeout on the webserver wrt. the amount 
of files that it has to list. if so there is nothing duplicity can do. you 
could to raise the volume size to minimize the number of volumes, but htat 
wouldn't guarantee anything as the number of files will increase on the backend.

..ede/duply.net

On 06.02.2016 09:09, address@hidden wrote:
> Hi,
> 
> I have reproduced the error with verbosity 9 and sent the logfiles to 
> ede/duply.net.
> As stated in that mail it looks like this is a memory issue 
> (/var/log/messages shows that oom-killer kicks in)
> 
> In parallel I have now upgraded my system to 
> duplicity 0.7.06
> duply 1.11.1
> 
> and re-run the backup with backend lftp+webdavs as proposed below. This works 
> fine without errors
> 
> but
> 
> If I check the status of this backup via the same backend lftp+webdavs it 
> shows no backup present.
> If I change back to webdavs backend for the status command I can see the 1 
> successfull full backup I created with the lftp+webdavs backend. 
> I tested the same with a small backup repository, here the backend 
> lftp+webdavs works fine both for the backup and the status command
> 
> A quick investigation I did shows that the status command for the big backup 
> (roughly 14GB of photos) has an issue when retrieving the file info from the 
> webdav server. There is no data returned - in the log below there should be a 
> long file list after STDOUT: but this is empty.
> 
> ——— snip from log —— 
> CMD: lftp -c 'source 
> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd 
> 'backup/photo/1971-2005/' || exit 0; ls'
> Reading results of 'lftp -c 'source 
> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd 
> 'backup/photo/1971-2005/' || exit 0; ls''
> STDERR:
> ---- Resolving host address...
> ---- 1 address found: xxxxxxx
> ---- Connecting to sd2dav. xxxxxxx port 443
> ---- Sending request...
> ---> PROPFIND / HTTP/1.1
> ---> Host: sd2dav. xxxxxxx
> ---> User-Agent: lftp/4.6.0
> ---> Accept: */*
> ---> Depth: 0
> ---> Authorization: Basic xxxxxxx
> ---> Connection: keep-alive
> ---> 
> Certificate: xxxxxxx
>   Trusted
> <--- HTTP/1.1 207 Multi-Status
> <--- Date: Sat, 06 Feb 2016 06:55:09 GMT
> <--- Server: Apache
> <--- ETag: "xxxxxxx ="
> <--- Content-Length: 1857
> <--- Content-Type: text/xml; charset="utf-8"
> <--- Vary: Accept-Encoding
> <--- Keep-Alive: timeout=3, max=100
> <--- Connection: Keep-Alive
> <--- 
> ---- Receiving body...
> ---- Hit EOF
> ---- Closing HTTP connection
> ---- Connecting to sd2dav. xxxxxxx port 443
> ---- Sending request...
> ---> PROPFIND /backup/photo/1971-2005/ HTTP/1.1
> ---> Host: sd2dav. xxxxxxx
> ---> User-Agent: lftp/4.6.0
> ---> Accept: */*
> ---> Depth: 0
> ---> Authorization: Basic xxxxxxx
> ---> Connection: keep-alive
> ---> 
> Certificate: xxxxxxx
>   Trusted
> <--- HTTP/1.1 207 Multi-Status
> <--- Date: Sat, 06 Feb 2016 06:55:19 GMT
> <--- Server: Apache
> <--- ETag: "xxxxxxx ="
> <--- Content-Length: 1368
> <--- Content-Type: text/xml; charset="utf-8"
> <--- Vary: Accept-Encoding
> <--- Keep-Alive: timeout=3, max=100
> <--- Connection: Keep-Alive
> <--- 
> ---- Receiving body...
> ---- Hit EOF
> ---- Closing HTTP connection
> ---- Connecting to sd2dav. xxxxxxx port 443
> ---- Sending request...
> ---> GET /backup/photo/1971-2005/ HTTP/1.1
> ---> Host: sd2dav. xxxxxxx
> ---> User-Agent: lftp/4.6.0
> ---> Accept: */*
> ---> Authorization: Basic xxxxxxx
> ---> Connection: keep-alive
> ---> 
> Certificate: xxxxxxx
>   Trusted
> <--- HTTP/1.1 200 OK
> <--- Date: Sat, 06 Feb 2016 06:55:20 GMT
> <--- Server: Apache
> <--- Last-Modified: Mon, 01 Feb 2016 19:44:00 GMT
> <--- ETag: "xxxxxxx ="
> <--- Accept-Ranges: bytes
> <--- Content-Length: 0
> <--- Content-Type: text/html; charset="UTF-8"
> <--- Vary: Accept-Encoding
> <--- Keep-Alive: timeout=3, max=100
> <--- Connection: Keep-Alive
> <--- 
> ---- Receiving body...
> ---- Received all
> ---- Closing HTTP connection
> 
> STDOUT:
> 
> Local and Remote metadata are synchronized, no sync needed.
> 
> ——— snip end from log ——  
> 
>> Am 02.02.2016 um 21:27 schrieb address@hidden:
>>
>> On 02.02.2016 21:18, address@hidden wrote:
>>> Hi,
>>>
>>> I am trying to use Duplicity and duply to backup appr 60GB data to webdavs 
>>> storage.
>>>
>>> The script is running on a raspberrypi with raspbian jessie
>>>
>>> Small backups are working.
>>> I have tried to run this backup in one go, it failed; I then split into 
>>> 15GB chunks it failed again with exit code 137
>>>
>>> Any ideas what to do / where to look for?
>>
>> can you run duplicity in max. verbosity '-v9' and send me the _complete_ 
>> output privately?
>>
>>> Is the webdavs access not stable enough for such backup?
>>
>> should make no difference. the backup is split into volumes of the same size 
>> anyway.
>>
>>> Is it maybe a performance issue with the raspberry
>>>
>>> btw I started the backup script via a cron job, it was then running several 
>>> hours before it failed.
>>>
>>
>> what's your duplicity version? with the latest duplicity you can install 
>> lftp and use that as alernative webdav backend via lftp+webdav://
>>
>> ..ede/duply.net
>>
>>
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>