Re: [Duplicity-talk] webdavs and large sync volume failing

[address@hidden]

Hi Edgar,

no news on this, I have not further tested it with lftp+webdavs backend as I am 
now using webdav directly.

Though I need to admit I have not run these huge backups any more as the 
increments now are much smaller.

Anything I could still test to support the lftp+webdavs backend trouble 
shooting?

Gruss,
Wolle


> Am 17.04.2016 um 16:58 schrieb address@hidden:
> 
> hey Wolle,
> 
> any news on this issue? i'd assume a timeout on the webserver wrt. the amount 
> of files that it has to list. if so there is nothing duplicity can do. you 
> could to raise the volume size to minimize the number of volumes, but htat 
> wouldn't guarantee anything as the number of files will increase on the 
> backend.
> 
> ..ede/duply.net
> 
> On 06.02.2016 09:09, address@hidden wrote:
>> Hi,
>> 
>> I have reproduced the error with verbosity 9 and sent the logfiles to 
>> ede/duply.net.
>> As stated in that mail it looks like this is a memory issue 
>> (/var/log/messages shows that oom-killer kicks in)
>> 
>> In parallel I have now upgraded my system to 
>> duplicity 0.7.06
>> duply 1.11.1
>> 
>> and re-run the backup with backend lftp+webdavs as proposed below. This 
>> works fine without errors
>> 
>> but
>> 
>> If I check the status of this backup via the same backend lftp+webdavs it 
>> shows no backup present.
>> If I change back to webdavs backend for the status command I can see the 1 
>> successfull full backup I created with the lftp+webdavs backend. 
>> I tested the same with a small backup repository, here the backend 
>> lftp+webdavs works fine both for the backup and the status command
>> 
>> A quick investigation I did shows that the status command for the big backup 
>> (roughly 14GB of photos) has an issue when retrieving the file info from the 
>> webdav server. There is no data returned - in the log below there should be 
>> a long file list after STDOUT: but this is empty.
>> 
>> ——— snip from log —— 
>> CMD: lftp -c 'source 
>> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd 
>> 'backup/photo/1971-2005/' || exit 0; ls'
>> Reading results of 'lftp -c 'source 
>> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd 
>> 'backup/photo/1971-2005/' || exit 0; ls''
>> STDERR:
>> ---- Resolving host address...
>> ---- 1 address found: xxxxxxx
>> ---- Connecting to sd2dav. xxxxxxx port 443
>> ---- Sending request...
>> ---> PROPFIND / HTTP/1.1
>> ---> Host: sd2dav. xxxxxxx
>> ---> User-Agent: lftp/4.6.0
>> ---> Accept: */*
>> ---> Depth: 0
>> ---> Authorization: Basic xxxxxxx
>> ---> Connection: keep-alive
>> ---> 
>> Certificate: xxxxxxx
>>  Trusted
>> <--- HTTP/1.1 207 Multi-Status
>> <--- Date: Sat, 06 Feb 2016 06:55:09 GMT
>> <--- Server: Apache
>> <--- ETag: "xxxxxxx ="
>> <--- Content-Length: 1857
>> <--- Content-Type: text/xml; charset="utf-8"
>> <--- Vary: Accept-Encoding
>> <--- Keep-Alive: timeout=3, max=100
>> <--- Connection: Keep-Alive
>> <--- 
>> ---- Receiving body...
>> ---- Hit EOF
>> ---- Closing HTTP connection
>> ---- Connecting to sd2dav. xxxxxxx port 443
>> ---- Sending request...
>> ---> PROPFIND /backup/photo/1971-2005/ HTTP/1.1
>> ---> Host: sd2dav. xxxxxxx
>> ---> User-Agent: lftp/4.6.0
>> ---> Accept: */*
>> ---> Depth: 0
>> ---> Authorization: Basic xxxxxxx
>> ---> Connection: keep-alive
>> ---> 
>> Certificate: xxxxxxx
>>  Trusted
>> <--- HTTP/1.1 207 Multi-Status
>> <--- Date: Sat, 06 Feb 2016 06:55:19 GMT
>> <--- Server: Apache
>> <--- ETag: "xxxxxxx ="
>> <--- Content-Length: 1368
>> <--- Content-Type: text/xml; charset="utf-8"
>> <--- Vary: Accept-Encoding
>> <--- Keep-Alive: timeout=3, max=100
>> <--- Connection: Keep-Alive
>> <--- 
>> ---- Receiving body...
>> ---- Hit EOF
>> ---- Closing HTTP connection
>> ---- Connecting to sd2dav. xxxxxxx port 443
>> ---- Sending request...
>> ---> GET /backup/photo/1971-2005/ HTTP/1.1
>> ---> Host: sd2dav. xxxxxxx
>> ---> User-Agent: lftp/4.6.0
>> ---> Accept: */*
>> ---> Authorization: Basic xxxxxxx
>> ---> Connection: keep-alive
>> ---> 
>> Certificate: xxxxxxx
>>  Trusted
>> <--- HTTP/1.1 200 OK
>> <--- Date: Sat, 06 Feb 2016 06:55:20 GMT
>> <--- Server: Apache
>> <--- Last-Modified: Mon, 01 Feb 2016 19:44:00 GMT
>> <--- ETag: "xxxxxxx ="
>> <--- Accept-Ranges: bytes
>> <--- Content-Length: 0
>> <--- Content-Type: text/html; charset="UTF-8"
>> <--- Vary: Accept-Encoding
>> <--- Keep-Alive: timeout=3, max=100
>> <--- Connection: Keep-Alive
>> <--- 
>> ---- Receiving body...
>> ---- Received all
>> ---- Closing HTTP connection
>> 
>> STDOUT:
>> 
>> Local and Remote metadata are synchronized, no sync needed.
>> 
>> ——— snip end from log ——  
>> 
>>> Am 02.02.2016 um 21:27 schrieb address@hidden:
>>> 
>>> On 02.02.2016 21:18, address@hidden wrote:
>>>> Hi,
>>>> 
>>>> I am trying to use Duplicity and duply to backup appr 60GB data to webdavs 
>>>> storage.
>>>> 
>>>> The script is running on a raspberrypi with raspbian jessie
>>>> 
>>>> Small backups are working.
>>>> I have tried to run this backup in one go, it failed; I then split into 
>>>> 15GB chunks it failed again with exit code 137
>>>> 
>>>> Any ideas what to do / where to look for?
>>> 
>>> can you run duplicity in max. verbosity '-v9' and send me the _complete_ 
>>> output privately?
>>> 
>>>> Is the webdavs access not stable enough for such backup?
>>> 
>>> should make no difference. the backup is split into volumes of the same 
>>> size anyway.
>>> 
>>>> Is it maybe a performance issue with the raspberry
>>>> 
>>>> btw I started the backup script via a cron job, it was then running 
>>>> several hours before it failed.
>>>> 
>>> 
>>> what's your duplicity version? with the latest duplicity you can install 
>>> lftp and use that as alernative webdav backend via lftp+webdav://
>>> 
>>> ..ede/duply.net
>>> 
>>> 
>> 
>> 
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk