Re: [Duplicity-talk] webdavs and large sync volume failing

[edgar . soldin]

Wolle,

sorry, lost you in my email tsunami!
there are obviously several issues in the mix here. let's try to tackle them in 
order below

first some more general questions?

A. where did you get your duplicity? did you build it yourself, which 
repository.
B. how much ram are we talking about in your raspberrypi? (never played around 
w/ it)

usually memory issues in the past were caused by maintainers fiddling w/ our 
gpginterface, that's why i am asking where you've got your duplicity from to 
check that.

What you can try.

1. webdav & oom-killing

run duplicity w/ a command you know that caused the issue in the past and in a 
second terminal observe memory usage and processes. tyr to find out which 
(sub)processes stuff your memory.
also make sure that /tmp or the folder you gave for temp files is not mounted 
to a in-memory file system.

2. lftp not listing

lftp backend probably has an issue in the listing code. please run the listing 
again in max. verbosity but backup/replace duplicity/backends/lftpbackend.py w/ 
the copy attached beforehand.

the resulting -v9 log output might tell us why the returned list is empty.

..ede/duply.net

On 18.02.2016 21:51, address@hidden wrote:
> Hi,
> noone an idea what this problem could be or where I could continue trouble 
> shooting?
> 
> Thanks for help.
> 
> BR,
> Wolle
> 
> 
>> Am 06.02.2016 um 09:09 schrieb address@hidden:
>>
>> Hi,
>>
>> I have reproduced the error with verbosity 9 and sent the logfiles to 
>> ede/duply.net.
>> As stated in that mail it looks like this is a memory issue 
>> (/var/log/messages shows that oom-killer kicks in)
>>
>> In parallel I have now upgraded my system to 
>> duplicity 0.7.06
>> duply 1.11.1
>>
>> and re-run the backup with backend lftp+webdavs as proposed below. This 
>> works fine without errors
>>
>> but
>>
>> If I check the status of this backup via the same backend lftp+webdavs it 
>> shows no backup present.
>> If I change back to webdavs backend for the status command I can see the 1 
>> successfull full backup I created with the lftp+webdavs backend. 
>> I tested the same with a small backup repository, here the backend 
>> lftp+webdavs works fine both for the backup and the status command
>>
>> A quick investigation I did shows that the status command for the big backup 
>> (roughly 14GB of photos) has an issue when retrieving the file info from the 
>> webdav server. There is no data returned - in the log below there should be 
>> a long file list after STDOUT: but this is empty.
>>
>> ——— snip from log —— 
>> CMD: lftp -c 'source 
>> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd 
>> 'backup/photo/1971-2005/' || exit 0; ls'
>> Reading results of 'lftp -c 'source 
>> '/home/backup/tmp/duplicity-uzDqze-tempdir/mkstemp-yXoZe9-1'; cd 
>> 'backup/photo/1971-2005/' || exit 0; ls''
>> STDERR:
>> ---- Resolving host address...
>> ---- 1 address found: xxxxxxx
>> ---- Connecting to sd2dav. xxxxxxx port 443
>> ---- Sending request...
>> ---> PROPFIND / HTTP/1.1
>> ---> Host: sd2dav. xxxxxxx
>> ---> User-Agent: lftp/4.6.0
>> ---> Accept: */*
>> ---> Depth: 0
>> ---> Authorization: Basic xxxxxxx
>> ---> Connection: keep-alive
>> ---> 
>> Certificate: xxxxxxx
>>  Trusted
>> <--- HTTP/1.1 207 Multi-Status
>> <--- Date: Sat, 06 Feb 2016 06:55:09 GMT
>> <--- Server: Apache
>> <--- ETag: "xxxxxxx ="
>> <--- Content-Length: 1857
>> <--- Content-Type: text/xml; charset="utf-8"
>> <--- Vary: Accept-Encoding
>> <--- Keep-Alive: timeout=3, max=100
>> <--- Connection: Keep-Alive
>> <--- 
>> ---- Receiving body...
>> ---- Hit EOF
>> ---- Closing HTTP connection
>> ---- Connecting to sd2dav. xxxxxxx port 443
>> ---- Sending request...
>> ---> PROPFIND /backup/photo/1971-2005/ HTTP/1.1
>> ---> Host: sd2dav. xxxxxxx
>> ---> User-Agent: lftp/4.6.0
>> ---> Accept: */*
>> ---> Depth: 0
>> ---> Authorization: Basic xxxxxxx
>> ---> Connection: keep-alive
>> ---> 
>> Certificate: xxxxxxx
>>  Trusted
>> <--- HTTP/1.1 207 Multi-Status
>> <--- Date: Sat, 06 Feb 2016 06:55:19 GMT
>> <--- Server: Apache
>> <--- ETag: "xxxxxxx ="
>> <--- Content-Length: 1368
>> <--- Content-Type: text/xml; charset="utf-8"
>> <--- Vary: Accept-Encoding
>> <--- Keep-Alive: timeout=3, max=100
>> <--- Connection: Keep-Alive
>> <--- 
>> ---- Receiving body...
>> ---- Hit EOF
>> ---- Closing HTTP connection
>> ---- Connecting to sd2dav. xxxxxxx port 443
>> ---- Sending request...
>> ---> GET /backup/photo/1971-2005/ HTTP/1.1
>> ---> Host: sd2dav. xxxxxxx
>> ---> User-Agent: lftp/4.6.0
>> ---> Accept: */*
>> ---> Authorization: Basic xxxxxxx
>> ---> Connection: keep-alive
>> ---> 
>> Certificate: xxxxxxx
>>  Trusted
>> <--- HTTP/1.1 200 OK
>> <--- Date: Sat, 06 Feb 2016 06:55:20 GMT
>> <--- Server: Apache
>> <--- Last-Modified: Mon, 01 Feb 2016 19:44:00 GMT
>> <--- ETag: "xxxxxxx ="
>> <--- Accept-Ranges: bytes
>> <--- Content-Length: 0
>> <--- Content-Type: text/html; charset="UTF-8"
>> <--- Vary: Accept-Encoding
>> <--- Keep-Alive: timeout=3, max=100
>> <--- Connection: Keep-Alive
>> <--- 
>> ---- Receiving body...
>> ---- Received all
>> ---- Closing HTTP connection
>>
>> STDOUT:
>>
>> Local and Remote metadata are synchronized, no sync needed.
>>
>> ——— snip end from log ——  
>>
>>> Am 02.02.2016 um 21:27 schrieb address@hidden:
>>>
>>> On 02.02.2016 21:18, address@hidden wrote:
>>>> Hi,
>>>>
>>>> I am trying to use Duplicity and duply to backup appr 60GB data to webdavs 
>>>> storage.
>>>>
>>>> The script is running on a raspberrypi with raspbian jessie
>>>>
>>>> Small backups are working.
>>>> I have tried to run this backup in one go, it failed; I then split into 
>>>> 15GB chunks it failed again with exit code 137
>>>>
>>>> Any ideas what to do / where to look for?
>>>
>>> can you run duplicity in max. verbosity '-v9' and send me the _complete_ 
>>> output privately?
>>>
>>>> Is the webdavs access not stable enough for such backup?
>>>
>>> should make no difference. the backup is split into volumes of the same 
>>> size anyway.
>>>
>>>> Is it maybe a performance issue with the raspberry
>>>>
>>>> btw I started the backup script via a cron job, it was then running 
>>>> several hours before it failed.
>>>>
>>>
>>> what's your duplicity version? with the latest duplicity you can install 
>>> lftp and use that as alernative webdav backend via lftp+webdav://
>>>
>>> ..ede/duply.net
>>>
>>>
>>
> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>

lftpbackend.py
Description: Text document