[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Duplicity hanging after errors
From: |
edgar . soldin |
Subject: |
Re: [Duplicity-talk] Duplicity hanging after errors |
Date: |
Thu, 25 Dec 2014 11:36:21 +0100 |
User-agent: |
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 |
On 25.12.2014 01:40, E.B. wrote:
>>> Hi,
>>> I'm running duplicity from a cron script on two ubuntu 12.04 servers
>>> (installed from apt-get, so duplicity version 0.6.25)
>
> my mistaken, this is a hand-compiled installation of 0.6.25
shouldn't be an issue
>>> Big problem that the error causes duplicity to hang and not exit.
>>> Wondering why that is? If I add "--progress" then it will die (the
>>> trace goes through dummy_backup first where it dies, but
>>> write_multivol just hangs). Is my only solution here to add
>>> "--progress" to my cron script?
>>
>> no, usually you would raise --verbosity=debug (-v9) and run
>> from command line to see if you get the same error, and more
>> details.
>
> i did that before my first post, the output is from the CLI.
> My question here was precisely that unless this bug becomes
> fixed, is my only way to be certain a cron job doesn't hang
> to use the --progress option?
consider using timeout
http://serverfault.com/questions/539665/setting-timeout-for-cron-jobs
> right now to be safe, i use --progress with grep -v ETA to
> remove the progress indicators from the cron log. but i'm not
> sure how heavy performance hit is from using --progress
depends on the size of your backup. it does an unnecessary dry run first. also
it is known to be broken in some scenarios. generally i would advise not to use
it in the current state, especially not when using duplicity not interactively.
>> the error indicates that at least one of the backup's files
>> are corrupted. you seem to use compression but no encryption
>> (right?) as gzip tries to decompress and fails.
>
> No, I use gpg and asymmetric encryption. Why did you guess I
> don't use GPG? Is it possible I think I'm using GPG but I am
> mistaken?
ok, my bad.. it indicates you use gpg, but that duplicity tarfile should be
compressed internally surprises me somewhat.
> But I'm also not sure you saw BOTH errors.
yes
>My problem is with
> duplicity hanging after different kinds of errors on two
> machines.
both die in gzip.py
> So do BOTH of the following errors indicate the same thing
> of corrupted backup file(s)?
that's my guess. how can i be sure?
>
>>> IOError: CRC check failed 0x43872847 != 0x7d505e15L
>
>>> error: Error -3 while decompressing: invalid stored block lengths
>
> (see my first mail for full error traces)
>
>> PS: on errors please always send the complete
>> output of a maximum verbosity (-v9) duplicity
>> run that reproduces the error. obfuscate text
>> that you deem private in that log dump and
>> attach-post it to this mailing list.
>
> sorry, i wasn't sure if it was useful to post
> so much data to the mailing list
ok, you missed your chance. again. can't do anything further without the log
dump.
>> you may attach a compressed file if the mailing
>> list file size limit is reached.
>
> i was looking at the next level below 9, but now
> I see maybe another clue on level 9. problem is
> the verbose (-v9) log for first machine is 200MB.
> log for 2nd machine is 20MB. even if i can post
> this somewhere, it's too much private info to
> clean up, I'm sorry. So if you know what parts you
> want, I am pleased to post them.
sorry, i can't. either you provide it, or not. you are free to obfuscate your
local file name, as far as i am concerned. but i need to see the rest
(duplicity command, volume names ...)
> For now, I post the parts right at the error below
> in this email also on bug tracker
>
>> it shouldn't hang though. please file a bug
>> https://bugs.launchpad.net/duplicity
>
> ok, i filed a bug.
> https://bugs.launchpad.net/duplicity/+bug/1405502
>
> i can't hold on to my corrupted backups for much
> longer (maybe a week) due to space limitations, so
> please let me know what I can provide to help fix
> this problem at your earliest convenience
if you're interested to solve it, download the backup chains in question to a
local drive. duplicity backup files have a proper naming with type/date.
download everything from the last
duplicity-full-signatures.*.gpg
to the
latest file (duplicity-inc.*.gpg or duplicity-new-signatures.*.sigtar.gpg)
when the directory is listed via name/time ascending.
NOTE: files with an identical timestamp indicate one backup and hence belong
together.
>> consider using "par2+", see manpage
>> http://duplicity.nongnu.org/duplicity.1.html
>
> Thank you so much for your suggestion. I think i am interested
> in using par2 but the manpage doesn't tell much what it is.
> I read wikipedia and google a bit about it, seems a parity
> tool to help reconstruct corrupt archive files but still limited
> info about it, especially in context uf duplicity. I'd be glad
> to read more about it anyother places.
research what par2 does and then imagine duplicity is running it for you for
each file up/downloaded transparently. it'll only complain if it cant restore
something.
> But seems easy because only change is add "par2+" to begining
> of my backend config on duplicty cli. Question is can I add
> par2+ to existing archives? Or must they be new?
you'll need to start with a new backup location as the old is missing the par2
redundancy files.
>
> Here is -v9 log for the error messages on TWO machines,
> different kinds of errors (but both hang).
>
SNIP
need the full ones.. sorry.. ede
- [Duplicity-talk] Duplicity hanging after errors, E.B., 2014/12/18
- Re: [Duplicity-talk] Duplicity hanging after errors, edgar . soldin, 2014/12/19
- Re: [Duplicity-talk] Duplicity hanging after errors, E.B., 2014/12/24
- Re: [Duplicity-talk] Duplicity hanging after errors,
edgar . soldin <=
- Re: [Duplicity-talk] Duplicity hanging after errors, edgar . soldin, 2014/12/25
- Re: [Duplicity-talk] Duplicity hanging after errors, E.B., 2014/12/25
- Re: [Duplicity-talk] Duplicity hanging after errors, edgar . soldin, 2014/12/25
- Re: [Duplicity-talk] Duplicity hanging after errors, E.B., 2014/12/26
- Re: [Duplicity-talk] Duplicity hanging after errors, edgar . soldin, 2014/12/26