Re: [Duplicity-talk] Duplicity hanging after errors

[E.B.]

> > Hi,
> > I'm running duplicity from a cron script on two ubuntu 12.04 servers
> > (installed from apt-get, so duplicity version 0.6.25)

my mistaken, this is a hand-compiled installation of 0.6.25

> > Big problem that the error causes duplicity to hang and not exit.
> > Wondering why that is? If I add "--progress" then it will die (the
> > trace goes through dummy_backup first where it dies, but
> > write_multivol just hangs). Is my only solution here to add
> > "--progress" to my cron script?
>
> no, usually you would raise --verbosity=debug (-v9) and run
> from command line to see if you get the same error, and more
> details.

i did that before my first post, the output is from the CLI.
My question here was precisely that unless this bug becomes
fixed, is my only way to be certain a cron job doesn't hang
to use the --progress option?

right now to be safe, i use --progress with grep -v ETA to
remove the progress indicators from the cron log. but i'm not
sure how heavy performance hit is from using --progress

> the error indicates that at least one of the backup's files
> are corrupted. you seem to use compression but no encryption
> (right?) as gzip tries to decompress and fails.

No, I use gpg and asymmetric encryption. Why did you guess I
don't use GPG? Is it possible I think I'm using GPG but I am
mistaken?

But I'm also not sure you saw BOTH errors. My problem is with
duplicity hanging after different kinds of errors on two
machines.

So do BOTH of the following errors indicate the same thing
of corrupted backup file(s)?

> > IOError: CRC check failed 0x43872847 != 0x7d505e15L

> > error: Error -3 while decompressing: invalid stored block lengths

(see my first mail for full error traces)

> PS: on errors please always send the complete
> output of a maximum verbosity (-v9) duplicity
> run that reproduces the error. obfuscate text
> that you deem private in that log dump and
> attach-post it to this mailing list.

sorry, i wasn't sure if it was useful to post
so much data to the mailing list

> you may attach a compressed file if the mailing
> list file size limit is reached.

i was looking at the next level below 9, but now
I see maybe another clue on level 9.  problem is
the verbose (-v9) log for first machine is 200MB.
log for 2nd machine is 20MB.  even if i can post
this somewhere, it's too much private info to
clean up, I'm sorry. So if you know what parts you
want, I am pleased to post them.

For now, I post the parts right at the error below
in this email also on bug tracker

> it shouldn't hang though. please file a bug
> https://bugs.launchpad.net/duplicity

ok, i filed a bug.
https://bugs.launchpad.net/duplicity/+bug/1405502

i can't hold on to my corrupted backups for much
longer (maybe a week) due to space limitations, so
please let me know what I can provide to help fix
this problem at your earliest convenience

> consider using "par2+", see manpage
> http://duplicity.nongnu.org/duplicity.1.html

Thank you so much for your suggestion. I think i am interested
in using par2 but the manpage doesn't tell much what it is.
I read wikipedia and google a bit about it, seems a parity
tool to help reconstruct corrupt archive files but still limited
info about it, especially in context uf duplicity. I'd be glad
to read more about it anyother places.

But seems easy because only change is add "par2+" to begining
of my backend config on duplicty cli.  Question is can I add
par2+ to existing archives? Or must they be new?


Here is -v9 log for the error messages on TWO machines,
different kinds of errors (but both hang).


#########
Machine 1
#########
Selecting /home/johnm/Maildir/dovecot.index.log.2
Releasing lockfile <lockfile.LinkFileLock instance at 0x3057d88>
Removing still remembered temporary file 
/tmp/duplicity-c8o1G4-tempdir/mktemp-KV1uB3-47
Removing still remembered temporary file 
/tmp/duplicity-c8o1G4-tempdir/mkstemp-uPMJ5D-1
Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1509, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1503, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1352, in main
    do_backup(action)
  File "/usr/local/bin/duplicity", line 1485, in do_backup
    incremental_backup(sig_chain)
  File "/usr/local/bin/duplicity", line 633, in incremental_backup
    globals.backend)
  File "/usr/local/bin/duplicity", line 399, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 331, in 
GPGWriteFile
    data = block_iter.next().data
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 518, 
in next
    result = self.process(self.input_iter.next())
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 190, 
in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 281, 
in collate2iters
    relem2 = riter2.next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 350, 
in combine_path_iters
    refresh_triple_list(triple_list)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 336, 
in refresh_triple_list
    new_triple = get_triple(old_triple[1])
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 322, 
in get_triple
    path = path_iter_list[iter_index].next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 234, 
in sigtar2path_iter
    for tarinfo in tf:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/tarfile.py", line 
2470, in next
    tarinfo = self.tarfile.next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/tarfile.py", line 
2319, in next
    self.fileobj.seek(self.offset)
  File "/usr/lib/python2.7/gzip.py", line 429, in seek
    self.read(1024)
  File "/usr/lib/python2.7/gzip.py", line 256, in read
    self._read(readsize)
  File "/usr/lib/python2.7/gzip.py", line 320, in _read
    self._read_eof()
  File "/usr/lib/python2.7/gzip.py", line 342, in _read_eof
    hex(self.crc)))
IOError: CRC check failed 0x43872847 != 0x7d505e15L

^C

close failed in file object destructor:
IOError: [Errno 32] Broken pipe
Exception KeyboardInterrupt in <module 'threading' from 
'/usr/lib/python2.7/threading.pyc'> ignored




#########
Machine 2
#########
note the error seems to happen on an old duplicity file
that I moved away for safe keeping while restoring from
a corrupted disk.  maybe this is the cause of the problem?

Selecting 
/var/old-corrupted/root/.cache/duplicity/a5af988f0861715f4b14466a0c03b4ed/duplicity-full.20141112T033613Z.manifest
Releasing lockfile <lockfile.LinkFileLock instance at 0x1fdcbd8>
Removing still remembered temporary file 
/tmp/duplicity-6G4eQj-tempdir/mkstemp-IaPa4Q-1
Removing still remembered temporary file 
/tmp/duplicity-6G4eQj-tempdir/mktemp-JzVNTP-3
Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1509, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1503, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1352, in main
    do_backup(action)
  File "/usr/local/bin/duplicity", line 1485, in do_backup
    incremental_backup(sig_chain)
  File "/usr/local/bin/duplicity", line 633, in incremental_backup
    globals.backend)
  File "/usr/local/bin/duplicity", line 399, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 331, in 
GPGWriteFile
    data = block_iter.next().data
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 518, 
in next
    result = self.process(self.input_iter.next())
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 190, 
in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 281, 
in collate2iters
    relem2 = riter2.next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 350, 
in combine_path_iters
    refresh_triple_list(triple_list)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 336, 
in refresh_triple_list
    new_triple = get_triple(old_triple[1])
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 322, 
in get_triple
    path = path_iter_list[iter_index].next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/diffdir.py", line 234, 
in sigtar2path_iter
    for tarinfo in tf:
  File "/usr/local/lib/python2.7/dist-packages/duplicity/tarfile.py", line 
2470, in next
    tarinfo = self.tarfile.next()
  File "/usr/local/lib/python2.7/dist-packages/duplicity/tarfile.py", line 
2319, in next
    self.fileobj.seek(self.offset)
  File "/usr/lib/python2.7/gzip.py", line 429, in seek
    self.read(1024)
  File "/usr/lib/python2.7/gzip.py", line 256, in read
    self._read(readsize)
  File "/usr/lib/python2.7/gzip.py", line 307, in _read
    uncompress = self.decompress.decompress(buf)
error: Error -3 while decompressing: invalid stored block lengths

Removing still remembered temporary file 
/root/.cache/duplicity/a5af988f0861715f4b14466a0c03b4ed/duplicity-N8H4EE-tempdir/mktemp-rDiL3N-1
Removing still remembered temporary file 
/root/.cache/duplicity/a5af988f0861715f4b14466a0c03b4ed/duplicity-nA91wx-tempdir/mktemp-S2Tpqn-1

^C

close failed in file object destructor:
IOError: [Errno 32] Broken pipe
Exception KeyboardInterrupt in <module 'threading' from 
'/usr/lib/python2.7/threading.pyc'> ignored