Re: [Duplicity-talk] Backup succeeding but not writing manifest / sigtar

[edgar . soldin]

On 01.03.2016 05:50, Mark Grandi wrote:
> I am backing up root, because on Mac OS X, almost all of the applications are 
> stored in /Applications , (shared between all users), so I am backing up 
> root, and excluding all of the other directories besides /Users , but that 
> doesn't seem to stop Duplicity from trying to access them anyway, and failing 
> due to not having the read permissions.
> 
> Anyway, I looked into my problem myself since I had a free afternoon, and 
> found myself honestly quite concerned with what some of the code does, or 
> rather doesn't do. For instance, with Par2Backend, if for some reason the 
> par2 command doesn't return 0, then it just silently doesn't create any par2 
> files without any notice to the user. In LocalBackend, if any of the methods 
> in there that have try/except blocks, in the except blocks, they are either 
> 'pass' or just 'return False', and often the caller doesn't care about the 
> return result anyway, so any errors that occur during moving/copying files 
> are silently discarded and not presented to the user, nor does it invoke any 
> of the retry logic in Duplicity.
> 
> I mentioned the two above things, because I had trouble compiling the par2 
> binary on my Mac, and upon googling, I found a Mac OS X app that is a GUI 
> frontend to par and par2, and it appears that the author created their own 
> fork of par2, that works on mac (and has improvements such as using grand 
> central dispatch, the website is here: 
> https://gp.home.xs4all.nl/Site/MacPAR_deLuxe.html). Anyway, it sees this 
> version of par2 does not like symlinks, and the Par2Backend class uses 
> symlinks when creating the par2 files. Discovering why the par2 files were 
> not being created required me stepping into the debugger, because no errors 
> are logged to the user.
> 
> Once I changed the symlink to a hardlink, and modified the command line being 
> passed to par2 somewhat, I was stuck on the issue that I originally posted:  
> the difftar files + par2 files were being moved to the correct spot, but not 
> the manifest or sigtar files and their par2 counterparts. Stepping into the 
> debugger, i found that the difftar files are being written with Par2Backend 
> wrapping LocalBackend.put(), but the sigtar and manifest files are wrapping 
> LocalBackend._move(). However, LocalBackend._move() is using Path.rename() 
> instead of Path.move() ! It even says in the docstring for Path.move(): "Like 
> rename but destination may be on different file system". 
> 
> So there lies my problem, my backup destination was on a different file 
> system, but due to LocalBackend._move() calling the wrong underlying Path 
> method, and swallowing exceptions and only returning false (that Par2Backend 
> doesn't even check), it was not moving the sigtar and manifest files + par2 
> files because of "[Errno 18] Cross-device link". 
> 
> The patch to fix this is just simply replacing LocalBackend._move() with:
> 
>     def _move(self, source_path, remote_filename):
>         target_path = self.remote_pathdir.append(remote_filename)
>         try:
>             source_path.setdata() # maybe only needed in Par2Backend?
>             source_path.move(target_path)
>             return True
>         except OSError as e:
>             return False
> 
> This does not solve the concerns I mentioned with swallowing exceptions 
> however. 
> 

hey Mark,

well done debugging the issue! i totally agree w/ you that the duplicity code 
is not in perfect shape. thanks for investing the time to find and fix your 
issue.

wrt. to the sym-/hardlink par2 issue. as hardlinks are only avail to root 
users, would you mind to have a look if you can come up w/ a solution that 
works w/o links at all for the par2 backend? if you provide a patch i will 
create a branch that fixes your issues that can be merged into the next release 
by Ken.

..ede