Re: [Duplicity-talk] Recreates and uploads already uploaded volume upon restart?

[Roman Yepishev]

Hi,

With "treats backends as dumb" - do you mean that duplicity does not
trust backends to actually upload/download/list files when requested?


MediaFire upload requires[0] sending file hash along with the payload.

If file is less than 4MB - there is a single request call. If local
checksum/size does not match the server-side, the file upload is marked
as error and the file does not appear on the server-side.

If file is more than 4MB a hash is calculated on the whole file, then
the file is split into a number of units, each carrying its own hash and
then uploaded to the server. The server then reconstructs the original
file when all units are uploaded and if original hash/size does not
match, then the upload is marked as error.

The uploads can fail for a number of reasons, including server-side
errors, and some of them don't finish correctly at all, so the
requirement to re-upload already uploaded volume increases the chance of
unsuccessful backup. The MediaFire team seems to be aware of these
issues with uploads[0], but that's beyond the scope of the backend.

[0]: http://www.mediafire.com/developers/core_api/1.3/upload/#upload_top
[1]: http://forum.mediafiredev.com/showthread.php?588


On Sun, 2015-05-17 at 15:28 +0200, address@hidden wrote:
> duplicity treats backends as dumb. there are plans to support the
> retrieval of backend specific metadata (size,dates ...), but this is
> far from being implemented completely.
> 
> how does mediafire guarantee to show only properly finished uploads?
> 
> ..ede/duply.net
> 
> 
> On 17.05.2015 15:15, Roman Yepishev wrote:
> > Aaand I found exactly why this is happening:
> > 
> > https://bugs.launchpad.net/deja-dup/+bug/487720
> > Restore fails with "Invalid data - SHA1 hash mismatch"
> > 
> > From bin/duplicity:
> > """
> > # We start one volume back in case we weren't able to finish writing
> > # the most recent block.  Actually checking if we did (via hash) would
> > # involve downloading the block.  Easier to just redo one block.
> > self.start_vol = max(len(last_backup) - 1, 0)
> > """
> > 
> > Since MediaFire upload operations are atomic (the files are either
> > uploaded completely or not), this introduces a sizable overhead when
> > volumes are 500+ MiB.
> > 
> > So at this point I am wondering: should the backends be required to
> > implement atomic uploads instead of duplicity core working around the
> > issues with backends?
> >

signature.asc
Description: This is a digitally signed message part