Re: [Duplicity-talk] Moving old sets to cold storage (listing files to be deleted)

[edgar . soldin]

On 15.04.2016 13:18, Ladislav Laska wrote:
>> feel free to offer a patch to duplicity to (re)enable what you need ;)
> 
> It's already on my todo-list, I'll take a look during the weekend to see how
> deep would I need to dig. I might post here for some pointers if I decide to 
> do
> something about it soon.
> 
>>> Well, listing uses only signatures, as you said, and I think it won't even
>>> notice if volumes are missing. 
>>
>> it should.. on every run duplicity builds a list of existing chains. run in 
>> max. verbosity "-v9" and you'll see.
> 
> I just did a crude test: backed up a 100M directory, removed one difftar from
> the backup.
> 
> It kind of noticed that the volume is gone in collection-status. Not in a way 
> I'd expect though, it
> just wrote:
> 
> Total number of contained volumes: 3
> 
> Instead of 4. I removed volume 2. One might think it could have noticed that
> volume 2 is missing! The -v9 output does not say anything about volume 2, so I
> guess it just looks for what's there, and doesn't check for what's not.
> 
> Is this a feature? 

guess not.. volumes are numbered ascendingly

> I would consider it a bug.

me too
 
> Moreover, list-current-files happily lists all the files, even though the
> volume 2 is missing and the files are not accessible. This is what I expected,
> as it's reasonable to have this cached...

yeah.. the signature usage

> So to run verify/restore seems to be the best option for checking.
> 
>>  
>>> To verify missing or corrupt volumes, I really want to avoid restoring all 
>>> the
>>> files and some obscurities:
>>>
>>> Suppose I use duplicity verify with time after the last incremental backup I
>>> just moved. It would extract volumes and check with other files. For 
>>> example,
>>> I'm not sure what happens if:
>>>
>>> Volume xyz is missing, but all the files in the volume (can be easily a
>>> single big file) have been deleted. There is no need to extract it, as it 
>>> should
>>> not exist at the specific time.
>>
>> duplicity holds a checksum for every volume file (in the manifest i think), 
>> so if a volume is opened, it is checked against the checksum first and a 
>> error(or at least warning) is printed.
>>  
>>> The same applies if xyz is corrupted (incomplete after upload), the file has
>>> changed completely and has newer copy somewhere in other archive...
>>
>> how.. file names are unique because of the time stamps.
> 
> I don't have an answer, as I need to familiarize myself with how the files are
> stored in duplicity archives:-)
> 
>>
>>> I'm really not sure what happens in this case, and haven't found an answer, 
>>> but
>>> unless someone tells me otherwise, I'd have to run verify/restore once for 
>>> every
>>> incremental backup, and I'm not going to do that.
>>>
>>> It would actually be great if I could do backend check and ensure that all 
>>> the
>>> files that may be needed in the future are on my backend and intact.
>>
>> that's not really duplicity's purview. duplicity treats backends as dumb, to 
>> make integrating backends as easy as possible.
> 
> Sure, that's great, but it's also a reason why duplicity should be able to 
> check
> if the backend contains all expected data, isn't it? After all, the backend is
> dumb and we can't expect it to do it for us...
> 
>>
>> disadvantage of that approach of course is that there is no backend logic as 
>> such.
>>
>> what is currently called verify is actually a compare against the 
>> datasource. a mere verify would test exactly what you want, but 
>> unfortunately the famous no one didn't contribute it so far.
> 
> Yeah, and someone really should. I'll try to take a look, but unfrotunately, I
> have more pressing work to do for the next few months.
> 

haven't we all? ;).. have a nice weekend, ede/duply.net