Re: [Duplicity-talk] Moving old sets to cold storage (listing files to be deleted)

[edgar . soldin]

On 15.04.2016 12:35, Ladislav Laska wrote:
> On Fri, Apr 15, 2016 at 12:23:45PM +0200, address@hidden wrote:
>> On 15.04.2016 12:18, Ladislav Laska wrote:
>>>> how about just sorting the file list alphabetically and then selecting the 
>>>> first file with full in the name (memorizing the timestamp of it) and keep 
>>>> selecting until the next file with a different full timestamp, excluding 
>>>> it.
>>>
>>> Well, yeah, but 11k files. Although I might put something heavy on my 
>>> insert key
>>> in mc :-)
>>
>> consider using shell magic. like piping ls into awk doing exec mv or so.
> 
> Sure, I'll have to, but that's what I wanted to avoid and expected it'll be
> easily done, as it seems like a pretty common task.

feel free to offer a patch to duplicity to (re)enable what you need ;)
 
>>
>>>>> For now, I'll try to use filesystem date and hope it's accruate. Is there 
>>>>> any
>>>>> option to verify if the set is complete and no volume is missing, besides
>>>>> restoring the whole set?
>>>>>
>>>>
>>>> you could locally, or wherever verify the chain. that's a restore file by 
>>>> file and compare with a checksum. it obviously only checks files existing 
>>>> at that point of time of course, meaning files flagged as deleted before 
>>>> are not checked.
>>>> but as a basic check that should suffice, as it would throw up on corrupt 
>>>> or missing volumes/files.
>>>
>>> So the 'verify' command? Seems a bit sad. It'll do, but is it too much to 
>>> ask
>>> for a way to just check if the backend storage is instact, possibly 
>>> comparing
>>> checksums of the individual volumes?
>>>
>>
>> there are checksums that are used during verify/restore. simple listing is 
>> done via signature files, volumes are not needed for that.
>> i
>> actually to verify you have to read the volumes one by one anyway which is 
>> pretty much what a restore/verify does ;)
> 
> Well, listing uses only signatures, as you said, and I think it won't even
> notice if volumes are missing. 

it should.. on every run duplicity builds a list of existing chains. run in 
max. verbosity "-v9" and you'll see.
 
> To verify missing or corrupt volumes, I really want to avoid restoring all the
> files and some obscurities:
> 
> Suppose I use duplicity verify with time after the last incremental backup I
> just moved. It would extract volumes and check with other files. For example,
> I'm not sure what happens if:
> 
> Volume xyz is missing, but all the files in the volume (can be easily a
> single big file) have been deleted. There is no need to extract it, as it 
> should
> not exist at the specific time.

duplicity holds a checksum for every volume file (in the manifest i think), so 
if a volume is opened, it is checked against the checksum first and a error(or 
at least warning) is printed.
 
> The same applies if xyz is corrupted (incomplete after upload), the file has
> changed completely and has newer copy somewhere in other archive...

how.. file names are unique because of the time stamps.

> I'm really not sure what happens in this case, and haven't found an answer, 
> but
> unless someone tells me otherwise, I'd have to run verify/restore once for 
> every
> incremental backup, and I'm not going to do that.
> 
> It would actually be great if I could do backend check and ensure that all the
> files that may be needed in the future are on my backend and intact.

that's not really duplicity's purview. duplicity treats backends as dumb, to 
make integrating backends as easy as possible.

disadvantage of that approach of course is that there is no backend logic as 
such.

what is currently called verify is actually a compare against the datasource. a 
mere verify would test exactly what you want, but unfortunately the famous no 
one didn't contribute it so far.

regards.. ede/duply.net