Re: Discussion about file format for the future

[Eric L. Zolf]

Hi,

that's an interesting idea, I was more in the optic of keeping a simple
file based structure but if there are requirements, we can think about
alternatives, it could be a SQL or even a noSQL DB, or a keystore.

Regarding the terabytes of data, I take the point, I never foresaw to
convert the data itself, only the metadata, whereas I haven't yet a
clear view on a strategy forward: either a converting tool to "upgrade"
a repo, or convert transparently within rdiff-backup itself, or keep old
data and only use the new format for new metadata (I don't really like
this last one, but if it's the only solution...).

scanning the metadata files is a nightmare -> from which point of view?
Performance and/or complexity of code?

KR, Eric

On 04/06/2020 18:43, Patrik Dufresne wrote:
> Hi Eric,
> 
> My priority in that regard would be to make sure all of this backward 
> compatible with the previous repository. I mean, I have terabytes of data 
> and I don't foresee a way to convert all these metadata files to a new 
> format. And I'm probably not alone with this situation.
> We should probably restructure de code to put the read and write of 
> metadata into a single library. Last time I checked, all those metadata are 
> not that different. Then we could make this metadata read write library 
> evolve and allow us to read or write the old format or the new format.
> 
> If we stick with a file base approach yaml is good for me.
> 
> But two cent on the subject is, should we really keep this filebase ? For 
> rdiffweb, scanning the metadata files is a nightmare. When I just need a 
> subset of the data to be displayed to the user. I always thought a database 
> could be better fit for the job. Something like a key store or similar.
> 
> 
> On Thu, Jun 4, 2020 at 9:52 AM <rhkramer@gmail.com> wrote:
> 
>> Thanks!
>>
>> On Thursday, June 04, 2020 08:37:24 AM EricZolf wrote:
>>> Correct, configuration files and "metadata" files, but also potentially
>>> to exchange information between client and server (e.g. the version
>>> string could be enriched to exchange more information than just
>>> version).
>>
>>
>