Re: [rdiff-backup-users] Memory usage during regressions

[Claus-Justus Heine]

On 08/06/2011 07:54 PM, Claus-Justus Heine wrote:
> On 08/06/2011 07:51 PM, Maarten Bezemer wrote:
> > I don't know the internals of rdiff-backup, but I do know that there is
> > a runtime option to disable compression of .snapshot and .diff files.
> > When file operations are done properly, this might be helpful (using
> > memory mapped file access instead of decompressing/compressing, so
> > effectively using pointers to disk instead of ram).
> >
> > But I don't know whether it would work that way. The only thing I do
> > know is that it will at best help you in the future, as it is not going
> > to change anything wrt historic files.
>
> Thanks, Maarten, as Robert pointed out: only the two most recent backups
> are needed for regression, so it is at least worth trying. I was not
> aware of the switch (I probably should RTFM with more concentration).

I have started reading through the rdiff-backup sources. 
--no-compression doesn't seem to be fully implemented, and in particular 
might not work at all with regressions. Also, the 700MB meta-data should 
not be a problem. RB simply has a sliding 32kb window on that file, so 
this cannot the reason for the memory usage (32kb is already a waste of 
CPU time, a record in the meta-data file is something around 200 bytes). 
Maybe I try to dig this down. Perhaps one should first port the thing to 
python3, at least before starting to hack around. And maybe this changes 
already something (according to Murphy this should even increase the 
memory hungriness of RB).

Best,

Claus