Re: [Cfengine-develop] caching checksums with cfservd

[Mark . Burgess]

Hi - I've looked at this now and see that checksums are indeed cachced but
that there is a lot of checking to see if they need to be recomputed.

Solving this problem is a little complicated since it is server side and
must serve many clients. I think my feeling was originally that not many
comparisons will be made by checksum (mostly by date) to this would not
be too heavy. Probably that is not true in our paranoid age :)


> I'm getting to the point where I'm doing a lot of data transfer with 
> cfservd, and I'd like to stop having cfservd force a checksum on every 
> file transfer (as seen in cfservd.c where ChecksumChanged is passed 'true' 
> as the value of 'refresh').


True seems only to say that one should perform the worst case check, namely
that the file we are looking at is newer than the latest change to the
database (not necessarily to that file though!!)

 
> Would there be some problem if I made this change?  I know that I would 
> need some kind of aging mechanism, but even if I could cache sums for five 
> minutes I would see a performance benefit, because I'm looking at having 
> hundreds of machines connect and get checksums for hundreds of files, 
> possible up to hundreds of MB of data, and in the space of about half an 
> hour.  If I could cache those sums, I would be much much much happier.

I'm, not sure what change you are suggesting. Set this to false? That
would only prevent updates altogether.

One possibility would be to update the database periodically instead for
all files. That does not sound like a great solution to me.

Another would be to store the timestamp of the checksum in the database (change
the format of the db) so that more specific updates can be made. (This would
also be more correct with regard to update policy).   
 
> Any idea how hard it would be to make this work?  Maybe force a stat if 
> we're running inside cfservd, and only force a sum if the stat shows a 
> difference?
> 
> Luke
> 

What do you think?

M

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Work: +47 22453272            Email:  address@hidden
Fax : +47 22453205            WWW  :  http://www.iu.hio.no/~mark
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~