RE: Optimizing large copies

[Martin, Jason H]

While I agree that security takes longer, I'd suggest that a lot of the
time is spent on the highly syncronous nature of the network sync
operation. Looking at strace output, the procedure appears to be
something like:

1: stat a file
2. checksum a file
3. send request to master server to SYNC file
4. server checksums file
5. server returns results
6. client compares.

I think performance would be increased significantly if #3 was somehow
batched up into blocks larger then 1.  Perhaps at every directory level,
all the files in the directory (non-recursively) could be batched up and
send in a single request, without sacrificing security.

Thank you,
-Jason Martin

> -----Original Message-----
> From: Mark Burgess [mailto:Mark.Burgess@iu.hio.no] 
> Sent: Friday, September 23, 2005 7:05 AM
> To: Baker, Darryl
> Cc: Martin, Jason H; help-cfengine@gnu.org
> Subject: RE: Optimizing large copies
> 
> 
> 
> I concur with this. Cfengine insists on the correctness and 
> security of each operation -- and that takes time. A program 
> like rsync can achieve significantly better performance in 
> large copies if you have a trusted base.
> 
> M
> 
> On Fri, 2005-09-23 at 08:59 -0400, Baker, Darryl wrote:
> > I understand your concerns but with the number and size of 
> the files 
> > you are talking about you really should investigate a 
> program that is 
> > optimized for files copying. While cfengine can maintain 
> binary files 
> > this is not its primary focus. Its main focus is maintaining the 
> > system configuration. Rsync was designed to keep files identical on 
> > several machines. Using rsync in conjunction with ssh for 
> security you 
> > end up with a much better solution for maintaining binary 
> files. It is 
> > optimized just for this purpose. The use of rsync with ssh is well 
> > documented and in use in a large number of large sites.
> > 
> > 
> > _______________________________________________
> > Help-cfengine mailing list
> > Help-cfengine@gnu.org 
> > http://lists.gnu.org/mailman/listinfo/help-cfengine
> 
>