Re: Copy slower than rsync Re: Bugs and features

[Mark Burgess]

On  8 Nov, Adrian Phillips wrote:
>>>>>> "Mark" == Mark Burgess <Mark.Burgess@iu.hio.no> writes:
> 
>     >> Yes, embedding the rsync algorithm as an optional way to copy
>     >> files would be the only sensible way to marry cfengine & rsync.
>     >> This would allow the efficiency of rsync (over slowish links
>     >> with slightly changing data) with the authentication/security
>     >> already present in cfengine.
>     >> 
>     >> If the rsync algorithm is a big win, it is a shame to have to
>     >> hack up automated ssh logins and run it in a shell just to use
>     >> it securely from cfengine.
>     >> 
> 
>     Mark> Tests show that it isn't, except for certain types of copy
>     Mark> (which probably don't occur that often on a stable
>     Mark> system). The need for rsync efficiency seems to be mainly
>     Mark> folklore.
> 
> To carry on this conversation from September last year :-
> 
> I have a approx. 40MB directory that is synchronised from the cfengine
> server every hour. I have noticed that cfengine takes a relatively
> long time to do the copy, whereas an rsync of the same area (but in
> the reverse direction) takes no time at all :-
> 
>     *********************************************************************
>      Main Tree Sched: copy pass 1 @ Fri Nov  8 12:29:44 2002
>     *********************************************************************
> 
>     <snipped a few quick copies, takes a few seconds>
> 
>     Checking copy from cfengine:/usr/local/cfengine/dnmi/vb to /dnmi/vb
>     cfengine:vbtest: Checking link from /dnmi/vb/htdocs/web/exww/index.html 
> to exww.html
>     Not checking whether link pointed object exists
>     cfengine:vbtest: Link (/dnmi/vb/htdocs/web/exww/index.html->exww.html) 
> exists.
>     Checking copy from cfengine:/usr/local/cfengine/dnmi/vb/conf to 
> /dnmi/vb/conf
>     Saving the setuid log in /var/cfengine/cfagent..log
>     Job start time set to Fri Nov  8 12:33:44 2002
> 
> 
>     *********************************************************************
>      Main Tree Sched: resolve pass 1 @ Fri Nov  8 12:33:44 2002
>     *********************************************************************
> 
> Here is the copy section :-
>       # Copy vb from dawn - not day copy, apache conf, or mysql databases
>       # Don't use owner/group here
>       ${cfrep}/dnmi/vb dest=/dnmi/vb server=${cfmaster} ignore=day 
> ignore=conf ignore=databases ignore=kundedata recurse=inf
> 
> compare this to an rsync from the cfengine server to this machine :-
> 
>     freeze:~# time rsync --verbose -a -e ssh ~vb/ vbtest:/dnmi/vb 
>     building file list ... done
>     <snipped a few directories>
>     wrote 40726 bytes  read 72 bytes  9066.22 bytes/sec
>     total size is 38207548  speedup is 936.51
> 
>     real    0m4.500s
>     user    0m0.110s
>     sys     0m0.030s
> 
> Note, the cfengine copy takes approx. 4 minutes, whereas the similar
> rsync command takes 4 seconds !!
> 
> I have the same problem with syncing my cfengine "area" from the
> master cfengine server to the backup(s). Using copy would take 20 or
> more minutes, using rsync takes a few seconds.
> 
> I have absolutely no idea currently the differences between cfengine's
> copy code and rsync's method so don't know whether optimising copy is
> the better solution or bolting rsync's library into cfengine.
> 
> Sincerely,
> 
> Adrian Phillips
> 


I looked at this a while back and there are several issues

* seem to be version differences with rsync and cfengine

* cfengine spreads things out over time (low CPU, slower total time)
rsync (high CPU, short total time). Thus with multiple connections
one would expect cfengine to survive/scale longer than rsync.

* rsync trusts everything and caches like mad to go fast.

* cfengine trusts very little and does lots of checks. (In my opinion
this is the correct approach, since speed is rarely of the essence
in system admin).

I don't remember what total resource time looked like for these...

M