Well I suppose I should post a pseudo-rebuttal so here goes.
On Wed, 2002-08-14 at 05:24, Gary Polhill wrote:
Just to chime in here, we did a test of the RePast Heatbugs, the
JavaSwarm Heatbugs and the (Obj-C) Swarm Heatbugs (on a Sun Blade 1000),
and found the Obj-C Swarm heatbugs fastest, the JavaSwarm heatbugs slightly
slower, and the RePast heatbugs almost an order of magnitude slower.
(For example, in one minute, RePast runs 296 cycles of heatbugs, JavaSwarm
1760 cycles, and Swarm 2017.)
I'm not surprised that the Obj-c version is faster, but by that much!?.
I am surprised that the JavaSwarm version is that much faster. It was
roughly comparable to the repast version last I checked which was
admittedly with an earlier version of javaswarm. I know Marcus has done
lots of work on it since then. I also have anecdotal evidence of users
porting their models from swarm without any problematic loss of speed.
Firstly, I must apologise -- I was running RePast with Java 1.2 instead of 1.3, and the
Swarm models with their default 80x80 HeatSpace rather than RePast's 100x100 in my rush
to put some figures together. Not a fair test to begin with! Taking in these changes, the
RePast figure goes up slightly to over 300, and the Swarm figures come down ~1600 for
Obj-C Swarm, ~1400 for JavaSwarm. This is on the same unix box as before, and Swarm is
still 4-5 times faster than RePast, though it can hardly be called "almost an order
of magnitude" any more. On my PC (where I've now installed RePast for the first
time), I get ~800 for RePast and JavaSwarm, and ~1000 for Obj-C Swarm -- much less of a
difference. (All figures are cycles/min.) Sun are clearly better at doing graphics in
Java for Windows than they are on their own platform!
In general, the big speed issue in repast models and with java in
general is the display. With the display minimized (that is, not
updating) I get over 2700 cycles / min for heatbugs on my PIII laptop
running linux. With the display showing and thus updating I get just
over 700. Graphics on Java are slow.
Secondly, although obviously running the model with no display will be quicker,
it didn't occur to me there could be such a discrepancy between the different
display models (Java vs. Tcl/Tk here, presumably) and thus assumed that there
would be roughly the same size of discrepancy between the GUI and batchmode
versions. I wasn't able to get RePast working in batchmode quickly, but figures
with the display minimised (all I did was click on the button in the top right
of the window, I didn't actually stop the display updating like you did)
correspond roughly to your experience: On the PC: Swarm: ~2200, RePast: ~1800,
JavaSwarm: ~1300, and on the Unix box: Swarm: ~2200, JavaSwarm: ~1600, RePast:
~1500. The gap between RePast and JavaSwarm is pretty much eliminated.
Anyway, I dislike this kind of benchmarking for a several reasons. The
least important of which is that it makes me queasy and usually begins a
fruitless search for something to optimize. The more important reason is
that it rarely tells the whole story. If it takes you, say, twice as
long to code and debug a model before beginning your actual runs, the
actual speed of the runs themselves is much less of a factor and may be
irrelevant. I'm _NOT_ saying that this is the case between repast and
swarm (although I may tell myself this so I can sleep at night) as I
haven't really looked at swarm for a few years, and certainly have never
done any proper comparative benchmarks
Your argument depends on how often and for how long you are intending to run
your model, though I agree that unless benchmarking shows up huge discrepancies
like it does on the Unix box with displays, it is far from being the only
comparison that matters; usability, portability, documentation, stability,
support, and modifiability being much more significant factors once it is shown
that there is not too much in it speed-wise.
So, thanks for your rebuttal (which was far from pseudo), and I hope you sleep
a little easier tonight!
Gary
Gary Polhill
Research Scientist