Re: Formal methods for comparing model runs

[M. Lang / S. Railsback]

Robert Bell wrote:
> 
> "M. Lang / S. Railsback" wrote:

OK, thanks for all the responses. A few clarifications:

> 
> > I'm trying to write up some comparisons of model scenarios; for example,
> > comparing the distributions of water depth and velocity used by fish
> > when we make two alternative assumptions about how they select habitat.
> >
> > At first it seemed obvious to compare the scenarios with a
> > Kolmogorov-Smirnov test for differences between distributions. Now,
> > though, it seems just as obvious that hypothesis-testing statistics like
> > these are not useful for comparing agent-based model runs. First, the
> > basic purpose of hypothesis-testing methods is to distinguish
> > "information" from "noise". Our model's habitat selection methods,
> > though, are completely mechanistic and don't have any noise. 

Our whole model is stochastic, mainly in determining which fish die
when. But I'm just testing how they select habitat, which is very
un-stochastic (depends only on how the fish were initialized). There is
still other kinds of noise-movement over time, etc. but very little
difference between runs where I only change the random number seed. 

> 
> I have been having the same questions....though I do have a stochastic
> component, I am pounding my head on the wall attempting to separate
> computing artifact from biological significance. Then again, I think this is
> the nature of the beast.

What happens if you do multiple model runs with different random seeds
for each scenario you're comparing, use a t-test or something to compare
the groups of model runs? I've used that approach in the past. Just be
careful not to succumb to the temptation to compare the variation due to
random number seeds to variation in somebody's real data, unless their
real data were from a very very tightly controlled experiment!

> 
> > So the consequence is that statistically significant results clearly are
> > not biologically significant. (People that count real fish would be very
> > jealous.)
> 
> I am confused here...what led you to this conclusion?  Are the results
> with the different samplings so out of whack with reality as to conclude
> they are not possible?  What kind of variance was there between
> samplings of the same size?

No, results aren't out of whack with reality, it's just that the changes
we observe in the model are very small compared with the variation in
real biological systems. For example, I ran two scenarios to see if one
species of fish selects different habitat when it does vs. doesn't have
to compete with a second species. I may get a result that says there is
a statistically significant shift from using a mean depth of 20 cm to 21
cm. Unless this shift in the mean is due to a big shift at one end of
the distribution, then it is negligible compared even with our ability
to measure depth in a rocky stream.

Steve
-- 
address@hidden
Lang, Railsback & Assoc.
250 California Ave., Arcata CA 95521
707-822-0453; Fax 822-1868


                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================