Re: Formal methods for comparing model runs

[Dr. Miguel Villanueva]

Wow! What set of  intriguing questions you raise.  Forgive me, I know this
agent stuff is not my field, but I have to take a chance and get some
clarification (and not that I can help with any of this, but just so I can
understand the statistical issues you raise better):
 
> At first it seemed obvious to compare the scenarios with a
> Kolmogorov-Smirnov test for differences between distributions.

It is not obvious to me as to why you would use a test based on the Chi
statistic. My understanding is that Chi Square is a non parametric test that
requires nominal or ordinal data.  You talk of velocity and water depth --
both of which are based on the ratio scale.  So I do not see how you can use
a Chi with these data.  Chi is used generally with frequency data and with
preference data, not with data based on a zero point (ratio) or with data
based on an interval scale. But I am likely misreading something in your
post.

>Now, though, it seems just as obvious that hypothesis-testing statistics like
> these are not useful for comparing agent-based model runs. First, the
> basic purpose of hypothesis-testing methods is to distinguish
> "information" from "noise".

Here I am having a hard time understanding your vocabulary, since in my
field, the general purpose of hypothesis testing is to make an inference
from the sample to the population, not to distinguish noise from
information.  The question that intrigues me is what happens to statistical
inference in general when the population is, or approaches, infinity?  I am
also curious as to how do you set a 'p' value that tells you when you have
information rather than noise?  At the risk of asking the obvious (and I am
a novice here), but why don't you take your obtained distributions and
compare them with real distribution of fisheys?  I guess I am confused as to
what you comparing: Your model's first set of results with another set of
results, or your model's results with a naturally occurring distribution.

I am not certain I would agree with your comment "So the consequence is that
statistically significant results clearly are not biologically significant."
Seems you have significance that is based on a statistical artifact
(infinite sample sizes, infinite df, and thus artificial effect sizes),
rather than on your model.  You are right, though, you get a big enough n
(or N) and anything becomes possible. I would love to know how you folks in
this realm of agent based modeling determine true differences or even report
effect sizes.

I'll be curious to read what answers and comments you get.
Thanks.
mv



                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================