Re: Formal methods for comparing model runs

[Rick Riolo]

Some of these issues were raised in this SFI working paper:

  95-07-065
  Aligning Simulation Models: A Case Study and Results
  Robert Axtell, Robert Axelrod, Joshua M. Epstein, and Michael D. Cohen

I think this was later published in 
The Journal of Computational and Mathematical Organization Theory 

Axelrod may have also addressed some of these issues in
a paper of his in:

   Simulating Social Phenomena
   R. Conte, R. Hegselmann and P. Terna (eds).
   Springer-Verlag, 1997.

But alas, as I recall it, they didn't come up with some simple
answer or techniques to address these issues (esp. the
ones related to comparing distributions).

- r 

Rick Riolo                           address@hidden
Center for Study of Complex Systems (CSCS)
4477 Randall Lab                
University of Michigan         Ann Arbor MI 48109-1120
Phone: 734 763 3323                  Fax: 734 763 9267
http://www.pscs.umich.edu/PEOPLE/rlr-home.html

On Wed, 3 Nov 1999, M. Lang / S. Railsback wrote:

> Date: Wed, 03 Nov 1999 20:41:45 -0800
> From: M. Lang / S. Railsback <address@hidden>
> Reply-To: address@hidden
> To: "address@hidden" <address@hidden>
> Subject: Formal methods for comparing model runs
> 
> I'm trying to write up some comparisons of model scenarios; for example,
> comparing the distributions of water depth and velocity used by fish
> when we make two alternative assumptions about how they select habitat.
> 
> At first it seemed obvious to compare the scenarios with a
> Kolmogorov-Smirnov test for differences between distributions. Now,
> though, it seems just as obvious that hypothesis-testing statistics like
> these are not useful for comparing agent-based model runs. First, the
> basic purpose of hypothesis-testing methods is to distinguish
> "information" from "noise". Our model's habitat selection methods,
> though, are completely mechanistic and don't have any noise. Second,
> with a model, the sample size can be arbitrarily large. If I want, I can
> compare model runs with 800 fish each observed on each of 10 days
> (sample size: 8000), or observed on each of 20 days (sample size:
> 16,000). Because the test results depend on sample size, I'm finding
> that even the tiniest differences in results between scenarios are
> statistically significant. 
> 
> So the consequence is that statistically significant results clearly are
> not biologically significant. (People that count real fish would be very
> jealous.) Consequently, my current plan is to show the distributions
> resulting from each model scenario and talk about the potential
> biological significance of the differences in distributions.
> 
> Anybody know where this issue is discussed in the literature,
> specifically for agent-based models? 
> 
> Steve
> -- 
> address@hidden
> Lang, Railsback & Assoc.
> 250 California Ave., Arcata CA 95521
> 707-822-0453; Fax 822-1868
> 
> 
>                   ==================================
>    Swarm-Modelling is for discussion of Simulation and Modelling techniques
>    esp. using Swarm.  For list administration needs (esp. [un]subscribing),
>    please send a message to <address@hidden> with "help" in the
>    body of the message.
>                   ==================================
> 



                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================