Re: A whiff of reality...

[M. Lang & S. Railsback]

Doug Donalson wrote:

> Maybe it would be useful if some of us put together the methods we use to
> test our models and maybe some examples.  I started writing some of my stuff
> up as part of my dissertation but the work has been sidetracked.  After my
> defense, scheduled for June 29, I will resurect them.  I am also working on
> some stuff related to time vs. event driven schedules and choices of
> statistical distributions.  (Also mostly on hold until after 6/29.)

Alex Lancaster wrote:

> I've had this discussion enough times, that I'd really be interested
> in collaborating with some people and writing a pure methodology paper
> addressing these issues.  So once and for all, I could say to the
> critics: "well, read our paper".  Anyone interested?

Interestingly, part of what I had in this paper that got rejected was
model testing methods. I'll attach that text below. 

Alex: I agree it would be excellent to publish a paper addressing ABM
software methodology issues. The first two pages of the Swarm Intro
paper by Minar et al. possibly provides a nice outline of the issues. 

Maybe if somebody made a list of key issues, we could all contribute our
favorite examples and citations. Then a couple of us could piece it
together. Count me in.

E.g., concerning the importance of visual interfaces and the ability to
observe your agents: We have this downstream fish migration simulator in
which little salmon drift down the river. The first time I ran it I
immediately knew we had a roundoff/truncation error in the movement code
because the fish were drifting to the inside of the river bend. This
error would have seriously biased the results, and would never have been
found without the raster window.

Steve

*****************
We developed the following methods for testing IBM software. Considering
the impossibility of hand-checking model results thoroughly, these
methods provide a minimum appropriate level of testing IBM codes. (Note
that these methods do not include the all-important task of testing the
model's formulation; they only test whether the software faithfully
implements the formulation.)

Code reviews. Our programmer writes the software source code, then the
modeler checks the code thoroughly. The primary purpose of this check is
to ensure that the programmer understood the model formulation
correctly; another benefit is encouraging the programmer to write
understandable and well-documented code.

Spot checks using probes. The Swarm animation window and probes can be
used to display the input and output of key model subcomponents so they
can be hand-checked. This helps find code errors that have widespread
effects. However, spot-checks are not adequate by themselves because
they cannot test all the model's possible combinations of variables and
conditions.

Pattern tests. We next run a model and observe patterns of behavior with
the animation window. Unrealistic behavior often quickly leads to the
identification of a mistake in the code or formulation. 

Systematic tests against an independent implementation. Because an IBM
typically has many potential combinations of input variables, control
paths, and intermediate results, the only way to test an IBM thoroughly
is by implementing it independently in two codes and looking for
differences in the output of the two codes. For models that critical
management decisions will be made with, it may well be worth having two
programmers completely code the model in two languages. We program our
models' major components in spreadsheets and compare spreadsheet
calculations to intermediate results from the Swarm code. For example,
we run the model over a wide range of flows and temperatures and have
the Swarm code print out, for each fish on each time step, the factors
that affect its survival probabilities. In a spreadsheet, we
independently encode the survival methods then import the Swarm output
for direct comparison. This approach is easy and inexpensive for us
because the spreadsheet already exists: we use it to design the model
before writing its Swarm code.

-- 
address@hidden
Lang, Railsback & Assoc.
250 California Ave., Arcata CA 95521
707-822-0453; Fax 822-1868


                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================