Re: [Swarm-Modelling] SWARM on Clusters and "Models in the Wild"

[Darren Schreiber]

The mail-list for Swarm has been exciting in a way that it hasn't been 
in quite a while, and I regret I haven't been able to jump in earlier 
since these issues are so near and dear to my heart.  When I didn't 
know much about Swarm or agent-based modeling, I used to think how cool 
fine-grained parallelism would be.  But more and more, as I think of 
what I need to satisfy myself that the modeling I and others am doing 
is not "dodgy," I am convinced that a toolkit for "model evaluation" is 
needed.  And, part of that toolkit requires coarse-grained parallelism 
(perhaps very coarse grained would even be sufficient.)

I am quite convinced that the biggest thing the ABM community needs now 
is to develop some coherency with regards to model evaluation.  I use 
the term model evaluation, because after giving a paper on "Validation" 
at the Lake Arrowhead conference a few years ago, Steve Bankes 
convinced me completely that "validation" and "verification" had far 
too much baggage to be useful constructs for our enterprise.  In that 
paper, I describe a four part ontology (theory/model/phenomena/noumena) 
and a list of "validation" tools developed in the engineering 
literature.  I then argue that we could conceptualize those validation 
tools as evaluations in relation to the ontology (i.e. theory-model 
tests, model-model tests, model phenomena tests, and 
theory-model-phenomena tests (aka robustness.))

My vision for ABM is to have a toolset that would be something akin to 
a Drone on steroids, that would easily allow a modeler to put the model 
through a somewhat standardized set of evaluation procedures.  You 
could plug-in your model and get statistical analyses and visualization 
of what happened to your variables of interest as you randomized the 
initial conditions, varied the parameter settings, or changed the 
algorithms.  There might be a topology tool that would allow you to 
shift your agents from interacting on moore neighborhood, to a 
von-neuman, to a hex-grid, to a 3D grid, to a random network, to a 
soup, to a small-works network, etc.  There would be an easy to use 
"docking-clamp" that would make it pretty easy to run my version of 
Schelling's model against Paul's and see statistics on the differences. 
 I'd also have some tools that would facilitate tracing the life of a 
particular agent in the model and categorizing the decisions it made 
and the environment it was acting in.  I would want tools that would 
make it easy for me to show what assumptions I've made and what their 
consequences were.

Going a bit further in this dream, perhaps it would have some HTML 
writing code in it.  So, when I have run battery of tests, I get an 
"Evaluation Summary" page that describes the tools that were used and 
the results.  Then I could click on hyper-links that would lead to the 
results of my theory-model tests.  On that page, I might have a link to 
"narrative validity" test page.  Prior to running the model, I could 
have input text from some leading authorities who gave narrative 
accounts about how changes in my independent variables would impact my 
dependent variables.  And, on the narrative validity page, I would have 
a list of those quotes and sources alongside of graphs showing the 
actual behavior of the model.  Or, from the "Evaluation Summary" page, 
I would click on the "Model-Model" tests page, then click on the 
"Docking" tests to see comparative statistics that show how my model is 
similar to and different than Paul's.  Or, I could click on the 
"Analytic validity" tests page where the results of the formal proof I 
had input earlier are displayed side by side with the relevant output 
of my model.  Or, I could click on the "Validating Substructures" link 
that might take me to a description of my agent-class and some data 
about how an individual agent responds to different inputs and outputs.

My goal here would be the kind of transparency that formal proofs or 
good statistics offers.  The brain imaging analysis software I am using 
right now provides a similar facility to the one I am describing.  
Brain imaging analysis has many-many moving parts and a sophisticated 
consumer of the data wants to be able to see the raw structural MRI of 
the brain, how the brain tissue data was extracted, how the individual 
brains were warped onto to the common space, the time course of data 
for the statistically significant voxels, and a bunch of other stuff.  
The FSL people (http://www.fmrib.ox.ac.uk/fsl/) have made this easy 
with a toolkit that outputs a series of html pages containing the 
settings for the model and the outputs.  They even created a nice text 
generation tool that writes a paragraph summarizing what you did that 
could be cut and paste right into a manuscript (cites and all!)

In general, it isn't very easy to figure out what other people are 
doing with their modeling or with their testing.  In my little fantasy 
world, I imagine reading an article and then seeing a link to their 
"Evaluation Summary" webpage that contains copies of their code along 
with a variety of tests and results.  Since this is all very 
computationally intensive, I'd want the Evaluation tool to be able to 
send the model off to different processors (via a grid, ftp/ssh, 
cluster, or whatever) and get back the results.

None of what I have described requires miracles of programming.  But, 
it would certainly require some coordinated effort, vision, and 
probably funding.  I do think it would help me sleep better at night in 
believing my modeling and the modeling of others.  And, I believe it 
would provide the kind of "ABM advocacy" that no amount of persuasive 
theorizing could produce.

        Darren



On Jan 29, 2004, at 5:53 PM, address@hidden wrote:

> The issue is:  What do people do with Swarm and what do they _want_ to
> do with it that they currently can't or find hard?
>
> A coarse-grained parallelism is reasonable and you'll see an almost
> linear speedup using it.  For example, if you have a model that you
> have to run 10 times to explore the model's parameter space, then
> having a 10-node cluster makes that effort almost 10 times as fast.
>
> Forget the fine-grained stuff like managing event lists and think
> asynchronous monte-carlo.  After all, if you wanted a system for doing
> "real" parallel computation, you wouldn't be using a prototyping tool
> like Swarm, anyway.
>
> If the conversation about "start small and move steadily forward"
> means anything, it's referring to things like this.
>
> Assuming we had infrastructure for passing whole simulations out
> (grid-like) to many different nodes, then we could begin talking
> about where to cut the scheduling mechanism to make multiple,
> mostly unrelated, swarms talk to each other.
>
> Andy Cleary writes:
> > You just can't do any reasonable level of parallelism without taking 
> > a more
> > explicit "partitioned/domain decomposed" approach, and for that a new
> > generation of ABM infrastructure is needed...
>
> It's true that you have to [de]compose things at the domain level.
> But, I don't think it's true that you need a whole new infrastructure.
> It's more likely that all we need is a new piece of infrastructure.
>
> Perhaps as we develop ABM patterns, some attempts could be made to
> describe this new piece so that we can begin building it.
>
> --
> glen e. p. ropella              =><=                           Hail 
> Eris!
> H: 503.630.4505                              
> http://www.ropella.net/~gepr
> M: 971.219.3846                               
> http://www.tempusdictum.com
>
> _______________________________________________
> Modelling mailing list
> address@hidden
> http://www.swarm.org/mailman/listinfo/modelling