Re: ASM Patch 3 nearing completion; hdf5 question

[pauljohn]

"Marcus G. Daniels" wrote:
> 
> R can do these kind of manipulations in a direct way.  No need for an
> extra file.  For example, suppose you had two text or HDF5 files
> output by Swarm:
> 
> $ cat file1.txt
> 10 20 30
> 40 50 60
> 
> $ cat file2.txt
> 100 200 300
> 400 500 600
> 
> Then you could load the program below in order to collect the last
> records and do statistics on them.
> 
> $ cat demo.R
> lastrecords <- data.frame()
> 
> for(fi in 1:2) {
>   base <- paste("file", fi, sep="")
> 
>   txt <- paste(base, ".txt", sep="")
>   m <- read.table(txt)
> 
>   #hdf <- paste(base, ".hdf", sep="")
>   #m <- hdf5load(hdf),load=FALSE)$m
> 
>   lastrecords <- rbind(lastrecords,m[nrow(m),])
> }
> print(mean(lastrecords$V2))
> 
>  (Text files having
> floating point numbers don't record specifics about the binary
> representation scheme for those numbers.)
Now this last point I finally understand! Go hdf5!

I don't think we are disagreeing about R or the management of data files
written in text.  Here is what I can't get my brain around--with 500
runs of thousands of timesteps each, the hdf5 output seems unmanageable.
WHen I write out a data file in text form, it has the step by step
output, you know like

Run Time X  Y   (....imagine many more columns here)
1    0   21 12
1    1   33  31
1    2   55  22

and I output for each run a separate file, and so I can make a time
series picture of each run, or I can grab the last lines and see a cross
section.  I use read.tables with the header option, I get an automagical
data frame.

My inability to understand how you use R/hdf5 probably traces back to
the fact that I first learned to code and enter data in the IBM
punch-card era when data was a strictly rectangular thing, with one or
more record per observation.

With hdf5 output from a swarm program, I hdf5load the file in R, and
I've got many objects, one for each time step.  The keys I chose have
the timestep as part of the key, so I see objects with names that don't
even sort properly, as in

model0
model1
model10
model11
model12
model13
model14
model15
model16
model17
model18
model19
model2
model21
model22

and so forth. If I need to open 500 hdf files to grab results of the
last step for each one, I can't even figure how to close all the
unneeded R objects.  These are R things I can figure out, I think, if I
can just convince myself I've not made some massive blunder to get to
this spot.

Is there a typo in your commented-out line:
>   #m <- hdf5load(hdf),load=FALSE)$m
?
-- 
Paul E. Johnson                       email: address@hidden
Dept. of Political Science            http://lark.cc.ukans.edu/~pauljohn
University of Kansas                  Office: (785) 864-9086
Lawrence, Kansas 66045                FAX: (785) 864-5700


                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================