[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [h5md-user] Variable-size particle groups

From: Peter Colberg
Subject: Re: [h5md-user] Variable-size particle groups
Date: Tue, 29 May 2012 11:43:17 -0400
User-agent: Mutt/1.5.21 (2010-09-15)

On Tue, May 29, 2012 at 05:32:48PM +0200, Olaf Lenz wrote:
> Hi!
> On 05/29/2012 05:17 PM, Peter Colberg wrote:
> > At any given timestep, each processor owns a defined set of 
> > particles of varying number, which shall be written to an HDF5 
> > dataset. The task that needs to be solved is to calculate for each 
> > process the offset from the beginning of the “range” for the
> > current time-step.
> > 
> > This could be done in parallel using a prefix sum [1].
> Nah, the problem is not just knowing where to write in the file, but
> that it may not change throughout the process.
> The idea of parallel IO (with a distributed file system) is that a
> single file is sliced up over several hard disks, so that each task just
> has to write to its local disk. Otherwise, parallel IO wouldn't really
> help, as the bottleneck is the IO speed of the hard disks.

Oh, now I see what you mean with parallel I/O.

I was assuming a different definition of parallel I/O. At SciNet in
Toronto, they provide a parallel filesystem (GPFS), which allows
writing data in parallel from thousands of processes to a *single*
file. I believe this is the scenario that parallel HDF5 implements.

If each process writes to a separate file on a local disk, than the
range dataset would nevertheless work fine, would it not? Now you
would actually make use of both entries of range [first, last), and
ranges would be local to the process. After the simulation, the
files need to be pieced together manually, if desired.

Maybe I am missing the point, and there is a third definition ;-).


reply via email to

[Prev in Thread] Current Thread [Next in Thread]