[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [h5md-user] Variable-size particle groups

From: Olaf Lenz
Subject: Re: [h5md-user] Variable-size particle groups
Date: Tue, 29 May 2012 21:06:07 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20120502 Thunderbird/12.0.1


On 05/29/2012 05:43 PM, Peter Colberg wrote:
The idea of parallel IO (with a distributed file system) is that a
 single file is sliced up over several hard disks, so that each
task just has to write to its local disk. Otherwise, parallel IO
wouldn't really help, as the bottleneck is the IO speed of the
hard disks.

Oh, now I see what you mean with parallel I/O.

I was assuming a different definition of parallel I/O. At SciNet in
Toronto, they provide a parallel filesystem (GPFS), which allows
writing data in parallel from thousands of processes to a *single*
file. I believe this is the scenario that parallel HDF5 implements.

Yes, I also mean writing in parallel to a single file on a parallel file
system. To be able to do that, the single file has to be somehow
distributed over several disks, so that it is actually possible to write
to multiple disks at once. And this is really effective only when the
file has regular sized chunks on each disk.

I must admit, however, that I do not really have much practical
experience with parallel IO, mostly because so far the people I talked
to told me that it is only worth it when the file is split up regularly.
As soon as you start writing the file in a less regular fashion, the
advantage of parallel IO is mostly gone.

Maybe this is not true for modern PFS anymore. If that is so, I would be
happy to learn about it.

Dr. rer. nat. Olaf Lenz
Institut für Computerphysik, Pfaffenwaldring 27, D-70569 Stuttgart
Phone: +49-711-685-63607

Attachment: olenz.vcf
Description: Vcard

reply via email to

[Prev in Thread] Current Thread [Next in Thread]