[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [h5md-user] Dataset layouts
From: |
Pierre de Buyl |
Subject: |
Re: [h5md-user] Dataset layouts |
Date: |
Tue, 9 Jun 2015 22:30:31 +0200 |
User-agent: |
Mutt/1.5.23 (2014-03-12) |
Hi Peter,
On Tue, Jun 09, 2015 at 04:15:06PM -0400, Peter Colberg wrote:
> While working on a truly large-scale scientific application using
> the HDF5 MPI I/O driver, I noticed it would be useful to mandate that
> “meta-datasets” are always stored with compact layout. This would
> apply to proposal 100 (scalar step/time), and `box/edges` for the
> fixed-geometry case.
>
> When using parallel HDF5, most C API calls are collective, which means
> that a function must be called by all processes in the communicator
> used to access a file. Compact datasets are metadata and therefore
> must be written/read collectively. This goes nicely with the other
> metadata stored in attributes.
I understand the motivation. Did you notice a difference in performance or is
there some other motivation?
> In similar spirit, the specification states that time-dependent
> datasets have an infinite first dimension. I suggest we specify
> explicitly that these datasets must have a chunked layout.
This should not be controversial :-) One cannot make an extensible dataset
without chunking.
Cheers,
Pierre
- [h5md-user] Dataset layouts, Peter Colberg, 2015/06/09
- Re: [h5md-user] Dataset layouts,
Pierre de Buyl <=
- Re: [h5md-user] Dataset layouts, Peter Colberg, 2015/06/09
- Re: [h5md-user] Dataset layouts, Peter Colberg, 2015/06/09
- Re: [h5md-user] Dataset layouts, Pierre de Buyl, 2015/06/10
- Re: [h5md-user] Dataset layouts, Pierre de Buyl, 2015/06/15
- Re: [h5md-user] Dataset layouts, Felix Höfling, 2015/06/16
- Re: [h5md-user] Dataset layouts, Peter Colberg, 2015/06/24