Re: [h5md-user] Varying particle number

[Peter Colberg]

On Tue, Oct 07, 2014 at 09:35:57AM +0200, address@hidden wrote:
> Before going further with "particle_number", did you test your situation with 
> a
> compressed "id" element?

No, the particles are indistiguishable, so there is not id. The
compression should reduce the size dramatically; for a 64-bit “step”
dataset and a chunk size of 32 kB the compression ratio can be >100.

> Alternatively, a better compression could be achieved if we decided to allow
> "species" to contain also a fill value. This way, you'd compress a dataset of
> fill_value, 0, 1 (for two species, for instance) and that should be more
> efficient.

That is a good idea. We should add this as a minimal solution.

In any case, I am not concerned with the efficiency of storage, but
with the convenience of reading the particle data. It would be nice if
one would not have to loop over species/id to find non-empty regions.
Instead one reads the particle number, checks that both species and id
(if present) do not define a fill value (= no holes), and then selects
a single hyperslab for reading a particle dataset.

Peter