Skip to content

WIP: Allow compression in snapshots written in parallel

John Helly requested to merge parallel_compression into master

This adds a new parameter Snapshots:mpi_compression which can be enabled to write compressed snapshots in parallel. It only takes effect for HDF5 1.10.2 or later. It's a separate parameter because at least in HDF5 1.10.2 the feature is labelled as 'experimental', so we probably don't want to use it by default.

I also found that swift was always writing snapshots in independent mode, which is incompatible with compression. This looks like a mistake because there was a call to set H5FD_MPIO_COLLECTIVE on a property list that was never used in writeArray(). I've modified the code to always use collective mode for parallel writes. It was already using collective mode when reading snapshots.

Still need to investigate what effect this has on performance. It might be quite poor because all the data from one MPI rank is compressed by a single thread in the H5Dwrite call.

Merge request reports