Explore HDF5's new subfiling VFD interface
From version 1.13.2 hdf5 proposes a new interface allowing to split data over multiple files. See https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.13/hdf5-1.13.2/src/hdf5-1.13.2-RELEASE.txt
Their promise:
By allowing a configurable stripe size, number of I/O concentrators and
method for selecting MPI ranks as I/O concentrators, the Subfiling VFD
aims to enable an HDF5 application to find a middle ground between the
single shared file and file-per-process approaches to parallel file I/O
for the particular machine the application is running on. In general, the
goal is to avoid some of the complexity of the file-per-process approach
while also minimizing the locking issues of the single shared file approach
on a parallel file system.
This sounds like something halfway between our current two i/o strategies and could be more generic and more efficient.
One for the rainy days...