Explore HDF5's new subfiling VFD interface

From version 1.13.2 hdf5 proposes a new interface allowing to split data over multiple files. See https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.13/hdf5-1.13.2/src/hdf5-1.13.2-RELEASE.txt

Their promise:

      By allowing a configurable stripe size, number of I/O concentrators and
      method for selecting MPI ranks as I/O concentrators, the Subfiling VFD
      aims to enable an HDF5 application to find a middle ground between the
      single shared file and file-per-process approaches to parallel file I/O
      for the particular machine the application is running on. In general, the
      goal is to avoid some of the complexity of the file-per-process approach
      while also minimizing the locking issues of the single shared file approach
      on a parallel file system.

This sounds like something halfway between our current two i/o strategies and could be more generic and more efficient.

One for the rainy days...

Assignee Loading
Time tracking Loading