Skip to content

Add a virtual meta-file for the distributed snapshots

Matthieu Schaller requested to merge virtual_snap into master

These code changes introduce a meta-file for the distributed snapshots using hdf5's virtual layer. This allows to transparently have the data spread over multiple files but have a master file where the data looks like all the data is just in a single array. All the heavy lifting is done under the hood by hdf5. This is based on the suggestion #668 (closed) by @jch.

For instance, with 4 files, we have:

eagle_0000.0.hdf5
eagle_0000.1.hdf5
eagle_0000.2.hdf5
eagle_0000.3.hdf5
eagle_0000.hdf5    <------ Meta-file including info to transparently read the data in the other four files as if it were in a single dataset

This allows for the distributed i/o to produce data that works with swiftsimio as well as the xmf format used by ParaView.

The cell meta-data in the virtual file is written as if the data was all in a single file, making the whole mechanism transparent to the user. The individual files contain their own cell meta-data, which swiftsimio is not yet able to exploit.

Implements #668 (closed) and #667 (closed). Also fix a bug with the xmf file not written correctly when a subdir is used.

Merge request reports