Large parallel hdf5
Fix to #72 (closed).
This is a workaround to the limitation of parallel-HDF5. The low-level MPI-IO implementations limit writes to 2GB per rank (irrespective of the total amount being written across all nodes).
The solution involves writing chunks of 2GB (or in practice 2'000'000'000Bytes) and then repeat for the remaining chunks, if any, by shifting the position to write of each node in the file and in memory by 2GB. Ranks that did not pass the threshold just write nothing. In realistic scenarios we won't need more than a handful of iterations.
Merge request reports
Merged by
(May 28, 2025 2:56pm UTC)
Activity
Milestone changed to Scalable i/o (HDF5) that is fast and can run on ~100 nodes.
Reassigned to @pdraper
Managed to build and then run an
EAGLE_50job (5 nodes of COSMA4, nodes are paging a little), so looks better than what we had (master crashes with the same job).If this is ready for acceptance, then the only issue I can see is that the mean of the coordinates is different between the ICs and the dump of them. For the ICs I get
19.0558301+/-10.1132597and the dump18.2556886+/-10.2318977. All the other values have the same means.
Ok, I can reproduce this but I get
19.05583 +/- 10.39808for the snapshot when using 4 ranks. So clearly some interesting bug here. Back to the drawing board.Edited by Matthieu SchallerAdded 1 commit:
- 752eb874 - When writing a second 2GB chunk, consider the next chunk in memory and not the same one again...
Added 1 commit:
- ff7f35ee - Code formatting
-
Ok, that is now fixed. I missed one shift of 2GB leading to me writing the same chunk many times one after the other.
Also, it did not show up with the 100Mpc volume as the average of the coordinates was close enough to the ICs' value and I did not print enough digits to notice that it was wrong.
- Last reply by Peter W. Draper
297 h_err = H5Sset_extent_simple(h_filespace, rank, shape_total, NULL); 359 hid_t h_err = H5Sset_extent_simple(h_filespace, rank, shape_total, NULL); 298 360 if (h_err < 0) { 299 361 error("Error while changing data space (file) shape for field '%s'.", 300 362 props.name); 301 363 } 302 364 365 /* Dataset properties */ 366 const hid_t h_prop = H5Pcreate(H5P_DATASET_CREATE); 367 368 /* Set chunk size */ 369 /* h_err = H5Pset_chunk(h_prop, rank, chunk_shape); */ 370 /* if (h_err < 0) { */ 371 /* error("Error while setting chunk size (%llu, %llu) for field '%s'.", */ 372 /* chunk_shape[0], chunk_shape[1], props.name); */ 373 /* } */ - Last reply by Matthieu Schaller
297 h_err = H5Sset_extent_simple(h_filespace, rank, shape_total, NULL); 359 hid_t h_err = H5Sset_extent_simple(h_filespace, rank, shape_total, NULL); 298 360 if (h_err < 0) { 299 361 error("Error while changing data space (file) shape for field '%s'.", 300 362 props.name); 301 363 } 302 364 365 /* Dataset properties */ 366 const hid_t h_prop = H5Pcreate(H5P_DATASET_CREATE); 367 368 /* Set chunk size */ 369 /* h_err = H5Pset_chunk(h_prop, rank, chunk_shape); */ 370 /* if (h_err < 0) { */ 371 /* error("Error while setting chunk size (%llu, %llu) for field '%s'.", */ 372 /* chunk_shape[0], chunk_shape[1], props.name); */ 373 /* } */ Added 1 commit:
- fb2c10ac - Use LL instead of ll for the long long suffix of literals.
mentioned in commit f8efb3a4
mentioned in issue #72 (closed)