Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • SWIFTsim SWIFTsim
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 57
    • Issues 57
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 21
    • Merge requests 21
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • SWIFT
  • SWIFTsimSWIFTsim
  • Merge requests
  • !438

Large parallel hdf5

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Matthieu Schaller requested to merge large_parallel_hdf5 into master Oct 03, 2017
  • Overview 10
  • Commits 6
  • Changes 1

Fix to #72 (closed).

This is a workaround to the limitation of parallel-HDF5. The low-level MPI-IO implementations limit writes to 2GB per rank (irrespective of the total amount being written across all nodes).

The solution involves writing chunks of 2GB (or in practice 2'000'000'000Bytes) and then repeat for the remaining chunks, if any, by shifting the position to write of each node in the file and in memory by 2GB. Ranks that did not pass the threshold just write nothing. In realistic scenarios we won't need more than a handful of iterations.

Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: large_parallel_hdf5