Add option to write each VR output to a separate directory

When running Velociraptor on the fly with many MPI ranks it generates a very large number of output files and makes an unreasonably large directory. This merge request adds a parameter which makes the code create a new directory for each output. If StructureFinding:subdir_per_output is specified then VR output files will have names like

${subdir_per_output}_XXXX/stf\_$

{basename}_XXXX.*.

If we prepend to the output path like this then we can't include a directory in Snapshots:basename (this already didn't work in some cases because the code prepends 'stf_' to VR filenames). I've added a parameter Snapshots:subdir which can be used to put all snapshot and velociraptor output in the specified directory.

If neither parameter is specified then all files are written in the same locations as before.

Pascal is working on a version of Velociraptor that uses parallel HDF5, so maybe this isn't necessary.

Let's see what comes out of Pascal's current updates indeed.

If I am not mistaken I think @pdraper wasn't a big fan of creating new directories from inside the code but I can't remember why...

We do need to do something about this before we can run the 300Mpc EAGLE-XL box because right now it will try to write (768 ranks) * (~140 VR outputs) * (~6 file types) = 645,120 files to a single output directory. Hopefully we can just write far fewer files per output.

Reminds me of the good old Gadget group files.

Indeed, we should do something. That's going to be a problem. Even with many directories we might run out of inodes. Do we have the option to merge the files by hand after they have been dumped?

Looks like we have 2M inodes on the /scratch file system so we could run like this, but on /store we only have 100k so we would need to merge files if we wanted to copy the data there. We could merge the files in a quick, simple way by making each one a group in one big HDF5 file.

added 236 commits

• 58e55f4d...ad815e94 - 225 commits from branch master
• 39619900 - Add option to make a subdirectory for each stf output
• 8ab4e978 - Added optional parameter to set output directory
• 8bb718ce - Changed filename buffer length and add missing mkdir() call
• 724bad2e - Don't segfault if XMF file can't be created
• 3bd6f21d - Changed mode parameter in mkdir() for output directory
• 00719ca5 - Changed snapshot directory parameter for consistency with Restarts section
• f4235fc5 - Added new parameters to docs
• 33fd9ee7 - Changed VELOCIraptor example params in the documentation
• 8717693c - Fix default values for subdir and subdir_per_output in docs
• b9648e5e - Correct VR filenames when subdir is set but subdir_per_output is not
• cf628cf2 - Fix filename bug introduced when rebasing

Compare with previous version

@jch are you using this in your recent Paris run?

Yes. Velociraptor is using parallel hdf5 but there are still multiple files per output and I wanted to put the output on a different file system from the code, scripts etc.

assigned to @pdraper

added 225 commits

• cf628cf2...220dbc37 - 223 commits from branch master
• f034ad52 - Merge branch 'master' into stf_output_dirs
• bd7b05e2 - Formatting

Compare with previous version

All looks mainly OK to me, except we don't have the descriptions added to the parameter_example.yml file. I'm tempted to have non-NULL defaults for at least subdir, but wouldn't object to the same for the VR output, if that makes sense.

Agree with both of these. Non-NULL directory names are more issue-proof I'd think.

Shall I do the changes?

Sounds good to me. The null defaults were just to ensure it generates the same paths as before if you don't use the new parameters, but that does dump a lot of files in the current directory.

Thanks John.

added 1 commit

• 1a29391b - Removed debugging printf() and added new parameters to the parameter example file.

Compare with previous version

resolved all discussions

merged

mentioned in commit 8dafa3fb