Skip to content

WIP: Invoke HBT halo finder on the fly

John Helly requested to merge hbt_otf into master

This merge request allows running the HBT halo finder/merger tree builder on the fly in Swift simulations. It requires a modified version of HBT which provides a library interface similar to the one used for VELOCIraptor - see https://github.com/jchelly/HBTplus/tree/on-the-fly.

Compiling

HBT must be compiled using cmake to generate the library interface:

cd HBTplus
mkdir build
cd build
cmake .. -DHBT_INT8=ON -DHBT_REAL8=ON -DHBT_INPUT_INT8=ON
make

This produces libHBT.so and libHBT.h in the build directory. To compile Swift:

../configure \
    --enable-fof \
    --with-hbt=/path/to/hbt/HBTplus/build/

HBT requires FoF groups as input, so Swift must be compiled with FoF enabled.

Swift parameters

To run with HBT, the Swift .yml file must contain FoF and HBT sections:

FOF:
  basename:                        fof_output  # Filename for the FOF outputs.
  min_group_size:                  20          # The minimum no. of particles required for a group.
  linking_length_ratio:            0.2         # Linking length in units of the main inter-particle separation.
  seed_black_holes_enabled:        0           # Enable seeding of black holes in FoF groups

HBT:
  config_file_name:     ../HBT_SmallCosmoVol.conf   # HBT config file
  basename:             ./hbt/                      # Where to write output
  scale_factor_first:   0.1                         # Scale-factor of the first output
  delta_time:           1.1                         # Delta log-a between outputs
  output_list_on:       1                           # Whether to use output list file
  output_list:          hbt_output_list.txt         # Filename for output list
  keep_subhalos_in_memory: 0                        # Whether to keep subhalos in memory between invocations
                                                    # (0=off to save memory, 1=on to reduce I/O)
Snapshots:
  invoke_hbt:          1
  ...

and Swift must be run with the --fof and --hbt command line flags.

Several parameters in the HBT input file are ignored because the correct value is passed in from Swift:

  • SubhaloPath
  • omega_m0, omega_lambda0
  • BoxSize
  • Units: MassInMsunh, LengthInMpch, VelInKmS

Currently the softening is NOT passed in. In fact, we can only specify a single, fixed comoving softening for all particles in the input file. Would be better to set this automatically.

Restarting

HBT needs the particle subhalo membership information from the previous snapshot to process the next one. Fortunately, it can read its own output so the outputs serve as restart files. Restarting a Swift+HBT run works as long as we don't delete the last HBT output before the restart file.

A possible enhancement would be to have HBT able to serialize the subhalo data to Swift's restart files. We could generate the halo catalogue and merger trees without ever writing any particle data to disk, except for the contents of the restart files. This would be useful if we want merger trees with good temporal resolution but we don't have much disk space for snapshots. Only works if we're keeping the subhalo data in memory between HBT calls.

If we don't want to keep the subhalo data in memory and we also don't want to keep all the snapshots on disk, then I think we would need to keep the most recent HBT output and the last output prior to each set of restart files.

Edited by Matthieu Schaller

Merge request reports