Skip to content

RT Sub-cycling

Mladen Ivkovic requested to merge rt_time_steps_mladen_cleanup into master

This is a first draft version of the RT subcycling.

So far, the debug RT scheme without MPI seems to work without a hitch. The MPI version still needs verification, but I'm optimistic due to recent updates. However, I think it's come far enough for a first look of what's where and how things are done.

Noteable changes:

  • a new parameter in the parameter file, TimeIntegration:max_nr_rt_subcycles determines the maximal number of subcycles per hydro step. HOWEVER, currently this sets the exact number of subcycles we do, not an upper threshold.
  • moved all RT related cell variables into its own "rt" struct and "cell_rt.h" file
  • cells have new RT integertimes that are used to determine whether a cell should be active or not.
  • added new "cell_is_rt_active" and "part_is_rt_active" functions
  • the sub-cycling basically works like normal timestepping, but with "extras". To that end, there are new variables in the engine to keep track of the time during subcycles (current time, min/max active bins)
  • the subcycling itself never touches anything timestep related that other tasks make use of. It never changes ti_current, min_active_bin, max_active_bin etc.
  • there are several new tasks:
    • "rt advance cell time": does (part of) what the timestep tasks already do for all other phyics: advances (recursively) the ti_rt_end_min during subcycles. Particle bins are not affected.
    • "rt collect times": does (part of) what the time collect tasks already do for all other physics: collects the ti_rt_end min from super cells up to the top level. It's only activated during sub-cycles.
    • "RT sorts": used on foreign cells in case a cell needs to be sorted (due to drifts from gravity, starts, ...) but there is no hydro sort (because the cell itself and none of its neighbours are hydro active at that step). These tasks run only during main steps, never during subcycles. [but a sub-cycle step can now be executed during a main step as well, with everything except RT being inactive]
  • the there are 3 new cell flags for the RT sort.
    • cell_flag_do_rt_sub_sort: do we need an RT sub sort? I can't use the hydro flag there, because the hydro flag is also used to early-exit walking up the cell hierarchy when activating hydro subcell sorts
    • cell_flag_do_rt_sort: was this sort requested by an RT sort? If so, allow cell to be undrifted instead of crashing
    • cell_flag_skip_rt_sort: If a hydro sort has also been activated in this main step, don't run an RT sort.

ToDo's left:

  • compile check all RT schemes
  • dist-check
  • transfer updated debugging checks to GEAR-RT
  • update RT sanity checks py scripts
  • re-check run.sh and .yaml files in RT examples are still sensible
  • update debugging check in cell_is_rt_active
  • fix checks in runner_do_rt_advance_cell_time
  • fix collection of number of RT updates each subcycle. Maybe move into runner_do_collect_rt_times?
  • add rigorous drift checks, if possible
  • check debugging functions in engine.c, are they still necessary?
  • try early-exiting unnecessary RT sorts by just setting the correct flags within runner_do_sort_hydro. [eagle 6 with 3 ranks and 8 subcycles catches error.]
  • check task timers
  • add runtime dependency checks for collect_rt_times, make sure they don't run in main steps
  • add documentation

Implements #769 (closed).

Edited by Matthieu Schaller

Merge request reports