Task list overflow / Link table overflow when running with self-gravity
I'm trying to implement a new hydro test that needs self-gravity. However, I run into problems with the scheduling system when I try to run it.
The easiest way to reproduce the problem is by running the 3D Sedov blast with self-gravity. Add the following lines to sedov.yml:
Gravity:
eta: 0.025
epsilon: 0.001
theta: 0.1and run with
../swift -G -s -t 4 sedov.ymlOn my system, this crashes with the following error:
bv7@apsient:/data/bv7/swiftsim/examples/SedovBlast_3D$ ../swift -G -s -t 4 sedov.yml
Welcome to the cosmological hydrodynamical code
______ _________________
/ ___/ | / / _/ ___/_ __/
\__ \| | /| / // // /_ / /
___/ /| |/ |/ // // __/ / /
/____/ |__/|__/___/_/ /_/
SPH With Inter-dependent Fine-grained Tasking
Version : 0.6.0
Revision: v0.6.0-616-g602118e3, Branch: evrard, Date: 2017-10-31 12:08:37 +0000
Webpage : www.swiftsim.com
Config. options: '--disable-vec --with-hydro=gadget2 --with-riemann-solver=exact --with-hydro-dimension=3 --with-ext-potential=none --with-equation-of-state=ideal-gas --enable-mpi=no --enable-debug=no --enable-debugging-checks=no --enable-optimization=yes --enable-sanitizer=no --no-create --no-recursion'
Compiler: GCC, Version: 4.8.4
CFLAGS : '-O3 -fomit-frame-pointer -malign-double -fstrict-aliasing -ffast-math -funroll-loops -march=bdver2 -mxop -fno-tree-vectorize -pthread -Wall -Wextra -Wno-unused-parameter -Wshadow -Werror'
HDF5 library version: 1.8.11
[00000.0] main: CPU frequency used for tick conversion: 2300084016 Hz
[00000.0] main: Running on: apsient
[00000.0] main: sizeof(part) is 128 bytes.
[00000.0] main: sizeof(xpart) is 64 bytes.
[00000.0] main: sizeof(spart) is 64 bytes.
[00000.0] main: sizeof(gpart) is 96 bytes.
[00000.0] main: sizeof(multipole) is 236 bytes.
[00000.0] main: sizeof(grav_tensor) is 228 bytes.
[00000.0] main: sizeof(task) is 64 bytes.
[00000.0] main: sizeof(cell) is 640 bytes.
[00000.0] main: Reading runtime parameters from file 'sedov.yml'
[00000.0] main: Reading ICs from file './sedov.hdf5'
[00000.0] io_read_unit_system: Reading IC units from ICs.
[00000.0] read_ic_single: IC and internal units match. No conversion needed.
[00000.2] main: Reading initial conditions took 219.869 ms.
[00000.2] main: Read 262144 gas particles, 0 star particles and 262144 gparts from the ICs.
[00000.2] space_init: Re-scaling smoothing lengths by a factor 3.330000e+00
[00000.3] main: space_init took 48.214 ms.
[00000.3] main: space dimensions are [ 1.000 1.000 1.000 ].
[00000.3] main: space is periodic.
[00000.3] main: highest-level cell dimensions are [ 12 12 12 ].
[00000.3] main: 262144 parts in 1728 cells.
[00000.3] main: 262144 gparts in 1728 cells.
[00000.3] main: 0 sparts in 1728 cells.
[00000.3] main: maximum depth is 0.
[00000.3] engine_init: no processor affinity used
[00000.3] engine_policy: engine policies are [ steal keep numa_affinity hydro self_gravity ]
[00000.3] hydro_props_print: Equation of state: Ideal gas.
[00000.3] hydro_props_print: Adiabatic index gamma: 1.666667.
[00000.3] hydro_props_print: Hydrodynamic scheme: Gadget-2 version of SPH (Springel 2005) in 3D.
[00000.3] hydro_props_print: Hydrodynamic kernel: Cubic spline (M4) with eta=1.234800 (48.00 neighbours).
[00000.3] hydro_props_print: Hydrodynamic relative tolerance in h: 0.00010 (+/- 0.0144 neighbours).
[00000.3] hydro_props_print: Hydrodynamic integration: CFL parameter: 0.1000.
[00000.3] hydro_props_print: Hydrodynamic integration: Max change of volume: 1.40 (max|dlog(h)/dt|=0.112157).
[00000.3] gravity_props_print: Self-gravity scheme: FMM-MM with m-poles of order 5
[00000.3] gravity_props_print: Self-gravity time integration: eta=0.0250
[00000.3] gravity_props_print: Self-gravity opening angle: theta=0.1000
[00000.3] gravity_props_print: Self-gravity softening: epsilon=0.0030 (Plummer equivalent: 0.0010)
[00000.3] gravity_props_print: Self-gravity mesh smoothing-scale: a_smooth=1.250000
[00000.3] gravity_props_print: Self-gravity tree cut-off: r_cut_max=4.500000
[00000.3] gravity_props_print: Self-gravity truncation cut-off: r_cut_min=0.100000
[00000.3] engine_init: Absolute minimal timestep size: 3.469447e-19
[00000.3] engine_init: Minimal timestep size (on time-line): 9.536743e-08
[00000.3] engine_init: Maximal timestep size (on time-line): 9.765625e-05
[00000.3] main: engine_init took 1.251 ms.
[00000.3] main: Running on 262144 gas particles, 0 star particles and 0 DM particles (262144 gravity particles)
[00000.3] main: from t=0.000e+00 until t=5.000e-02 with 4 threads and 4 queues (dt_min=1.000e-07, dt_max=1.000e-04)...
[00000.3] engine_init_particles: Computing initial gas densities.
[00000.4] scheduler.c:scheduler_addtask():747: Task list overflow (117504). Need to increase Scheduler:tasks_per_cell.
Aborted (core dumped)I tried increasing Scheduler:tasks_per_cell, but no value seems to help (the number in the error message changes, but that's it). And when I increase all the way to 1000, I get this error:
bv7@apsient:/data/bv7/swiftsim/examples/SedovBlast_3D$ ../swift -G -s -t 4 sedov.yml
Welcome to the cosmological hydrodynamical code
______ _________________
/ ___/ | / / _/ ___/_ __/
\__ \| | /| / // // /_ / /
___/ /| |/ |/ // // __/ / /
/____/ |__/|__/___/_/ /_/
SPH With Inter-dependent Fine-grained Tasking
Version : 0.6.0
Revision: v0.6.0-616-g602118e3, Branch: evrard, Date: 2017-10-31 12:08:37 +0000
Webpage : www.swiftsim.com
Config. options: '--disable-vec --with-hydro=gadget2 --with-riemann-solver=exact --with-hydro-dimension=3 --with-ext-potential=none --with-equation-of-state=ideal-gas --enable-mpi=no --enable-debug=no --enable-debugging-checks=no --enable-optimization=yes --enable-sanitizer=no --no-create --no-recursion'
Compiler: GCC, Version: 4.8.4
CFLAGS : '-O3 -fomit-frame-pointer -malign-double -fstrict-aliasing -ffast-math -funroll-loops -march=bdver2 -mxop -fno-tree-vectorize -pthread -Wall -Wextra -Wno-unused-parameter -Wshadow -Werror'
HDF5 library version: 1.8.11
[00000.0] main: CPU frequency used for tick conversion: 2300084282 Hz
[00000.0] main: Running on: apsient
[00000.0] main: sizeof(part) is 128 bytes.
[00000.0] main: sizeof(xpart) is 64 bytes.
[00000.0] main: sizeof(spart) is 64 bytes.
[00000.0] main: sizeof(gpart) is 96 bytes.
[00000.0] main: sizeof(multipole) is 236 bytes.
[00000.0] main: sizeof(grav_tensor) is 228 bytes.
[00000.0] main: sizeof(task) is 64 bytes.
[00000.0] main: sizeof(cell) is 640 bytes.
[00000.0] main: Reading runtime parameters from file 'sedov.yml'
[00000.0] main: Reading ICs from file './sedov.hdf5'
[00000.0] io_read_unit_system: Reading IC units from ICs.
[00000.0] read_ic_single: IC and internal units match. No conversion needed.
[00000.1] main: Reading initial conditions took 132.250 ms.
[00000.1] main: Read 262144 gas particles, 0 star particles and 262144 gparts from the ICs.
[00000.1] space_init: Re-scaling smoothing lengths by a factor 3.330000e+00
[00000.2] main: space_init took 55.193 ms.
[00000.2] main: space dimensions are [ 1.000 1.000 1.000 ].
[00000.2] main: space is periodic.
[00000.2] main: highest-level cell dimensions are [ 12 12 12 ].
[00000.2] main: 262144 parts in 1728 cells.
[00000.2] main: 262144 gparts in 1728 cells.
[00000.2] main: 0 sparts in 1728 cells.
[00000.2] main: maximum depth is 0.
[00000.2] engine_init: no processor affinity used
[00000.2] engine_policy: engine policies are [ steal keep numa_affinity hydro self_gravity ]
[00000.2] hydro_props_print: Equation of state: Ideal gas.
[00000.2] hydro_props_print: Adiabatic index gamma: 1.666667.
[00000.2] hydro_props_print: Hydrodynamic scheme: Gadget-2 version of SPH (Springel 2005) in 3D.
[00000.2] hydro_props_print: Hydrodynamic kernel: Cubic spline (M4) with eta=1.234800 (48.00 neighbours).
[00000.2] hydro_props_print: Hydrodynamic relative tolerance in h: 0.00010 (+/- 0.0144 neighbours).
[00000.2] hydro_props_print: Hydrodynamic integration: CFL parameter: 0.1000.
[00000.2] hydro_props_print: Hydrodynamic integration: Max change of volume: 1.40 (max|dlog(h)/dt|=0.112157).
[00000.2] gravity_props_print: Self-gravity scheme: FMM-MM with m-poles of order 5
[00000.2] gravity_props_print: Self-gravity time integration: eta=0.0250
[00000.2] gravity_props_print: Self-gravity opening angle: theta=0.1000
[00000.2] gravity_props_print: Self-gravity softening: epsilon=0.0030 (Plummer equivalent: 0.0010)
[00000.2] gravity_props_print: Self-gravity mesh smoothing-scale: a_smooth=1.250000
[00000.2] gravity_props_print: Self-gravity tree cut-off: r_cut_max=4.500000
[00000.2] gravity_props_print: Self-gravity truncation cut-off: r_cut_min=0.100000
[00000.2] engine_init: Absolute minimal timestep size: 3.469447e-19
[00000.2] engine_init: Minimal timestep size (on time-line): 9.536743e-08
[00000.2] engine_init: Maximal timestep size (on time-line): 9.765625e-05
[00000.2] main: engine_init took 1.143 ms.
[00000.2] main: Running on 262144 gas particles, 0 star particles and 0 DM particles (262144 gravity particles)
[00000.2] main: from t=0.000e+00 until t=5.000e-02 with 4 threads and 4 queues (dt_min=1.000e-07, dt_max=1.000e-04)...
[00000.2] engine_init_particles: Computing initial gas densities.
[00000.9] engine.c:engine_addlink():125: Link table overflow.
Aborted (core dumped)I tried changing the value of cell_split_size to lower values, and managed to find some magical combination that works for one of my initial conditions. But all the other tests (including the Sedov blast with self-gravity) seem to fail for all possible combinations I tried.
I mentioned this to @matthieu, and he suggested that this might be an issue with the new task number guessing system. Any idea how to solve this?