g-particle did not interact gravitationally with all other gparts
I have a very basic test case that I use to test changes to the stars interactions. It consists of a uniform box (the 32^3 glass file) with 100 star particles randomly scattered in the box. The IC is generated using the script makeIC.py, using python3 makeIC.py 100. The parameter file is test_multi.yml. I configure SWIFT with --with-subgrid=EAGLE-XL --enable-task-debugging --enable-threadpool-debugging --enable-cell-graph --enable-debug --enable-debugging-checks. The test also needs all the input files for EAGLE-XL.
When I run with ../swift --hydro --limiter --sync --self-gravity --stars --feedback --cooling --task-dumps=1 --threadpool-dumps=1 --threads 3 test_multi.yml -v 1, I get the following error (full log):
[00105.3] runner_others.c:runner_do_end_grav_force():799: g-particle (id=25235, type=Gas) did not interact gravitationally with all other gparts gp->num_interacted=420, total_gparts=32868 (local num_gparts=32868 inhibited_gparts=0)
[00105.3] runner_others.c:runner_do_end_grav_force():799: g-particle (id=18172, type=Gas) did not interact gravitationally with all other gparts gp->num_interacted=2389, total_gparts=32868 (local num_gparts=32868 inhibited_gparts=0)
[00105.3] runner_others.c:runner_do_end_grav_force():799: g-particle (id=10766, type=Gas) did not interact gravitationally with all other gparts gp->num_interacted=1407, total_gparts=32868 (local num_gparts=32868 inhibited_gparts=0)When I run with mpirun -np 3 ../swift_mpi --hydro --limiter --sync --self-gravity --stars --feedback --cooling --task-dumps=1 --threadpool-dumps=1 --threads=3 test_multi.yml, I get (full log):
210 8.281250e-05 1.0000000 0.0000000 3.906250e-07 48 50 7763 7763 0 0 0 915.458 0 277.623
[0000] [00221.3] engine_drift_all: Drifting all to t=8.320312e-05
[0000] [00221.3] space_rebuild: (re)building space
[0000] [00221.3] space_regrid: (re)griding space cdim=(3 3 3)
[0001] [00221.3] space_regrid: (re)griding space cdim=(3 3 3)
[0000] [00221.3] space_regrid: basic cell dimensions have increased - recalculating the global partition.
[0000] [00221.3] check_complete: Region 2 is not present in partition
[0000] [00221.3] space_regrid: Failed to get a new partition, trying less optimal method
[0001] [00221.3] check_complete: Region 2 is not present in partition
[0001] [00221.3] space_regrid: Failed to get a new partition, trying less optimal method
[0002] [00221.3] space_regrid: (re)griding space cdim=(3 3 3)
[0002] [00221.3] check_complete: Region 2 is not present in partition
[0002] [00221.3] space_regrid: Failed to get a new partition, trying less optimal method
[0002] [00221.6] runner_others.c:runner_do_end_grav_force():799: g-particle (id=25235, type=Gas) did not interact gravitationally with all other gparts gp->num_interacted=1329, total_gparts=32868 (local num_gparts=11101 inhibited_gparts=0)
[0002] [00221.6] runner_others.c:runner_do_end_grav_force():799: g-particle (id=27054, type=Gas) did not interact gravitationally with all other gparts gp->num_interacted=2349, total_gparts=32868 (local num_gparts=11101 inhibited_gparts=0)
[0000] [00221.6] runner_others.c:runner_do_end_grav_force():799: g-particle (id=6633, type=Gas) did not interact gravitationally with all other gparts gp->num_interacted=3, total_gparts=32868 (local num_gparts=10822 inhibited_gparts=0)
[0001] [00221.6] runner_others.c:runner_do_end_grav_force():799: g-particle (id=10766, type=Gas) did not interact gravitationally with all other gparts gp->num_interacted=1438, total_gparts=32868 (local num_gparts=10945 inhibited_gparts=0)I got a binary dump of all the particles at the time of the crash, and as far as I can tell, none of them has any weird properties. When run with the explicit stars density check (--enable-stars-density-checks=1; not over MPI), the run works fine until the crash. This crash also happens with master, without any of my experimental changes.
This is a very artificial setup, so I wouldn't be surprised if that is somehow causing the crash. But it is not clear to me what is happening. Any suggestions to find out more are welcome.