Allow star formation when running over MPI

This extends the current version of the code by allowing star formation when running over MPI. This means:

• Creating space for the spare particles only for local + foreign cells and not all top-level cells,
• Sorting the spare particles only for local cells,
• Discarding the spare particles when redistributing.
• Fixing #560 (closed).
• Fix a memory leak in the freeing of stars' sort array that follows the creation of a star,
• Move the star formation task to the top-level since it does restructure the spart array in the whole cell,
• Add time-step communication tasks for each particle type rather than one single bundled operation.

The final step (coming soon in a separate branch) is to allow both Star formation and feedback at the same time over MPI.

So how do I force the creation of stars? My run of EAGLE_6 for 500 steps seems to have the same number as when it started.

Ah yes, sorry. The cooling needs to be on as well.

So:

./configure --with-subgrid=EAGLE --disable-hand-vec

cd examples/EAGLE_low_z/EAGLE_6/ 

../../Cooling/getEagleCoolingTable.sh

mpirun -np 4 ../../swift_mpi --cosmology --hydro --self-gravity --stars --star-formation --cooling --threads=4 eagle_6.yml

added 1 commit

• 992f4f92 - Use a better value (the default one) for the gas fraction assumed by the EAGLE…

Compare with previous version

added 1 commit

• c59e68ee - Update the md5sum of the EAGLE low-z examples now that stars have been added to the ICs.

Compare with previous version

You probably also want the updated ICs that already contain some stars. They are now the default in the repository but you may still have some older ones (without stars) in your test directories.

Thanks, thought I was missing a step, not all those... The bad news is that I see an issue with the debugging checks enabled:

       7   1.276163e-02    0.9091649    0.0999104   1.635742e-07   43   43           24           24            0                76.118      0
       8   1.276179e-02    0.9091755    0.0998976   1.635753e-07   43   46         2498        24160        17521             18323.580      0
[0000] [00102.4] engine_drift_all: Drifting all to a=9.091861e-01
[0003] [00102.5] drift.h:drift_spart():132: s-particle has not been drifted to the current time sp->ti_drift=140737488355328, c->ti_old=17592186044416, ti_current=158329674399744
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 3

Same place twice in a row, so consistent.

Seems like you are a victim of #560 (closed). That's good as I was struggling to find a run where it happens below 1000s of steps... Now I can go and fix it more easily. That has nothing to do with MPI though.

With EAGLE_12 it runs for 13 steps:

      12   1.276098e-02    0.9091226    0.0999616   4.089259e-08   41   43         3164         3166            2               222.398      0
      13   1.276102e-02    0.9091253    0.0999584   4.089259e-08   41   41           22           22            0                89.392      0
[0003] [00114.0] drift.h:drift_spart():132: s-particle has not been drifted to the current time sp->ti_drift=52776558133248, c->ti_old=4398046511104, ti_current=61572651155456
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 3

Think I'll wait for a fix before trying anything else.

added 1 commit

• b5206f3b - When forming a star in a cell that never had a star before, set the ti_old_part…

Compare with previous version

That is now fixed for the two examples that crashed.

That last bug was already there creeping in the dark. Looks like I never used the combination of flags above. With the improved version of the hydro that I normally use, the code forms less stars and did not hit that specific problem.

added 1 commit

• 55871447 - Prevent users from running with star-formation but without stars.

Compare with previous version

added 1 commit

• 86fbab05 - Undo unwanted change to the EAGLE-12 parameter file.

Compare with previous version

added 1 commit

• c57d8f39 - Allow the creation of hydro self-tasks in a cell that has only stars as they…

Compare with previous version

Ok. I think I am all done now with this one. I have added a fix for #560 (closed).

changed the description

added 1 commit

• 87b84844 - Release the star formation lock before exiting early with a warning

Compare with previous version

Just to check I reduced the space for extra stars and tested that. It deadlocked waiting for stars.star_formation_lock. Simple fix so I've pushed that. I now see the warning about "we ran out of star particles."

Ah, yes, thanks!

Back to normal testing. Ran a EAGLE_50 on 12 nodes of COSMA6 with the same options as above and got:

[0008] [02048.7] cell.c:cell_add_spart():4462: Cell had not been correctly drifted before star formation

after 7 steps.