Hang during 1024^3 mesh all-reduce with Intel MPI 2020.2 on 10 nodes

I have some tests that go ok, but I have a DMO test ~650**3 particles (zoom) that has some issues getting going with the new 2020 compiler set up.

With 10 nodes (14/rank, ie 20 ranks) it hangs during the mesh communications.

/cosma7/data/dp004/rttw52/swift_runs/runs/Sibelius/200Mpc/Sibelius_200Mpc_512/out_files/log_2686635.txt

If I reduce that to 6 ranks (3 nodes) it gets past the comms, but hangs during engine_launch.

/cosma7/data/dp004/rttw52/swift_runs/runs/Sibelius/200Mpc/Sibelius_200Mpc_512/out_files/log_2687406.txt

Edited Oct 20, 2020 by Matthieu Schaller

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information