Add option to have a separate MPI version of velociraptor

added Configuration enhancement feature request labels

requested review from @matthieu

Thanks Peter. Let's see whether this indeed solves the situation Yannick identified.

Sorry for the late reply, I didn't get around to testing this before the break. By the looks of it, there is some inconsistency in the configure file, I get an "unexpected end of file" error when running this branch:

checking for InitVelociraptor in -lvelociraptor... yes
checking for VR_NOMASS in -lvelociraptor... no
VELOCIraptor not compiled to so as to *not* store masses per particle.
./configure: line 31759: syntax error: unexpected end of file
make: *** [config.status] Error 2

I cannot immediately spot the upsetting change to configure, but maybe you have more luck?

Actually, on a second look, I think there is simply a missing f in the fi in one place:

--- a/configure.ac
+++ b/configure.ac
@@ -1367,7 +1367,7 @@ if test "$have_mpi_velociraptor" == "yes" -a "$have_velociraptor" != "yes"; then
 elif test "$have_velociraptor" == "yes" -a "$have_mpi_velociraptor" != "yes"; then
    VELOCIRAPTOR_MPI_LIBS="$VELOCIRAPTOR_LIBS"
    AC_SUBST([VELOCIRAPTOR_MPI_LIBS])
-i
+fi
 
 # Check for dummy VELOCIraptor.
 AC_ARG_ENABLE([dummy-velociraptor],

With that fix, at least the compilation goes smoothly. I'm trying it out on a low-res box now to see whether there are any surprises at runtime.

It looks like this does indeed work, modulo two (probably unrelated) problems:

(i) in non-MPI mode, the code crashes during structure finding with the "non-positive density issue"

terminate called after throwing an instance of 'vr::non_positive_density'
  what():  Particle density not positive, cannot continue.

Particle information: id=0, pid=473282, type=1, pos=(-0.0490349, -0.162667, -0.00342901), vel=(-515.424, 265.28, -495.509), mass=2.50233, density=4.24399e-314
Execution context: MPI enabled=no, OpenMP enabled=yes, thread=0/1
/var/slurm/slurmd/job4588448/slurm_script: line 23: 194817 Aborted                 ./swift -v 1 --pin --threads=28 --cosmology --eagle --velociraptor params.yml

(ii) in MPI mode, the code only runs when I compile without ParMETIS. Otherwise, I get a crash almost immediately after the start, during what I think is the initial partitioning

[0000] [00018.0] main: Running on 1331000 gas particles, 0 sink particles, 0 stars particles 0 black hole particles, 0 neutrino particles, and 1331000 DM particles (2662000 gravity particles)
[0000] [00018.0] main: from t=1.220e-05 until t=1.410e-02 with 2 ranks, 4 threads / rank and 4 task queues / rank (dt_min=1.000e-10, dt_max=1.000e-02)...
===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 107436 RUNNING AT login7b.pri.cosma7.alces.network
=   EXIT CODE: 11
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================

It may be that this is exacerbated/caused by trying to split such a ridiculously small (test) simulation in the first place...

In any case, I think that the "dual-VR" part can be merged. Thanks @pdraper!

added 1 commit

e31ef42f - Fixed bug in configure.ac

Compare with previous version

added 1 commit

3de65701 - Document VR MPI and non-MPI configure options

Compare with previous version

added 3 commits

3de65701...dee0a623 - 2 commits from branch master
3050c30a - Merge branch 'master' into velociraptor-options

Compare with previous version

marked this merge request as ready

assigned to @pdraper and unassigned @dc-bahe1

Thanks for the fixes and the tests. Will accept this now, just added some updates to the documentation.

merged

mentioned in commit 366aee80

Add option to have a separate MPI version of velociraptor

Merge request reports

Activity