WIP: MPI parallel mesh gravity

Currently the mesh gravity calculation stores a full copy of the gravity mesh and repeats the FFT calculations on every MPI rank. In runs with many MPI ranks this increases memory use, generates a lot of communication, and limits the mesh size that can be used.

I've been looking into using the MPI support in FFTW3. The sequence of operations would be something like

• Each rank accumulates it's own contributions to the density field. These would need to be stored in some kind of sparse array to avoid making any assumptions about how the domain decomposition is done. The hash map code used in the FoF implementation might be suitable.
• Each rank allocates a slab of the full mesh as required by FFTW
• The mesh contributions are sent to whichever rank stores the corresponding slab. This is reasonably straightforward because the coordinates of a cell indicate which rank it's stored on.
• The FFT/Green function/CIC deconvolution/inverse FFT are carried out to make a slab-distributed mesh containing the potential
• Each rank calculates which cells it needs to calculate the potential gradient for its particles and requests them from whichever node they're on. This is slightly trickier than constructing the mesh because we need several cells around each particle to evaluate the gradient.
• We can then evaluate the potential and acceleration on the particles as usual

This branch executes the first three steps alongside the usual mesh calculations and reports how well the distributed density mesh agrees with the full mesh. Then it just discards the distributed mesh. Currently it's all single threaded and the distributed mesh is not quite in the right format because MPI FFTW needs some specific padding in the last dimension.

added 7 commits

• 525509b9 - Adjust allocation of FFTW mesh slices to account for padding in MPI mode
• 78ecf4e4 - Moved row_major_id_periodic_size_t to header
• 005289c5 - Modified MPI mesh code to use padding needed by FFTW
• af795e62 - Now executes the MPI FFT. Results not checked yet.
• 6bfda3d4 - Code now reports fractional difference between MPI and non-MPI FFTs
• 053209ac - Modify mesh_apply_Green_function to work with the MPI mesh
• 1c98053e - Execute inverse MPI FFT and check potential against non-MPI result

Compare with previous version

This branch now calculates the potential on a slab distributed mesh using MPI FFTW and compares the result to the non_MPI calculation. The next thing to implement is to have each MPI rank retrieve the parts of the mesh it needs to calculate the acceleration on its local particles.

added 349 commits

• 1c98053e...026e9a4c - 347 commits from branch master
• cfb09acc - Only correct singularity in FFT if we have a non-zero thickness slice of the mesh starting at x=0
• 42cade05 - Merge remote-tracking branch 'origin' into hashmap_mesh

Compare with previous version

added 20 commits

• 11dcac29 - Test method using hashmap for mesh gravity
• 097a4f3a - Debugging of hashmap mesh code
• 89803ab0 - Test for FFTW MPI in configure.ac
• 65f9c395 - Wrote code to convert hashmap density field to slices for FFTW
• c355dd29 - Debugging of MPI mesh calculation. Now agrees with serial version.
• 09667730 - Ran formatting tool
• 6e600bd4 - Adjust allocation of FFTW mesh slices to account for padding in MPI mode
• 4e1c96c0 - Moved row_major_id_periodic_size_t to header
• 7b80e713 - Modified MPI mesh code to use padding needed by FFTW
• 6bbbe0e6 - Now executes the MPI FFT. Results not checked yet.
• 2b974e87 - Code now reports fractional difference between MPI and non-MPI FFTs
• 37df0c1d - Modify mesh_apply_Green_function to work with the MPI mesh
• 0cf78e99 - Execute inverse MPI FFT and check potential against non-MPI result
• 268bfa40 - Only correct singularity in FFT if we have a non-zero thickness slice of the mesh starting at x=0
• 3cbff397 - Check for message size overflow in mpi_mesh_gravity.c
• cedf200d - Generalize exchange_mesh_cells() to work on other structs
• 145b23d6 - Moved hashmap to array conversion code into a function
• 997d6520 - Added function to compute which mesh cells to import
• 9a65e66c - Avoid leaking hashmaps in find_cells_to_import()
• 07debc59 - Merged updates from master

Compare with previous version

added 4 commits

• 3ae3c1d9 - Remove whitespace at ends of lines
• 8534d94d - Added code to fetch needed potential values in mpi_mesh_gravity
• d084dd31 - Debugging of new fetch_potential() function in mpi_mesh_gravity
• 349efc05 - Check potential against global array after retreiving required cells

Compare with previous version

This version now contains code to take the slab distributed mesh and have each MPI rank retrieve the potential in the cells it needs. Each rank will store all mesh cells within some radius of its local top level cells. Particles can move up to half a top level cell size in between calculation of the mesh (at rebuild time) and evaluation of the force (during the timesteps) and we need adjacent mesh cells around each particle to evaluate the potential gradient. So I think the required radius is 0.5*(top level cell size)+2*(FFT mesh cell size).

added 5 commits

• fd592212 - Avoid linking FFTW MPI when not needed and fix compilation without MPI
• 5f2f97ee - Add a configure flag for MPI mesh gravity
• 92c5c657 - Separate pm_mesh_compute_potential into versions with/without FFTW MPI
• 51d01d89 - Modify mesh_to_gparts_CIC to work with distributed potential
• 6678df17 - Fix reporting of presence of FFTW MPI and threads

Compare with previous version

added 4 commits

• 58536c08 - Renamed several functions in mpi_mesh_gravity.c
• fa8e3d4a - Add a run time parameter for MPI mesh gravity
• 92748394 - Remove stray #endif in src/mesh_gravity.h
• 59bb1d8c - Fix compilation without mpi mesh gravity

Compare with previous version

added 3 commits

• adf67d85 - Correct type of loop index in mpi_mesh_gravity.c
• 9a2ec906 - Remove incorrect const attribute on function in mpi_mesh_gravity
• c8249e17 - Use hashmap_get instead of hashmap_put in mpi_mesh_gravity.c

Compare with previous version

added 1 commit

• f34eb279 - Remove (now redundant) hashmap_put() call

Compare with previous version

added 1 commit

• 2469f254 - Fix typo in mpi_mesh_gravity.c setting hashmap value

Compare with previous version

added 1 commit

• 6ec92985 - Use temporary mesh to accumulate gparts to hashmap

Compare with previous version

added 4 commits

• dd064df2 - Measure cell extent in accumulate_cell_to_hashmap
• af2a646e - Avoid passsing struct space into accumulate_cell_to_hashmap
• 90ab1c31 - Use threadpool for MPI mesh assignment
• 844fd827 - Hold lock while processing all hashmap updates in accumulate_cell_to_hashmap

Compare with previous version

added 149 commits

• 844fd827...7cbf85bd - 148 commits from branch master
• 38b1882f - Updated hashmap_mesh branch from master

Compare with previous version

added 1 commit

• 043707c8 - Fix overlooked merge conflict in src/Makefile.am

Compare with previous version

added 1 commit

• 399e6821 - Added code to find accelerations from MPI mesh

Compare with previous version

added 3 commits

• d8bdcefb - Revert unnecessary change in mesh_gravity.c
• 10bda3e3 - Correct size of local mesh, and add bounds checking
• 9d23053d - Move CIC acceleration calculation to separate function

Compare with previous version

added 5 commits

• 7a3aabce - Renamed mpi_mesh_gravity .c/h files to mesh_gravity_mpi
• 14d96a80 - Moved mesh patch related code to a new file
• 93957a0c - Access mesh patches via pm_mesh_CIC_get and set functions
• 662294ed - Removed possibly unsafe const attributes
• c488f4eb - Fix incorrect array indexing in mesh patch

Compare with previous version

added 11 commits

• c488f4eb...b56cb576 - 8 commits from branch master
• d5c47fde - Merge branch 'master' into hashmap_mesh
• 4d3ec9a7 - Remove extra newline in log message
• 9a58b96d - Increase size of imported boundary in distributed mesh

Compare with previous version

added 179 commits

• 9a58b96d...ea54b741 - 178 commits from branch master
• 1a9a55d9 - Merge branch 'master' into hashmap_mesh

Compare with previous version