Parallel mesh assignment also in single-node case

Change the strategy used when interpolating the gpart onto the gravity mesh. Each thread (aka. top-level cell) constructs a local patch of the mesh and assigns its particles to it. When done, the patch is written to the global mesh using atomics. This is now similar to what is done in the distributed MPI case.

This strategy seems to be beneficial when there are lots of particles in only a few cells, for instance in a zoom run. It seems to not be slower in other cases either.

The behaviour can be controlled by a runtime parameter (Gravity:mesh_uses_local_patches defaults to 1) if one wants to roll back to the "old" per-particle atomic assignment.

added SPH performance labels

added 1 commit

• a1d35e17 - Document the new function

Compare with previous version

@jch if you could check whether the way I am using the mesh patches here in the non-MPI case makes sense that would be much appreciated.

added 18 commits

• a1d35e17...accda3a4 - 17 commits from branch master
• 5831b52b - Merge branch 'master' into parallel_mesh_assignment

Compare with previous version

requested review from @jch

assigned to @matthieu

marked this merge request as ready

assigned to @bvandenbroucke and unassigned @matthieu

Each mesh patch contains an array that needs to be allocated. That happens in pm_mesh_patch_init(), which is called from accumulate_cell_to_local_patch(). I don't think the patches allocated in cell_gpart_to_mesh_CIC_mapper() get freed at the moment. We need a pm_mesh_patch_clean() at the end of the loop.

In cell_gpart_to_mesh_CIC_mapper() I think we'll need to skip cells with no gparts, if that can happen. accumulate_cell_to_local_patch() just returns without allocating the patch in that case.

Cells with no gparts might be a problem in the MPI case too.

added 33 commits

• 5831b52b...49c26843 - 32 commits from branch master
• f2d31f13 - Merge branch 'master' into parallel_mesh_assignment

Compare with previous version

added 1 commit

• e1c9d96f - Make sure the local patches are freed after use in cell_gpart_to_mesh_CIC_mapper()

Compare with previous version

Thanks John. All good points.

added 1 commit

• 8b1086d5 - Also escape empty cells in the MPI version of the cell->patch interpolation mapper

Compare with previous version

added 1 commit

• 51cff14a - Make the option to use local patches instead of direct atomic writes a...

Compare with previous version

added 1 commit

• f195dd04 - Document the new option

Compare with previous version

I have made this change a runtime parameter. I think we still want to run it in different scenarios, especially forthcoming zoom cases, to see whether it is a gain in general or not.

resolved all threads

@dc-rope1 any final thoughts?

changed the description

Nothing from me, this all seems to work (and is efficient) in the zoom case!

Great. So this is ready to go as well.

approved this merge request

merged

mentioned in commit 1501b338