New type for foreign gpart and gpart communications
Somewhat of an extension to and revival of #598.
The idea here is to solve two things in one go: Reduce the foreign memory usage of the
gpart and reduce the amount of data movement.
Given that foreign
gpart only act as source of gravity, we only need their mass and positions. This adds a separate type the
gpart_foreign to contain just that information. Cells then have a pointer to either a
gpart array or a
gpart_foreign array. The gravity code only needs a minor change since we were already
building thread-local caches from the particles before launching into the interactions themselves.
This leads to massive savings of memory.
The other possible benefit is that it reduces the amount of data we send. Here I have used
MPI_Type_indexed to select the subset of fields that need shipping. I don't know whether this is implemented in an efficient way. Maybe it reduces some of our memory copy speed issues.
At this stage I am looking for ideas and suggestions mainly.
@rttw52 how does that look for big runs?
Deal with the FOF-related
- Apply the same change to the other gravity scheme.
- Automate the construction of the foreign type via a macro?
When running with debugging checks, add the checking fields to the foreign