WIP: New type for foreign gpart and gpart communications
Somewhat of an extension to and revival of #598.
The idea here is to solve two things in one go: Reduce the foreign memory usage of the gpart
and reduce the amount of data movement.
Given that foreign gpart
only act as source of gravity, we only need their mass and positions. This adds a separate type the gpart_foreign
to contain just that information. Cells then have a pointer to either a gpart
array or a gpart_foreign
array. The gravity code only needs a minor change since we were already
building thread-local caches from the particles before launching into the interactions themselves.
This leads to massive savings of memory.
The other possible benefit is that it reduces the amount of data we send. Here I have used MPI_Type_indexed
to select the subset of fields that need shipping. I don't know whether this is implemented in an efficient way. Maybe it reduces some of our memory copy speed issues.
At this stage I am looking for ideas and suggestions mainly.
@rttw52 how does that look for big runs?
Todo:
-
Deal with the FOF-related gpart
communication. -
Apply the same change to the other gravity scheme. -
Automate the construction of the foreign type via a macro? -
When running with debugging checks, add the checking fields to the foreign gpart
?