Use `MPI_Type_vector` when send/recving hydro quantities
MPI_Type_vector
allows us to send/recv only certain parts of a struct, as opposed to sending a contiguous stretch of data.
E.g. to send only the x
and v
fields in a part:
struct part dummy;
MPI_Type_vector(/*count=*/ 1,
/*blocklength=*/ (char *)&dummy.v[3] - (char *)&dummy.x[0],
/*stride=*/ sizeof(struct part),
/*oldtype=*/ MPI_Byte,
/*newtype=*/ &xv_part_mpi_type);
MPI_Type_commit(&xv_part_mpi_type);
MPI_Isend(&t->ci->parts[0].x[0], c->hydro.count, xv_part_mpi_type, t->cj->nodeID, t->tag, MPI_Comm_world, &t->req);
The above simply states that our datatype is a single (count=1) block of length (char *)&dummy.v[3] - (char *)&dummy.x[0]
(blocklength, computed this way because alignment), which repeats every sizeof(struct part)
(stride) bytes (oldtype=MPI_BYTE). This can be understood as a filter to lay over the part
struct.
When we send data, we have to take care to pass, as a buffer, the address of the first data (&ci->parts[0].x[0]
) as opposed to the start of the array (ci->parts
).
Note that this implementation assumes that the x
and v
fields in the part
are contiguous. It therefore probably makes sense to implement this on a per-type basis, e.g. Minimal, Gadget2, PE, etc.
This change should significantly reduce the amount of data sent back and forth, albeit in a linear way, i.e. the number of particles send remains the same (currently too high), but the data per particle is much smaller. This will come at a computational cost when packing/unpacking the data, and maybe also in additional buffers within the MPI implementation.
Whoever implements this (assigning to myself for now, but not hopeful I can do this soon) should verify:
- Correctness (duh),
- Communication latencies, especially during low-compute timesteps when they are not masked by computation,
- Memory usage, which may increase because the MPI implementation uses additional buffers,
- CPU usage, although any change here is probably minimal, but you never know.