Skip to content

Use `MPI_Type_vector` when send/recving hydro quantities

MPI_Type_vector allows us to send/recv only certain parts of a struct, as opposed to sending a contiguous stretch of data.

E.g. to send only the x and v fields in a part:

struct part dummy;
MPI_Type_vector(/*count=*/ 1, 
                /*blocklength=*/ (char *)&dummy.v[3] - (char *)&dummy.x[0], 
                /*stride=*/ sizeof(struct part),
                /*oldtype=*/ MPI_Byte, 
                /*newtype=*/ &xv_part_mpi_type);
MPI_Type_commit(&xv_part_mpi_type);
MPI_Isend(&t->ci->parts[0].x[0], c->hydro.count, xv_part_mpi_type, t->cj->nodeID, t->tag, MPI_Comm_world, &t->req);

The above simply states that our datatype is a single (count=1) block of length (char *)&dummy.v[3] - (char *)&dummy.x[0] (blocklength, computed this way because alignment), which repeats every sizeof(struct part) (stride) bytes (oldtype=MPI_BYTE). This can be understood as a filter to lay over the part struct.

When we send data, we have to take care to pass, as a buffer, the address of the first data (&ci->parts[0].x[0]) as opposed to the start of the array (ci->parts).

Note that this implementation assumes that the x and v fields in the part are contiguous. It therefore probably makes sense to implement this on a per-type basis, e.g. Minimal, Gadget2, PE, etc.

This change should significantly reduce the amount of data sent back and forth, albeit in a linear way, i.e. the number of particles send remains the same (currently too high), but the data per particle is much smaller. This will come at a computational cost when packing/unpacking the data, and maybe also in additional buffers within the MPI implementation.

Whoever implements this (assigning to myself for now, but not hopeful I can do this soon) should verify:

  • Correctness (duh),
  • Communication latencies, especially during low-compute timesteps when they are not masked by computation,
  • Memory usage, which may increase because the MPI implementation uses additional buffers,
  • CPU usage, although any change here is probably minimal, but you never know.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information