Add logging for task MPI requests

Logs the sends, recvs and successful tests of the MPI requests made in the engine. Shares infrastructure with the memuse logger, so includes some refactoring of that. To use this you need to enable it at configure time with --enable-mpiuse-reports.

A log is created for each rank and step. Usually these requests will all match, so the logs may not sound interesting, but they do show the use of memory in MPI as the step progresses, and other useful things like the maximum, mean and sum of the sizes of packets sent (this is in a post-amble section), they also allow the inspection of how efficient we are or the MPI library, since we include the handoff time.

Matching between ranks can be done using the match_mpireports.py script that will be useful when diagnose the MPI library and fabric performance, but note that ticks are not necessarily synchronized between ranks. but should be close.

Also included is a cleanup of the MPI section of scheduler.c. Should be far more obvious what is happening now, but clearly this needs a good checking as a critical section of code.

doc/RTD/source/AnalysisTools/index.rst

@@ -31,7 +31,7 @@ To solve this problem, you will need to either access them through an existing s
 or install ``npm`` and then run the following commands
 
 .. code-block:: bash
-   
+
    npm install http-server -g
    http-server .
 
@@ -78,3 +78,38 @@ the step, and the total memory still in use per label. Note this includes
 memory still active from previous steps and the total memory is also continued
 from the previous dump.
 
+MPI task communication reports
+------------------------------
+
+When SWIFT is configured using the ``--enable-mpiuse-reports`` flag it will
+log any all asynchronous MPI communications made to send particle updates
+between nodes to support the tasks.
+
+The output files are called ``mpiuse_report-rank<m>-step<n>.dat``, i.e. one
+per rank per step. These have a line for each request for communication, either
+an MPI_Irecv or MPI_Isend and a line for the subsequent completion (successful
+MPI_Test).
+
+Each line of the logs contains the following information:
+
+.. code-block:: none
+
+   stic:             ticks since the start of this step
+   etic:             ticks since the start of the simulation
+   dtic:             ticks that the request was active
+   step:             current step
+   rank:             current rank
+   otherrank:        rank that the request was sent to or expected from
+   type itype:       task type as string and enum
+   subtype isubtype: task subtype as string and enum
+   activation:       1 if record for the start of a request, 0 if request completion
+   tag:              MPI tag of the request
+   size:             size, in bytes, of the request
+   sum:              sum, in bytes, of all requests that are currently not logged as complete 
+
+The stic values should be synchronized between ranks as all ranks have a
+barrier in place to make sure they start the step together, so should be
+suitable for matching between ranks. The unique keys to associate records
+between ranks (so that the MPI_Isend and MPI_Irecv pairs can be identified)
+are "otherrank/rank/subtype/tag/size" and "rank/otherrank/subtype/tag/size"
+for send and recv respectively. When matching ignore step0.