Log rank CPU times and memory use to assess the balance.

Writes the user and system CPU times per rank into a file rank_cpu_balance.log together with the current load balance estimate. Also records the resident set size of the process on each rank to a separate file rank_memory_balance.log. The resident size is essentially the core memory use (but note only done when considering whether to repartition, as are the CPU times, so is indicative rather than some peak use).

These should be useful when trying to understand how the overall balance per node is working.

As part of this the CPU times used to estimate the balance are changed to only include time used by the tasks in engine_launch(), previously all the time used in the step was considered. Given the steps considered are those that interact all particles this is not likely to make much difference, but clarifies what we are using.

src/clocks.c

@@ -276,22 +276,24 @@ double clocks_get_hours_since_start(void) {
 }
 
 /**
- * @brief return the cpu time used.
+ * @brief return the cpu times used.
  *
- * Uses the times(2) function to access the user cpu times and returns the sum
- * of these for the process tree, i.e. current process plus "waited-for"
- * children. This may be pthread implementation specific as to what that
- * exactly means. Note we do not include the system time as that includes
- * spin times and we don't want to give credit for that.
+ * Uses the times(2) function to access the user and system cpu times and
+ * returns the sum of these for the process tree, i.e. current process plus
+ * "waited-for" children. This may be pthread implementation specific as to
+ * what that exactly means.
  *
- * @result cpu time used in sysconf(_SC_CLK_TCK) ticks, usually 100/s not our
- *         usual ticks.
+ * Note cpu times are reported in sysconf(_SC_CLK_TCK) ticks, usually 100/s
+ * not our usual ticks.
+ *
+ * @param usertime the user time.
+ * @param systime the system time.
  */
-double clocks_get_cputime_used(void) {
-
+void clocks_get_cputimes_used(double *usertime, double *systime) {
   struct tms tmstic;
   times(&tmstic);
-  return (double)(tmstic.tms_utime + tmstic.tms_cutime);
+  *usertime = (tmstic.tms_utime + tmstic.tms_cutime);
+  *systime = (tmstic.tms_stime + tmstic.tms_cstime);
 }
 
 /**