Change the task splitting criterion for gravity tasks

Fixes #457 (closed).

Implements three main changes:

• When deciding whether or not to split a gravity task use a relative depth and not an absolute depth.
• Send the maximal local depth to the proxies when exchanging cells.
• Add a drift_out implicit task to resolve the high number of gpart_drift -> gravity task dependencies.

The parameter file gets a new parameter to control the depth level of the gravity tasks Scheduler:cell_subdepth_diff_grav which defaults to 4. That is, a gravity task can only be split if it is 4 levels (or more) above the leaf cells in that part of the tree. The best default value of this parameter needs to be investigated.

changed the description

Plots matching the test in #457 (closed) (http://astro.dur.ac.uk/~pdraper/swift-temp/ticks/) will be posted here as soon as I get the nodes to run on.

In the mean time, EAGLE-50 with hydro+gravity+cosmology+stars seems to be doing well on 10 nodes of cosma5 (with parmetis on).

Ok, I can't get this example to run. METIS tells me on step 2 that there are no good partitions.

But the time per step is quite good...

This may be due to the task weights now being completely off...

Actually, I can run it provided I do not apply the patch @pdraper used and just run it out of the box:

--- a/src/partition.c
+++ b/src/partition.c
@@ -1109,9 +1109,12 @@ static void repart_edge_metis(int vweights, int eweights, int timebins,
     /* Skip un-interesting tasks. */
     if (t->cost == 0.f) continue;

-    /* Get the task weight based on costs. */
+    /* Get the task weight based on costs or ticks. */
+#ifdef SWIFT_DEBUG_TASKS
+    double w = (double)t->toc - (double)t->tic;
+#else 
     double w = (double)t->cost;
-
+#endif
     /* Get the top-level cells involved. */
     struct cell *ci, *cj;
     for (ci = t->ci; ci->parent != NULL; ci = ci->parent)

Basically, I can partition if I use the costs but not if I use the tic/toc.

Not surprised, the tic/toc only works correctly after a full step, in my tests it only just managed to get past step2 (and I was forcing repartitions more frequently than you so saw the improvements).

Ok. That makes sense.

So, I think this version does what it should. Now, for the calibration of the new parameter and the verification of whether this helps with the costing estimation I think I'll need your help.

added 1 commit

• d8428c4e - Added more timers to the main sections of the engine to understand where the un-accounted time is.

Compare with previous version

unmarked as a Work In Progress

added 26 commits

• d8428c4e...5d1c3d6e - 24 commits from branch master
• 48cb132b - Merge branch 'master' into gravity_depth2
• 83103e40 - Post-merge fixes.

Compare with previous version

Just for reference I've merged my fitted costs branch into this and am attempting some runs to see how the balance looks under that.

Excellent, thanks. I look forward to the results.

First batch. These use my fitted costs, but with a large fixed cost for pair/grav:

• http://astro.dur.ac.uk/~pdraper/swift-temp/grav_depth_costs/

Seems I forgot to run with 16 threads, so we have 28 instead on 8 nodes of COSMA5.

Tick weighted plots are at:

• http://astro.dur.ac.uk/~pdraper/swift-temp/grav_depth_ticks/

These fail for the first repartition, so look at step30 and later.

So that run completed and the full plots are available for the tick weighted run. Looks very good.

It does indeed!

I'll work separately on the splitting of the mesh tasks. It may also be solved automatically by having them have large weights. In principle, there is no reason to not have these tasks run first (i.e. no dependencies bseides the drifts) and so that could automagically solve the problem of large trailing tasks.

One last piece of good news before the weekend (also for @rgb ): This branch can run the EAGLE-100 on 32 nodes (64 ranks) out of the box. I only increased the number of top-level cells to 64 (but arguably did not try with the default 32). The partition attempt at step 0 fails but we survive with just the geometrical one. The partition on step 2 then succeeds and we run smoothly after that.

One the downside, the imbalance seems to be higher than on the EAGLE-50 case. So still some work for the next few weeks. :)

@pdraper if you are interested in the details, this is job 259769 in /cosma7/data/dp004/jlvc76/SWIFT/master/swiftsim/examples/EAGLE_100

Thanks, that is new. I've never seen the initial partition fail before. Wonder how that happened...