Change the task splitting criterion for gravity tasks
Fixes #457 (closed).
Implements three main changes:
- When deciding whether or not to split a gravity task use a relative depth and not an absolute depth.
- Send the maximal local depth to the proxies when exchanging cells.
- Add a
drift_out
implicit task to resolve the high number of gpart_drift -> gravity task dependencies.
The parameter file gets a new parameter to control the depth level of the gravity tasks Scheduler:cell_subdepth_diff_grav
which defaults to 4
. That is, a gravity task can only be split if it is 4 levels (or more) above the leaf cells in that part of the tree. The best default value of this parameter needs to be investigated.
Merge request reports
Activity
Plots matching the test in #457 (closed) (http://astro.dur.ac.uk/~pdraper/swift-temp/ticks/) will be posted here as soon as I get the nodes to run on.
This may be due to the task weights now being completely off...
Edited by Pedro GonnetActually, I can run it provided I do not apply the patch @pdraper used and just run it out of the box:
--- a/src/partition.c +++ b/src/partition.c @@ -1109,9 +1109,12 @@ static void repart_edge_metis(int vweights, int eweights, int timebins, /* Skip un-interesting tasks. */ if (t->cost == 0.f) continue; - /* Get the task weight based on costs. */ + /* Get the task weight based on costs or ticks. */ +#ifdef SWIFT_DEBUG_TASKS + double w = (double)t->toc - (double)t->tic; +#else double w = (double)t->cost; - +#endif /* Get the top-level cells involved. */ struct cell *ci, *cj; for (ci = t->ci; ci->parent != NULL; ci = ci->parent)
added 1 commit
- d8428c4e - Added more timers to the main sections of the engine to understand where the un-accounted time is.
added 26 commits
-
d8428c4e...5d1c3d6e - 24 commits from branch
master
- 48cb132b - Merge branch 'master' into gravity_depth2
- 83103e40 - Post-merge fixes.
-
d8428c4e...5d1c3d6e - 24 commits from branch
First batch. These use my fitted costs, but with a large fixed cost for
pair/grav
:Seems I forgot to run with 16 threads, so we have 28 instead on 8 nodes of COSMA5.
Tick weighted plots are at:
These fail for the first repartition, so look at step30 and later.
Edited by Peter W. DraperIt does indeed!
I'll work separately on the splitting of the mesh tasks. It may also be solved automatically by having them have large weights. In principle, there is no reason to not have these tasks run first (i.e. no dependencies bseides the drifts) and so that could automagically solve the problem of large trailing tasks.
One last piece of good news before the weekend (also for @rgb ): This branch can run the EAGLE-100 on 32 nodes (64 ranks) out of the box. I only increased the number of top-level cells to 64 (but arguably did not try with the default 32). The partition attempt at step 0 fails but we survive with just the geometrical one. The partition on step 2 then succeeds and we run smoothly after that.
One the downside, the imbalance seems to be higher than on the EAGLE-50 case. So still some work for the next few weeks. :)
@pdraper if you are interested in the details, this is job 259769 in
/cosma7/data/dp004/jlvc76/SWIFT/master/swiftsim/examples/EAGLE_100