Threadpool task plots

Adds the following:

• --enable-threadpool-debugging to the ./configure flags,
• -Y to main.c to enable dumping threadpool timers much like the task timers.

This generates threadpool_info-step%d.dat files every -Y steps which contain the mapper function name, the id of the thread that executed it, the number of chunks processed, and the tic and toc timers for the mapper function.

Oh, yeah, and I suck at git...

Please delete this branch after merging!

Added 8 commits:

• 7d140c20...a0106530 - 7 commits from branch master
• 3bcec4cf - Merge remote-tracking branch 'origin/master' into threadpool_task_plots

Knocked up some plotting scripts and came up with the following for a 4 core SedovBlast analysis. Interesting things already apparent:

http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/sedov/

Added 1 commit:

• 9fa6d93e - Add basic plotting script for threadpool timers

Added 4 commits:

• 9a634a15 - if chunk is zero, split the input into num_thread * threadpool_default_chunk_ratio chunks.
• 3984768d - take the thread id from the number of running threads, not waiting threads.
• 05943c2b - use the new default chunk size.
• e337840d - Merge branch 'threadpool_task_plots' of gitlab.cosma.dur.ac.uk:swift/swiftsim in…

Added 2 commits:

• e8da91c6 - dump to separate files for each rank when doing MPI.
• f53a81f9 - add a second header which is parseable as a python dict with some extra information.

Repeated the SodShock this time without task debugging and with the new chunking:

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/sedov2/

We also have EAGLE_25 (x16):

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/eagle_25/

Note I've let the EAGLE_25 graphics autorange the time limit so that we can see all the details...

Added 2 commits:

• b32c270b - increase the default chunk ratio.
• c13c0a3f - keep a separate log buffer per thread id. this avoids having to protect against …

Added 1 commit:

• 63098d78 - switch to automatic chunk size selection.

Added 1 commit:

• a7f901a4 - Read the header to recover the CPU frequency

We now have:

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/sedov3/

with all the above changes and correct times. Note autoscaling the graphics.

And EAGLE_25:

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/eagle_25_2/

Added 4 commits:

• acaed9cf - hard-code a finer grained chunk size.
• 38a33a55 - use a larger queue_incoming_buffer.
• b110fb4a - smaller chunk ratio of 7 (prime).
• 7e5ca34a - Merge branch 'threadpool_task_plots' of gitlab.cosma.dur.ac.uk:swift/swiftsim in…

New versions with these fixes in place:

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/eagle_25_3/

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/sedov4/

Added 1 commit:

• 03735423 - go back to being super-conservative in runner_do_unskip_mapper.

Added 1 commit:

• b1aa697a - make the logging work even if we're using a single thread.

New plots using latest code at:

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/eagle_25_4/
• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/sedov5/

Plus a new set of plots for EAGLE_25 with a single row per thread.

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/eagle_25_4-1/

Runs using ICC, so I'm happy to accept this. Ready to go, or do we want to look at tuning some more?

No, I think this is good for now, thanks!

Added 2 commits:

• 0de74269 - set engine_taskreweight to 1 so that it never happens.
• 0bac5a1b - log the total cost of the call to threadpool_map as thread id -1.

New versions with fake thread plotted. This is shown as the top thread.

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/eagle_25_5-1/index.html

Sorry, seems the mysterious time was I/O to the terminal. Forgot I had -v 1 enabled and only looked at the output not the log. All much better now.

Added 1 commit:

• 345d9c4a - make the thread that calls threadpool_map do work as well, as if it was the Nth thread in the pool.

Added 2 commits:

• d7bb1cc1 - Handle fake thread with index of -1. This is plotted at the top as an additional thread
• cb374bbc - Don't plot the fake thread we just use its range to colour the background

Is this wanted ?

Enjoy:

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/eagle_25_6-1

with latest fix and new background shading no fake thread.

Added 1 commit:

• d7419412 - don't hold the thread mutex while the calling thread is working.

And once more:

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/eagle_25_7-1

Still no joy.

Added 1 commit:

• d9c69983 - use pthread_barrier instead of mutexes and condvars, might be faster.

One more try: replaced our own barrier based on mutexes and cond-vars with two pthread_barriers. Can you check if this improves anything? Thanks!

Btw, regardless whether this works or not, we should also re-think what we're doing in engine_barrier.

Added 1 commit:

• 145b4377 - get rid of engine_check_sort_tasks, which is not needed in the new sorting scheme.

Looking pretty good now:

• http://astro.dur.ac.uk/~pdraper/swift/threadpool-plots/eagle_25_8-1/index

Excellent!

Added 1 commit:

• b82a57ba - Use shading correctly when expanding threads

That is good to go and be tested thoroughly for integration into master.

All looks good to me, so accepting.

Status changed to merged

mentioned in commit 58a6ff20

The threadpool unit test does not like these changes.

It is strange as I didn't see this with an interactive run. Will look more closely.

Hmm, this seems to be happening during the exit phase of the program and is hard to reproduce, that is add any extra configure options and it goes away and you cannot reproduce when ran in a debugger. I'm going to revert this until we understand what is going on.

mentioned in commit e3309772

mentioned in merge request !383 (merged)

Might be that the test itself needs updating following the changes in the way the threads exit the pool.

Probably not. We're dying in at the line:

unsigned int max_in_before_reset = BARRIER_IN_THRESHOLD
 117                                    - BARRIER_IN_THRESHOLD % count;

in pthread_barrier_wait, when count is zero, so maybe we have a barrier with no threads.

mentioned in commit b9389d4e

I think I've fixed this in threadpool_task_plots2, at least it runs clean even with --enable-sanitizer. The problem was that the threads and the barriers need to be cleaned-up explicitly and in the correct order.

Can either of you have another look?