Report memory use of SWIFT

Fixes #414 (closed)

Adds a simple mechanism for getting reports of the main memory use in SWIFT. When not chosen the calls become no-ops. Each report is formatted like:

[0002] [00107.8] :memuse: 0:parts engine.c:engine_exchange_strays:1806 2052274

Where :memuse: is a parsing label for extracting the records, 0:parts shows that this is step 0 and a report of the memory used by parts, the next field is where the report originates and the final number the number of KBs in use.

These can be processed to give sums of each report per step by the process_memuse.py script.

Note that we report particle use, whether or not a reallocation has occurred during rebuild/repartition steps and the special label :step is used to report the memory in use by the process (i.e. resident set size) so that the fractions are clear and how much memory in actually used.

Changes to consider:

• write output to a file not stdout.

added 27 commits

• 2ba32d06...d6c10934 - 26 commits from branch master
• 96b84400 - Merge remote-tracking branch 'origin/master' into memreport-test

Compare with previous version

added 1 commit

• fcdb742b - Formatting

Compare with previous version

added 1 commit

• 432a0853 - Undo junk from other developments

Compare with previous version

added 1 commit

• ab9bacd9 - Fix report to show xparts as expected

Compare with previous version

changed the description

unmarked as a Work In Progress

mentioned in issue #414 (closed)

added 1 commit

• 1e2a88dc - Include missing memuse.h

Compare with previous version

changed the description

added 1 commit

• edc2ac85 - Add protection against garbled lines, that happens when sharing output with other ranks

Compare with previous version

added 1 commit

• 017c570d - Always report the task memory use when rebuilding, not just when reallocating

Compare with previous version

added 1 commit

• 01673dd9 - Fix typos

Compare with previous version

added 54 commits

• 01673dd9...c08467a4 - 53 commits from branch master
• 889b1ed8 - Merge remote-tracking branch 'origin/master' into memreport-test

Compare with previous version

Thinking a bit more about this following our quick chat on Friday. Is there a way to track the free() calls as well?

added 974 commits

• 889b1ed8...1311f784 - 973 commits from branch master
• d12d05d5 - Merge branch 'master' into memreport-test

Compare with previous version

added 1 commit

• 60502af8 - Post-merge fixes and formatting.

Compare with previous version

added 2130 commits

• 60502af8...a0f68fd8 - 2129 commits from branch master
• 635f2f70 - Merge branch 'master' into memreport-test

Compare with previous version

added 1 commit

• 792e8d20 - Instrument some more memory use

Compare with previous version

I've completely revisited this topic and created the new branch memreport-swift, which replaces all the posix_memalign and associated free calls with swift versions that include a label, so we know what the memory is for. The information is gathered using a logger that you can dump, so we can build up a timeline of allocations and frees and the cumulative memory in these calls.

See !757 (merged) which supersedes this work.