diff --git a/.clang-format b/.clang-format new file mode 100644 index 0000000000000000000000000000000000000000..49524f944860b51ba9aa461b3299b70a68df46ae --- /dev/null +++ b/.clang-format @@ -0,0 +1,6 @@ +--- +Language: Cpp +BasedOnStyle: Google +KeepEmptyLinesAtTheStartOfBlocks: true +PenaltyBreakAssignment: 2 +... diff --git a/README.md b/README.md index 36b4e1070ee1f9deeaf5ade6438924706b8cdc5f..2601f6ecf09cb6a2eca4b1422afd8ce5834ef078 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,47 @@ +SWIFTmpistepsim +=============== -This package is a standalone part of [SWIFT](http://www.swiftsim.com) that -aims to roughly simulate the MPI interactions that taking a single step -of a simulation makes. +This project is a standalone part of [SWIFT](http://www.swiftsim.com) that +aims to roughly simulate the MPI interactions that taking a single step of a +SWIFT simulation makes. Making it possible to more easily see the performance +of MPI calls, and also investigations of tuning more obvious. -The interactions are captured from a run of SWIFT when configured using the -configure option `--enable-mpiuse-reports`. When this is enabled each step of -the simulation produces logs for each rank which record when the MPI -interaction was started and when it completed. Other information such as the -ranks involved, the size of the data exchanged, the MPI tags used and which -SWIFT task types were used are also recorded. +The actual process within SWIFT is that queues of cell-based tasks are ran, +with their priorities and dependencies determining the order that the tasks +are ran in. Tasks are only added to a queue when they are ready to run, that +is they are not waiting for other tasks. This order also determines when the +sends and recvs needed to update data on other ranks are initiated as this +happens when the associated task is queued. The sends and recvs are considered +to be complete when `MPI_Test` returns true and this unlocks any dependencies +they have. Obviously a step cannot complete until all the sends and recvs are +themselves also complete, so the performance of the MPI library and lower +layers is critical. This seems to be most significant, not when we have a lot +of work, or very little, but for intermediary busy steps, when the local work +completes much sooner than the MPI exchanges. + +In SWIFT the enqueuing of tasks, thus send and recvs initiation (using +`MPI_Isend` and `MPI_Irecv`) can happen from all the available threads, but the +polling of `MPI_Test` is done primarily using two queues, but these can steal +work from other queues, and other queues can steal `MPI_Test` calls as well. +Enqueuing and processing can happen at the same time. + +To keep this simple this package uses three threads to simulate all this, a +thread that does the task of initiating the sends and recvs and two threads +that poll for completion of the sends and recvs. All threads run at the same +time. + +The send and recvs themselves are captured from a run of SWIFT when configured +using the configure option `--enable-mpiuse-reports`. When this is enabled +each step of the simulation produces logs for each rank which record when the +MPI interaction was started and when it completed. Other information such as +the ranks involved, the size of the data exchanged, the MPI tags used and +which SWIFT task types were used are also recorded. + +We read a concatenated log of all these outputs for a single step, and try to +use the relative times that the interaction were started as a guide, the +completions are just polled in time completion order until completion really +occurs. It is also possible to just start all the interactions as quickly as +possible for comparisons. To use the program `swiftmpistepsim` you need to select the step of interest (for instance one whose run-time seems dominated by the MPI tasks) and then @@ -18,16 +51,16 @@ run using: mpirun -np <nranks> swiftmpistepsim <step-log> <output-log> ``` which will output timings for the various MPI calls and record a log -for the reproduction in the file `<output-log>`. +for the reproduction in the file `<output-log>`. Note you must use the same +numbers of ranks as the original run of SWIFT. -To simulate SWIFT we use three threads, which run simultaneously, one that -injects the MPI commands, i.e. initiates the interaction using calls to -`MPI_Isend` and `MPI_Irecv`, and two other threads that poll the MPI library -using `MPI_Test` to discover when the exchanges have been completed. +The verbose output and output log can be inspected to see what delays are +driving the elapsed time for the step. Mainly these seem to be outlier +`MPI_Test` calls that take tens of milliseconds. -SWIFT itself uses more threads than this for the injection and polling phases, -but it is not thought to make a large difference. A later development could -explore that... +A script `post-process.py` can be ran on the output log to pair the sends and +recvs across the ranks. This allows the inspection of how well things like +eager exchanges are working and what effect the size of the packets has. --------------------------- -Peter W. Draper 18 Sep 2019. +Peter W. Draper 24 Sep 2019. diff --git a/format.sh b/format.sh new file mode 100755 index 0000000000000000000000000000000000000000..91346334c9b2eaf9fbb343aba44f8a02d866d1ef --- /dev/null +++ b/format.sh @@ -0,0 +1,80 @@ +#!/bin/bash + +# Clang format command, can be overridden using CLANG_FORMAT_CMD. +# We currrently use version 5.0 so any overrides should provide that. +clang=${CLANG_FORMAT_CMD:="clang-format-5.0"} + +# Formatting command +cmd="$clang -style=file $(git ls-files | grep '\.[ch]$')" + +# Test if `clang-format-5.0` works +command -v $clang > /dev/null +if [[ $? -ne 0 ]] +then + echo "ERROR: cannot find $clang" + exit 1 +fi + +# Print the help +function show_help { + echo -e "This script formats SWIFT according to Google style" + echo -e " -h, --help \t Show this help" + echo -e " -t, --test \t Test if SWIFT is well formatted" +} + +# Parse arguments (based on https://stackoverflow.com/questions/192249) +TEST=0 +while [[ $# -gt 0 ]] +do + key="$1" + + case $key in + # print the help and exit + -h|--help) + show_help + exit + ;; + # check if the code is well formatted + -t|--test) + TEST=1 + shift + ;; + # unknown option + *) + echo "Argument '$1' not implemented" + show_help + exit + ;; + esac +done + +# Run the required commands +if [[ $TEST -eq 1 ]] +then + # Note trapping the exit status from both commands in the pipe. Also note + # do not use -q in grep as that closes the pipe on first match and we get + # a SIGPIPE error. + echo "Testing if SWIFT is correctly formatted" + $cmd -output-replacements-xml | grep "<replacement " > /dev/null + status=("${PIPESTATUS[@]}") + + # Trap if first command failed. Note 141 is SIGPIPE, that happens when no + # output + if [[ ${status[0]} -ne 0 ]] + then + echo "ERROR: $clang command failed" + exit 1 + fi + + # Check formatting + if [[ ${status[1]} -eq 0 ]] + then + echo "ERROR: needs formatting" + exit 1 + else + echo "...is correctly formatted" + fi +else + echo "Formatting SWIFT" + $cmd -i +fi diff --git a/mpiuse.c b/mpiuse.c index ed6007a3a7883df053b062f1ce495e7eb73fa52f..6bd7b3116e0910b152e49112be86057bad80a0f6 100644 --- a/mpiuse.c +++ b/mpiuse.c @@ -204,15 +204,18 @@ void mpiuse_dump_logs(int nranks, const char *dumpfile) { fd = fopen(dumpfile, "a"); /* And append our logs. Note log->tic is not necessarily from this - machine, so the conversion to ms may be suspect. */ + * machine, so the conversion to ms may be suspect. We also rebase a + * version to match the expected injection times for this new run. */ size_t nlogs = mpiuse_log_count; + ticks basetics = 0; for (size_t k = 0; k < nlogs; k++) { struct mpiuse_log_entry *log = &mpiuse_log[k]; if (log->rank == myrank && log->endtic > 0) { + if (basetics == 0) basetics = log->tic; fprintf(fd, "%lld %.4f %.4f %.4f %.6f %d %d %d %d %d %d %zd %d %.4f %.6f " "%.6f\n", - log->tic, clocks_from_ticks(log->tic), + log->tic, clocks_from_ticks(log->tic - basetics), clocks_from_ticks(log->injtic - clocks_start_ticks), clocks_from_ticks(log->endtic - clocks_start_ticks), clocks_from_ticks(log->endtic - log->injtic), log->step, diff --git a/post-process.py b/post-process.py new file mode 100755 index 0000000000000000000000000000000000000000..746db3503a9c1f6db7196595c7b4796a99100379 --- /dev/null +++ b/post-process.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python +""" +Usage: + post-process.py [options] output-log + +Match the sends and recvs across the ranks and output a unified log. + +This file is part of SWIFT. + +Copyright (C) 2019 Peter W. Draper (p.w.draper@durham.ac.uk) +All Rights Reserved. + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Lesser General Public License as published +by the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU Lesser General Public License +along with this program. If not, see <http://www.gnu.org/licenses/>. +""" + +import sys +import argparse + +# Handle the command line. +parser = argparse.ArgumentParser(description="Match MPI reports") + +parser.add_argument( + "input", + help="Output log from simulator") +parser.add_argument( + "-v", + "--verbose", + dest="verbose", + help="Verbose output", + default=False, + action="store_true", +) +args = parser.parse_args() +infile = args.input + +# Indices for words in a line. +logticcol=0 +logcol=1 +injcol=2 +endcol=3 +dticcol=4 +stepcol=5 +rankcol=6 +otherrankcol=7 +itypecol=8 +isubtypecol=9 +tagcol=10 +sizecol=11 +ntestcol=12 +sumcol=13 +mincol=14 +maxcol=15 + +# Keyed lines. +sends = {} +recvs = {} + +# Gather keys from the input log. We created dicts with matchable keys +# for when sends start and recvs end. Other pairings are possible... +# Note size of completion recv is negative. +with open(infile, "r") as fp: + for line in fp: + if line[0] == '#': + continue + words = line.split() + if words[itypecol] == "22": + key = words[otherrankcol] + "/" + \ + words[rankcol] + "/" + \ + words[isubtypecol] + "/" + \ + words[tagcol] + "/" + \ + words[sizecol] + if not key in sends: + sends[key] = [line[:-1]] + else: + sends[key].append(line[:-1]) + + elif words[itypecol] == "23": + key = words[rankcol] + "/" + \ + words[otherrankcol] + "/" + \ + words[isubtypecol] + "/" + \ + words[tagcol] + "/" + \ + words[sizecol] + if not key in recvs: + recvs[key] = [line[:-1]] + else: + recvs[key].append(line[:-1]) + +# Now output. Note we could have unmatched recv keys, we don't check for that. +print "# send_logticin send_logtic send_injtic send_endtic send_dtic send_step send_rank send_otherrank send_itype send_isubtype send_tag send_size send_nr_tests send_tsum send_tmin send_tmax recv_logticin recv_logtic recv_injtic recv_endtic recv_dtic recv_step recv_rank recv_otherrank recv_itype recv_isubtype recv_tag recv_size recv_nr_tests recv_tsum recv_tmin recv_tmax" +for key in sends: + if key in recvs: + if len(sends[key]) == 1 and len(recvs[key]) == 1: + print sends[key][0], recvs[key][0] + else: + print "# ERROR: found ", len(sends[key]), "/", len(recvs[key]), " matches for key: ", key, " should be 1/1" + else: + print "# ERROR: missing recv key: ", key + + +sys.exit(0) diff --git a/swiftmpistepsim.c b/swiftmpistepsim.c index b8a660cba292d4dbed9cb556b9e6fd98e6ce52f7..578c22960277b031642dc51a856260bf6d97c07c 100644 --- a/swiftmpistepsim.c +++ b/swiftmpistepsim.c @@ -83,6 +83,8 @@ static void *inject_thread(void *arg) { ticks basetic = reqs_queue[0]->tic; ticks looptics = 0; double deadtime = 0.0; + struct timespec sleep; + sleep.tv_sec = 0; struct timespec sleep; sleep.tv_sec = 0; @@ -130,7 +132,7 @@ static void *inject_thread(void *arg) { ind_req++; /* Set looptics on the first pass. Assumes MPI_Isend and MPI_Irecv are - * equally timed. */ + * equally timed. Note we include a nanosleep, they are slow. */ if (looptics == 0 && usetics) { /* Not well matched, so try a lower value. */ looptics = (getticks() - starttics) / 4; @@ -301,14 +303,14 @@ static void pick_logs(void) { size_t nlogs = mpiuse_nr_logs(); /* Duplicate of logs. */ - reqs_queue = (struct mpiuse_log_entry **)malloc( - sizeof(struct mpiuse_log_entry *) * nlogs); + reqs_queue = (struct mpiuse_log_entry **)calloc( + nlogs, sizeof(struct mpiuse_log_entry *)); nr_reqs = 0; - sends_queue = (struct mpiuse_log_entry **)malloc( - sizeof(struct mpiuse_log_entry *) * nlogs); + sends_queue = (struct mpiuse_log_entry **)calloc( + nlogs, sizeof(struct mpiuse_log_entry *) * nlogs); nr_sends = 0; - recvs_queue = (struct mpiuse_log_entry **)malloc( - sizeof(struct mpiuse_log_entry *) * nlogs); + recvs_queue = (struct mpiuse_log_entry **)calloc( + nlogs, sizeof(struct mpiuse_log_entry *)); nr_recvs = 0; for (int k = 0; k < nlogs; k++) { @@ -331,6 +333,13 @@ static void pick_logs(void) { /* Sort into increasing time. */ qsort(reqs_queue, nr_reqs, sizeof(struct mpiuse_log_entry *), cmp_logs); + + /* Check. */ + for (int k = 0; k < nr_reqs - 1; k++) { + if (reqs_queue[k]->tic > reqs_queue[k + 1]->tic) + message("reqs_queue: %lld > %lld", reqs_queue[k]->tic, + reqs_queue[k + 1]->tic); + } } /** @@ -409,7 +418,8 @@ int main(int argc, char *argv[]) { /* Each rank requires its own queue, so extract them. */ pick_logs(); - /* Time to start time. */ + /* Time to start time. Try to make it synchronous across the ranks. */ + MPI_Barrier(MPI_COMM_WORLD); clocks_set_cpufreq(0); if (myrank == 0) { message("Start of MPI tests"); diff --git a/testdata/EAGLE_25-mpiuse_report-step18302-16ranks-inter-step.dat.gz b/testdata/EAGLE_25-mpiuse_report-step18302-16ranks-inter-step.dat.gz new file mode 100644 index 0000000000000000000000000000000000000000..e73fbb7c19801e2c30ec387d669a5df33db747b8 Binary files /dev/null and b/testdata/EAGLE_25-mpiuse_report-step18302-16ranks-inter-step.dat.gz differ diff --git a/testdata/mpiuse_report-step2.dat b/testdata/EAGLE_6-mpiuse_report-step2-4ranks-small-step.dat similarity index 100% rename from testdata/mpiuse_report-step2.dat rename to testdata/EAGLE_6-mpiuse_report-step2-4ranks-small-step.dat diff --git a/testdata/README b/testdata/README new file mode 100644 index 0000000000000000000000000000000000000000..7db64db49a977a62a8863905bb3490cac9b5d1ab --- /dev/null +++ b/testdata/README @@ -0,0 +1,9 @@ +There are two example SWIFT mpi logging output logs. A small one for checking +program developments and a larger one that exhibits the unwanted behaviour on +the COSMA7 cluster. + +The small dataset: `EAGLE_6-mpiuse_report-step2-4ranks-small-step.dat`, is the +second step of a 4 rank run of the EAGLE_6 low z example. + +The larger dataset: `EAGLE_25-mpiuse_report-step18302-16ranks-inter-step.dat`, +uses the high z EAGLE_25 with 16 ranks. Decompresses to 21M. diff --git a/testdata/mpiuse_report-rank0-step2.dat b/testdata/mpiuse_report-rank0-step2.dat deleted file mode 100644 index 51043d68d599e8c7f0d52bec0a9740fd0ec1f0b1..0000000000000000000000000000000000000000 --- a/testdata/mpiuse_report-rank0-step2.dat +++ /dev/null @@ -1,91 +0,0 @@ -# stic etic dtic step rank otherrank type itype subtype isubtype activation tag size sum -608817 21543633709 0 2 0 2 recv 23 xv 11 1 445 142720 142720 -610727 21543635619 0 2 0 3 recv 23 xv 11 1 453 311296 454016 -612045 21543636937 0 2 0 2 recv 23 xv 11 1 446 219648 673664 -613550 21543638442 0 2 0 3 recv 23 xv 11 1 38 449024 1122688 -615327 21543640219 0 2 0 3 recv 23 xv 11 1 62 282624 1405312 -616285 21543641177 0 2 0 1 recv 23 xv 11 1 177 379648 1784960 -618295 21543643187 0 2 0 3 recv 23 xv 11 1 50 400128 2185088 -619247 21543644139 0 2 0 2 recv 23 xv 11 1 160 94208 2279296 -620661 21543645553 0 2 0 3 recv 23 xv 11 1 49 318720 2598016 -623155 21543648047 0 2 0 2 recv 23 xv 11 1 158 131072 2729088 -625506 21543650398 0 2 0 3 recv 23 xv 11 1 40 416512 3145600 -627165 21543652057 0 2 0 2 recv 23 xv 11 1 156 111360 3256960 -629619 21543654511 0 2 0 3 recv 23 xv 11 1 67 312448 3569408 -631490 21543656382 0 2 0 3 recv 23 xv 11 1 65 292096 3861504 -632856 21543657748 0 2 0 3 recv 23 xv 11 1 465 77696 3939200 -635292 21543660184 0 2 0 2 recv 23 xv 11 1 3 583168 4522368 -636746 21543661638 0 2 0 2 recv 23 xv 11 1 2 107136 4629504 -639251 21543664143 0 2 0 3 recv 23 xv 11 1 130 711424 5340928 -640726 21543665618 0 2 0 1 recv 23 xv 11 1 118 553984 5894912 -642551 21543667443 0 2 0 1 recv 23 xv 11 1 45 328576 6223488 -644510 21543669402 0 2 0 3 recv 23 xv 11 1 192 345984 6569472 -646036 21543670928 0 2 0 1 recv 23 xv 11 1 44 1056768 7626240 -647586 21543672478 0 2 0 3 recv 23 xv 11 1 230 255232 7881472 -649599 21543674491 0 2 0 1 recv 23 xv 11 1 194 143872 8025344 -651388 21543676280 0 2 0 1 recv 23 xv 11 1 202 360832 8386176 -653752 21543678644 0 2 0 1 recv 23 xv 11 1 192 1066112 9452288 -655523 21543680415 0 2 0 2 recv 23 xv 11 1 249 28544 9480832 -657245 21543682137 0 2 0 2 recv 23 xv 11 1 245 83840 9564672 -659152 21543684044 0 2 0 2 recv 23 xv 11 1 244 78720 9643392 -661120 21543686012 0 2 0 1 recv 23 xv 11 1 205 547968 10191360 -664106 21543688998 0 2 0 3 recv 23 xv 11 1 200 368000 10559360 -666953 21543691845 0 2 0 1 recv 23 xv 11 1 137 212608 10771968 -668385 21543693277 0 2 0 3 recv 23 xv 11 1 137 418304 11190272 -669960 21543694852 0 2 0 3 recv 23 xv 11 1 136 199808 11390080 -671970 21543696862 0 2 0 1 recv 23 xv 11 1 57 332160 11722240 -6447305 21549472197 5791782 2 0 2 recv 23 xv 11 0 249 -28544 11693696 -11419335 21554444227 10796180 2 0 2 recv 23 xv 11 0 158 -131072 11562624 -11758915 21554783807 11111329 2 0 3 recv 23 xv 11 0 230 -255232 11307392 -11799763 21554824655 11172598 2 0 2 recv 23 xv 11 0 156 -111360 11196032 -11838133 21554863025 11222806 2 0 3 recv 23 xv 11 0 62 -282624 10913408 -11910866 21554935758 11253621 2 0 2 recv 23 xv 11 0 245 -83840 10829568 -13014189 21556039081 12405372 2 0 2 recv 23 xv 11 0 445 -142720 10686848 -13215819 21556240711 12580527 2 0 2 recv 23 xv 11 0 3 -583168 10103680 -13730903 21556755795 13111656 2 0 2 recv 23 xv 11 0 160 -94208 10009472 -14435788 21557460680 13776636 2 0 2 recv 23 xv 11 0 244 -78720 9930752 -15491583 21558516475 14823198 2 0 3 recv 23 xv 11 0 137 -418304 9512448 -15614696 21558639588 14981840 2 0 3 recv 23 xv 11 0 465 -77696 9434752 -15728921 21558753813 15116876 2 0 2 recv 23 xv 11 0 446 -219648 9215104 -16170362 21559195254 15533616 2 0 2 recv 23 xv 11 0 2 -107136 9107968 -16572275 21559597167 15961548 2 0 3 recv 23 xv 11 0 453 -311296 8796672 -16928290 21559953182 16289039 2 0 3 recv 23 xv 11 0 130 -711424 8085248 -28452779 21571477671 27827273 2 0 3 recv 23 xv 11 0 40 -416512 7668736 -28707936 21571732828 28054184 2 0 1 recv 23 xv 11 0 192 -1066112 6602624 -28977681 21572002573 28364131 2 0 3 recv 23 xv 11 0 38 -449024 6153600 -30058812 21573083704 29440517 2 0 3 recv 23 xv 11 0 50 -400128 5753472 -30160189 21573185081 29539528 2 0 3 recv 23 xv 11 0 49 -318720 5434752 -30309729 21573334621 29639769 2 0 3 recv 23 xv 11 0 136 -199808 5234944 -30514052 21573538944 29849946 2 0 3 recv 23 xv 11 0 200 -368000 4866944 -31055012 21574079904 30410502 2 0 3 recv 23 xv 11 0 192 -345984 4520960 -38056623 21581081515 37427004 2 0 3 recv 23 xv 11 0 67 -312448 4208512 -38085547 21581110439 37454057 2 0 3 recv 23 xv 11 0 65 -292096 3916416 -38835374 21581860266 38189338 2 0 1 recv 23 xv 11 0 44 -1056768 2859648 -40237431 21583262323 39586043 2 0 1 recv 23 xv 11 0 202 -360832 2498816 -40553084 21583577976 39891964 2 0 1 recv 23 xv 11 0 205 -547968 1950848 -40777829 21583802721 40161544 2 0 1 recv 23 xv 11 0 177 -379648 1571200 -40836656 21583861548 40164686 2 0 1 recv 23 xv 11 0 57 -332160 1239040 -40873449 21583898341 40206496 2 0 1 recv 23 xv 11 0 137 -212608 1026432 -41885432 21584910324 0 2 0 2 send 22 tend_part 7 1 38 2328 1028760 -41896300 21584921192 0 2 0 3 send 22 tend_part 7 1 38 2328 1031088 -41942824 21584967716 57392 2 0 2 send 22 tend_part 7 0 38 -2328 1028760 -41950298 21584975190 53998 2 0 3 send 22 tend_part 7 0 38 -2328 1026432 -43257053 21586281945 42607454 2 0 1 recv 23 xv 11 0 194 -143872 882560 -43529999 21586554891 42887448 2 0 1 recv 23 xv 11 0 45 -328576 553984 -43591111 21586616003 42950385 2 0 1 recv 23 xv 11 0 118 -553984 0 -48599962 21591624854 0 2 0 2 send 22 tend_part 7 1 138 2712 2712 -48619658 21591644550 0 2 0 1 send 22 tend_part 7 1 138 2712 5424 -48642246 21591667138 0 2 0 3 send 22 tend_part 7 1 138 2712 8136 -48695836 21591720728 53590 2 0 3 send 22 tend_part 7 0 138 -2712 5424 -48704250 21591729142 104288 2 0 2 send 22 tend_part 7 0 138 -2712 2712 -48924980 21591949872 305322 2 0 1 send 22 tend_part 7 0 138 -2712 0 -48963599 21591988491 0 2 0 1 send 22 tend_part 7 1 192 6936 6936 -48968850 21591993742 5251 2 0 1 send 22 tend_part 7 0 192 -6936 0 -48990086 21592014978 0 2 0 3 send 22 tend_part 7 1 192 6936 6936 -48995836 21592020728 5750 2 0 3 send 22 tend_part 7 0 192 -6936 0 -## -## Number of requests: 42 -## Maximum request size: 1.0167 (MB) -## Sum of all requests: 11.2046 (MB) -## Mean of all requests: 0.2668 (MB) -## diff --git a/testdata/mpiuse_report-rank1-step2.dat b/testdata/mpiuse_report-rank1-step2.dat deleted file mode 100644 index 964543c8084bab9769faae1aec76523160064cc7..0000000000000000000000000000000000000000 --- a/testdata/mpiuse_report-rank1-step2.dat +++ /dev/null @@ -1,57 +0,0 @@ -# stic etic dtic step rank otherrank type itype subtype isubtype activation tag size sum -12953800 21558567216 0 2 1 3 recv 23 tend_part 7 1 32 1368 1368 -12964804 21558578220 0 2 1 0 recv 23 tend_part 7 1 138 2712 4080 -12970634 21558584050 0 2 1 0 recv 23 tend_part 7 1 192 6936 11016 -12983064 21558596480 0 2 1 2 recv 23 xv 11 1 321 310784 321800 -12985183 21558598599 0 2 1 3 recv 23 xv 11 1 440 876032 1197832 -12986818 21558600234 0 2 1 3 recv 23 xv 11 1 403 171136 1368968 -12989387 21558602803 0 2 1 2 recv 23 xv 11 1 395 487040 1856008 -12993612 21558607028 0 2 1 2 recv 23 xv 11 1 383 146048 2002056 -12998091 21558611507 0 2 1 2 recv 23 xv 11 1 322 235904 2237960 -18459228 21564072644 0 2 1 0 send 22 xv 11 1 192 1066112 3304072 -23740217 21569353633 0 2 1 0 send 22 xv 11 1 44 1056768 4360840 -25940939 21571554355 0 2 1 0 send 22 xv 11 1 205 547968 4908808 -28186716 21573800132 0 2 1 0 send 22 xv 11 1 177 379648 5288456 -29239247 21574852663 0 2 1 3 send 22 xv 11 1 453 282752 5571208 -30275585 21575889001 17281973 2 1 2 recv 23 xv 11 0 383 -146048 5425160 -30291458 21575904874 0 2 1 0 send 22 xv 11 1 202 360832 5785992 -33012165 21578625581 0 2 1 0 send 22 xv 11 1 57 332160 6118152 -33756463 21579369879 20771280 2 1 3 recv 23 xv 11 0 440 -876032 5242120 -33892572 21579505988 20903185 2 1 2 recv 23 xv 11 0 395 -487040 4755080 -33925103 21579538519 0 2 1 0 send 22 xv 11 1 137 212608 4967688 -34725100 21580338516 0 2 1 0 send 22 xv 11 1 194 143872 5111560 -36049973 21581663389 0 2 1 0 send 22 xv 11 1 45 328576 5440136 -36265488 21581878904 17806260 2 1 0 send 22 xv 11 0 192 -1066112 4374024 -38193925 21583807341 0 2 1 3 send 22 xv 11 1 56 472064 4846088 -38822832 21584436248 15082615 2 1 0 send 22 xv 11 0 44 -1056768 3789320 -40220268 21585833684 12033552 2 1 0 send 22 xv 11 0 177 -379648 3409672 -40540832 21586154248 7528667 2 1 0 send 22 xv 11 0 57 -332160 3077512 -40705244 21586318660 6780141 2 1 0 send 22 xv 11 0 137 -212608 2864904 -40722144 21586335560 10430686 2 1 0 send 22 xv 11 0 202 -360832 2504072 -40735977 21586349393 14795038 2 1 0 send 22 xv 11 0 205 -547968 1956104 -42881032 21588494448 0 2 1 0 send 22 xv 11 1 118 553984 2510088 -43243043 21588856459 5049118 2 1 3 send 22 xv 11 0 56 -472064 2038024 -43254760 21588868176 14015513 2 1 3 send 22 xv 11 0 453 -282752 1755272 -43431687 21589045103 7381714 2 1 0 send 22 xv 11 0 45 -328576 1426696 -43441434 21589054850 8716334 2 1 0 send 22 xv 11 0 194 -143872 1282824 -43447599 21589061015 566567 2 1 0 send 22 xv 11 0 118 -553984 728840 -44195173 21589808589 0 2 1 3 send 22 xv 11 1 59 183552 912392 -44432889 21590046305 0 2 1 3 send 22 xv 11 1 47 46720 959112 -44593620 21590207036 160731 2 1 3 send 22 xv 11 0 47 -46720 912392 -44602907 21590216323 407734 2 1 3 send 22 xv 11 0 59 -183552 728840 -44639981 21590253397 31656917 2 1 2 recv 23 xv 11 0 321 -310784 418056 -44960058 21590573474 31961967 2 1 2 recv 23 xv 11 0 322 -235904 182152 -48461460 21594074876 35474642 2 1 3 recv 23 xv 11 0 403 -171136 11016 -48919281 21594532697 35954477 2 1 0 recv 23 tend_part 7 0 138 -2712 8304 -48984643 21594598059 36030843 2 1 3 recv 23 tend_part 7 0 32 -1368 6936 -49409002 21595022418 36438368 2 1 0 recv 23 tend_part 7 0 192 -6936 0 -56492967 21602106383 0 2 1 2 send 22 tend_part 7 1 359 2520 2520 -56507501 21602120917 14534 2 1 2 send 22 tend_part 7 0 359 -2520 0 -56512908 21602126324 0 2 1 3 send 22 tend_part 7 1 359 2520 2520 -56518112 21602131528 5204 2 1 3 send 22 tend_part 7 0 359 -2520 0 -## -## Number of requests: 25 -## Maximum request size: 1.0167 (MB) -## Sum of all requests: 7.8303 (MB) -## Mean of all requests: 0.3132 (MB) -## diff --git a/testdata/mpiuse_report-rank2-step2.dat b/testdata/mpiuse_report-rank2-step2.dat deleted file mode 100644 index b5c0e6e44f823aa741d22eacd87fddb78870fcfd..0000000000000000000000000000000000000000 --- a/testdata/mpiuse_report-rank2-step2.dat +++ /dev/null @@ -1,53 +0,0 @@ -# stic etic dtic step rank otherrank type itype subtype isubtype activation tag size sum -987251 21546628055 0 2 2 3 recv 23 tend_part 7 1 32 1368 1368 -1025615 21546666419 0 2 2 0 recv 23 tend_part 7 1 38 2328 3696 -1029326 21546670130 0 2 2 0 recv 23 tend_part 7 1 138 2712 6408 -1038964 21546679768 0 2 2 1 recv 23 tend_part 7 1 359 2520 8928 -1304769 21546945573 0 2 2 3 send 22 xv 11 1 444 98816 107744 -1404115 21547044919 0 2 2 3 send 22 xv 11 1 2 107136 214880 -1406430 21547047234 0 2 2 0 send 22 xv 11 1 2 107136 322016 -1892700 21547533504 0 2 2 0 send 22 xv 11 1 446 219648 541664 -3170516 21548811320 0 2 2 1 send 22 xv 11 1 395 487040 1028704 -3551697 21549192501 0 2 2 0 send 22 xv 11 1 3 583168 1611872 -3608675 21549249479 0 2 2 1 send 22 xv 11 1 383 146048 1757920 -3722238 21549363042 0 2 2 0 send 22 xv 11 1 445 142720 1900640 -4474542 21550115346 0 2 2 1 send 22 xv 11 1 322 235904 2136544 -4660091 21550300895 0 2 2 1 send 22 xv 11 1 321 310784 2447328 -4802628 21550443432 0 2 2 0 send 22 xv 11 1 249 28544 2475872 -4947390 21550588194 144762 2 2 0 send 22 xv 11 0 249 -28544 2447328 -7943938 21553584742 6537508 2 2 0 send 22 xv 11 0 2 -107136 2340192 -7961657 21553602461 6068957 2 2 0 send 22 xv 11 0 446 -219648 2120544 -7971824 21553612628 4420127 2 2 0 send 22 xv 11 0 3 -583168 1537376 -8524640 21554165444 0 2 2 0 send 22 xv 11 1 158 131072 1668448 -8974356 21554615160 0 2 2 0 send 22 xv 11 1 156 111360 1779808 -9378345 21555019149 0 2 2 0 send 22 xv 11 1 160 94208 1874016 -9846859 21555487663 0 2 2 0 send 22 xv 11 1 245 83840 1957856 -10082623 21555723427 6360385 2 2 0 send 22 xv 11 0 445 -142720 1815136 -10425140 21556065944 0 2 2 3 send 22 xv 11 1 1 64512 1879648 -11258279 21556899083 1411420 2 2 0 send 22 xv 11 0 245 -83840 1795808 -12594280 21558235084 2169140 2 2 3 send 22 xv 11 0 1 -64512 1731296 -12624810 21558265614 11220695 2 2 3 send 22 xv 11 0 2 -107136 1624160 -12806376 21558447180 3428031 2 2 0 send 22 xv 11 0 160 -94208 1529952 -12864222 21558505026 3889866 2 2 0 send 22 xv 11 0 156 -111360 1418592 -12923342 21558564146 11618573 2 2 3 send 22 xv 11 0 444 -98816 1319776 -13069781 21558710585 4545141 2 2 0 send 22 xv 11 0 158 -131072 1188704 -13360446 21559001250 0 2 2 3 send 22 xv 11 1 30 79232 1267936 -13833681 21559474485 0 2 2 3 send 22 xv 11 1 44 64128 1332064 -13842742 21559483546 9061 2 2 3 send 22 xv 11 0 44 -64128 1267936 -14347042 21559987846 0 2 2 0 send 22 xv 11 1 244 78720 1346656 -14447458 21560088262 100416 2 2 0 send 22 xv 11 0 244 -78720 1267936 -14549561 21560190365 1189115 2 2 3 send 22 xv 11 0 30 -79232 1188704 -29619186 21575259990 26448670 2 2 1 send 22 xv 11 0 395 -487040 701664 -29973620 21575614424 26364945 2 2 1 send 22 xv 11 0 383 -146048 555616 -30232105 21575872909 25757563 2 2 1 send 22 xv 11 0 322 -235904 319712 -30884935 21576525739 26224844 2 2 1 send 22 xv 11 0 321 -310784 8928 -41905408 21587546212 40879793 2 2 0 recv 23 tend_part 7 0 38 -2328 6600 -48456241 21594097045 47468990 2 2 3 recv 23 tend_part 7 0 32 -1368 5232 -48605647 21594246451 47576321 2 2 0 recv 23 tend_part 7 0 138 -2712 2520 -56499592 21602140396 55460628 2 2 1 recv 23 tend_part 7 0 359 -2520 0 -## -## Number of requests: 23 -## Maximum request size: 0.5562 (MB) -## Sum of all requests: 3.0355 (MB) -## Mean of all requests: 0.1320 (MB) -## diff --git a/testdata/mpiuse_report-rank3-step2.dat b/testdata/mpiuse_report-rank3-step2.dat deleted file mode 100644 index 2c17b6d4bcea8e7ef7184f1772c65390d8a86e46..0000000000000000000000000000000000000000 --- a/testdata/mpiuse_report-rank3-step2.dat +++ /dev/null @@ -1,71 +0,0 @@ -# stic etic dtic step rank otherrank type itype subtype isubtype activation tag size sum -8154566 21553806876 0 2 3 0 recv 23 tend_part 7 1 38 2328 2328 -8241482 21553893792 0 2 3 0 recv 23 tend_part 7 1 138 2712 5040 -8252462 21553904772 0 2 3 1 recv 23 xv 11 1 59 183552 188592 -8255236 21553907546 0 2 3 2 recv 23 xv 11 1 44 64128 252720 -8265527 21553917837 0 2 3 0 recv 23 tend_part 7 1 192 6936 259656 -8267192 21553919502 0 2 3 1 recv 23 xv 11 1 47 46720 306376 -8281049 21553933359 0 2 3 1 recv 23 tend_part 7 1 359 2520 308896 -8285576 21553937886 0 2 3 1 recv 23 xv 11 1 56 472064 780960 -8290638 21553942948 0 2 3 1 recv 23 xv 11 1 453 282752 1063712 -8312534 21553964844 0 2 3 2 recv 23 xv 11 1 444 98816 1162528 -8314496 21553966806 0 2 3 2 recv 23 xv 11 1 30 79232 1241760 -8318186 21553970496 0 2 3 2 recv 23 xv 11 1 2 107136 1348896 -8321900 21553974210 0 2 3 2 recv 23 xv 11 1 1 64512 1413408 -8551050 21554203360 0 2 3 0 send 22 xv 11 1 465 77696 1491104 -9240343 21554892653 0 2 3 0 send 22 xv 11 1 62 282624 1773728 -9516048 21555168358 0 2 3 0 send 22 xv 11 1 230 255232 2028960 -10222671 21555874981 1910137 2 3 2 recv 23 xv 11 0 444 -98816 1930144 -11396780 21557049090 3074880 2 3 2 recv 23 xv 11 0 1 -64512 1865632 -11919637 21557571947 0 2 3 0 send 22 xv 11 1 130 711424 2577056 -12416538 21558068848 496901 2 3 0 send 22 xv 11 0 130 -711424 1865632 -12449098 21558101408 0 2 3 1 send 22 xv 11 1 403 171136 2036768 -12525243 21558177553 4207057 2 3 2 recv 23 xv 11 0 2 -107136 1929632 -13886887 21559539197 0 2 3 0 send 22 xv 11 1 453 311296 2240928 -13993363 21559645673 5738127 2 3 2 recv 23 xv 11 0 44 -64128 2176800 -14353159 21560005469 6038663 2 3 2 recv 23 xv 11 0 30 -79232 2097568 -14361689 21560013999 0 2 3 0 send 22 xv 11 1 137 418304 2515872 -15260362 21560912672 898673 2 3 0 send 22 xv 11 0 137 -418304 2097568 -16761542 21562413852 0 2 3 0 send 22 xv 11 1 200 368000 2465568 -17561805 21563214115 0 2 3 0 send 22 xv 11 1 136 199808 2665376 -17768823 21563421133 0 2 3 1 send 22 xv 11 1 440 876032 3541408 -18876067 21564528377 0 2 3 0 send 22 xv 11 1 192 345984 3887392 -19483556 21565135866 0 2 3 0 send 22 xv 11 1 40 416512 4303904 -20583795 21566236105 0 2 3 0 send 22 xv 11 1 38 449024 4752928 -21166982 21566819292 0 2 3 0 send 22 xv 11 1 49 318720 5071648 -21367099 21567019409 7480212 2 3 0 send 22 xv 11 0 453 -311296 4760352 -21428196 21567080506 11912148 2 3 0 send 22 xv 11 0 230 -255232 4505120 -21489677 21567141987 12249334 2 3 0 send 22 xv 11 0 62 -282624 4222496 -21545174 21567197484 12994124 2 3 0 send 22 xv 11 0 465 -77696 4144800 -27653028 21573305338 10891486 2 3 0 send 22 xv 11 0 200 -368000 3776800 -27725141 21573377451 10163336 2 3 0 send 22 xv 11 0 136 -199808 3576992 -27785208 21573437518 8909141 2 3 0 send 22 xv 11 0 192 -345984 3231008 -27844800 21573497110 8361244 2 3 0 send 22 xv 11 0 40 -416512 2814496 -27904150 21573556460 7320355 2 3 0 send 22 xv 11 0 38 -449024 2365472 -29544767 21575197077 0 2 3 0 send 22 xv 11 1 50 400128 2765600 -29899938 21575552248 355171 2 3 0 send 22 xv 11 0 50 -400128 2365472 -29964381 21575616691 8797399 2 3 0 send 22 xv 11 0 49 -318720 2046752 -31237607 21576889917 0 2 3 0 send 22 xv 11 1 67 312448 2359200 -32076277 21577728587 23785639 2 3 1 recv 23 xv 11 0 453 -282752 2076448 -32997616 21578649926 20548518 2 3 1 send 22 xv 11 0 403 -171136 1905312 -33045540 21578697850 15276717 2 3 1 send 22 xv 11 0 440 -876032 1029280 -35555962 21581208272 0 2 3 0 send 22 xv 11 1 65 292096 1321376 -37724703 21583377013 6487096 2 3 0 send 22 xv 11 0 67 -312448 1008928 -38038978 21583691288 2483016 2 3 0 send 22 xv 11 0 65 -292096 716832 -39312145 21584964455 31026569 2 3 1 recv 23 xv 11 0 56 -472064 244768 -41913009 21587565319 33758443 2 3 0 recv 23 tend_part 7 0 38 -2328 242440 -44653176 21590305486 36385984 2 3 1 recv 23 xv 11 0 47 -46720 195720 -44672181 21590324491 36419719 2 3 1 recv 23 xv 11 0 59 -183552 12168 -48439936 21594092246 0 2 3 2 send 22 tend_part 7 1 32 1368 13536 -48448809 21594101119 0 2 3 1 send 22 tend_part 7 1 32 1368 14904 -48521474 21594173784 81538 2 3 2 send 22 tend_part 7 0 32 -1368 13536 -48529891 21594182201 81082 2 3 1 send 22 tend_part 7 0 32 -1368 12168 -48659245 21594311555 40417763 2 3 0 recv 23 tend_part 7 0 138 -2712 9456 -48993631 21594645941 40728104 2 3 0 recv 23 tend_part 7 0 192 -6936 2520 -56516622 21602168932 48235573 2 3 1 recv 23 tend_part 7 0 359 -2520 0 -## -## Number of requests: 32 -## Maximum request size: 0.8354 (MB) -## Sum of all requests: 7.2695 (MB) -## Mean of all requests: 0.2272 (MB) -##