Add restart facility

We need to be able to restart SWIFT for running on time limited queues and to deal with machine downtimes.

The basics of this approach are that:

• the engine struct references all the data that we need

• we will only allow restarts on the same architecture with the same numbers of MPI ranks, threads and parameters

• information that can be reconstructed should not be saved (cells/tasks).

To support this we now have various dump/restore functions that manage their structs as necessary (these are those that are allocated, rather than are statically referenced). It is assumed that specialist knowledge is held in these functions. All these are called, recursively as necessary to write a binary stream for each rank.

This is fragile in that the order of the dump sequence cannot be changed and new struct members or struct reorganization will break the restart. The necessary work to recover and restart looks a little piecemeal, i.e. it works and is not designed.

The dump format is a byte stream with sections for each struct. These are started with a header block consisting of the length of the data and a label (20 chars) followed by the data itself. The lengths of the data are checked against the sizes of the current structs, but the labels are just symbolic and may be used in the future.

Control of restart files is provided by the parameters:

Restarts:
  enable:      1        # (Optional) whether to enable dumping restarts at fixed intervals.
  onexit:      0        # (Optional) whether to dump restarts on exit (*needs enable*)
  subdir: restart       # (Optional) name of subdirectory for restart files.
  basename: swift       # (Optional) prefix used in naming restart files.
  delta_hours: 6.0      # (Optional) decimal hours between dumps of restart files.
  stop_steps:  100      # (Optional) how many steps to process before checking if the <subdir>/stop file exists. 
                          When present the application will attempt to exit early, dumping restart files first.

and is activated by the flag -r (most command-line options are ignored, but the parameter file is still required, that provides where to find the restart files). Output files are appended to on restart, this will not always make sense as the restart can happen sometime before the time when the application exits.

SWIFT can now be stopped by touching the file stop in the restarts directory. This will dump the restart files, regardless of the onexit setting, although if enable is false, that will be respected. The presence of this file is only checking every stop_steps steps (currently every 100).

Jobs to do:

• keep backup copies of the restart files
• decide how to poll the stop file.

Currently failing on some restarts with self-gravity and stars on the next repartition. Command:

mpirun -np 4 ../swift_mpi -G -S -s -t 8 -n 32 -PDomainDecomposition:trigger:5 eagle_6.yml

followed by:

mpirun -np 4 ../swift_mpi -G -S -s -t 8 -n 64 -PDomainDecomposition:trigger:5 eagle_6.yml -r

added 1 commit

• a9b7c600 - Need to make sure xparts are not freed when not used as well

Compare with previous version

added 3 commits

• 2ec7eca0 - Add function to convert millisec into ticks
• 0c470232 - Never use simple ticks in clocks, dumps require at least some relationship to actual time
• c4f7ae0a - Make restart files dump after a number of hours from the start and synchronize…

Compare with previous version

Suspect the bug was related to an issue estimating the number of tasks. On restart this could blow up.

added 1 commit

• 271c51d3 - We cannot derive the estimated tasks per cell on restart so save the old value and reuse that

Compare with previous version

added 1 commit

• 9a6d9757 - Remove debugging output, tidy up output and make sure we return restart files in…

Compare with previous version

unmarked as a Work In Progress

changed the description

@matthieu, @nnrw56. This is looking like a serious effort now, so is a good time for any additional ideas.

added 1 commit

• cfa0968d - Sugar of various kinds, new parameters to control restart file dumping, better…

Compare with previous version

changed the description

I will have a more complete look over the weekend but it looks like it is doing the job.

From the space, you only dump the part/xpart/gpart/spart arrays. Is that enough?

To stop the application, many codes I have used scan the directory where they are running and if they spot an (empty) file called "stop" they then proceed with writing a check-point and exit.

added 1 commit

• dc0aaaa6 - Add check for <Restarts:subdir>/stop file every no. of steps.

Compare with previous version

changed the description

added 1 commit

• 1ba3366a - Store the length of the packet written into the stream and add a label

Compare with previous version

changed the description

changed the description

added 1 commit

• 0ae5c4b3 - Allow parameters controlling restarts to be overridden by the runtime parameters

Compare with previous version

added 6 commits

• 0ae5c4b3...efd4db84 - 5 commits from branch master
• 93542fc8 - Merge remote-tracking branch 'origin/master' into restart-structs

Compare with previous version

added 1 commit

• 45945643 - Save the snapshot counter so we restart correctly

Compare with previous version

added 1 commit

• 6d62ad47 - Formatting

Compare with previous version

I think this is good enough for someone else to test now, so re-assigning to @matthieu.

assigned to @matthieu

added 1 commit

• 464b2120 - Add end marker to restart files, should make it possible to check for truncation

Compare with previous version

Just had a weird file truncation report when restarting EAGLE_50x8. I may have killed the job at a bad moment, so checking that again.

added 1 commit

• e6ba3187 - Actually write the END marker

Compare with previous version

I'll catch up on this soon.

Isn't EAGLE_50x8 much too big for cosma-6?

No, is running on 60 nodes, that is with SPH and Gravity. Had to remove the Stars, never managed to get that up and running. Oh, I see, EAGLE_50x8 means 8 EAGLE_50's, not 8x8x8, that would be too much!

Good, I had killed the application during the dump, and it worked the second time. The biggest dump took 75 secs and the application restarted (to the time it reported a new step) 88 seconds. Those numbers look good to me.

Last week you mentioned that you were not very happy with the code checking the disks for a stop file. Looks like you have come to terms with it but what was your concern?

Just that I didn't want to touch the filesystem for a non-existent file on each step. That would mean each rank waiting on the filesystem metadata server to send back packets (this cannot be cached locally). So the compromise is just to do this much less often than that.

Would reading into /tmp/ be a better choice, provided people have direct access to the nodes?

That would be messy, as you say the user would require access to the compute node, that isn't a given, plus you need to do some MPI synchronization to make sure all the ranks came together. Are you concerned that 100 steps is potentially too long for a forced stop?

Is there any way to open/read the file asynchronously, i.e. emit the read at the start of the step and only check the value at the end?

I don't know of any asynchronous calls to stat()-like functions, so the only way to do that would be to use a thread to do the polling (I guess you could make that lightweight by checking the content of the file, rather than its existence, that could probably be based on a select()). Sounds like a lot of effort.

How about we start a pthread that just does

while (1) {
  stat(...);
  sleep(...);
}

And then atomically sets some global variable if the file is found?

Indeed, just seems messy, and we'd still need to wait until the end of the step to act on this and that could be long running. It we want a timed interval for checking we could use some delta-tick alarm in the step logic, instead of counting steps, like I do for dumping restarts.

Messy: well, we've done worse

The main advantage though is that the latency of the stat call is kept out of the main thread, i.e. we never block on this.

marked as a Work In Progress

changed the description

To me, this is ready to go. The only question is whether you want to add the backup of files and checksum at this stage.

I want to add keeping backup copies and think some more about the stop file, so have temporarily marked it as WIP. Still not sure about the checksum, I'll need to see how much overhead it adds, plus I doubt it will give much meaningful protection.

Merge what you've got and keep working on it? This would force people to respect the restarts if they add stuff to the engine and/or space.

unmarked as a Work In Progress

OK, that makes sense, we want to mainline this code as soon as possible.

@matthieu go ahead and merge this. Please leave the branch for updates.

mentioned in issue #399 (closed)

Merged! Thanks.

merged