Restarts at runtime update

Use a single runtime across all ranks to avoid exact timing issues.

Put into collectgroup to avoid an extra synchronization point.

@matthieu assigned to John for further testing, but you will need to merge it.

Thanks Peter. I'll merge this into the code I'm using for the 50Mpc boxes and see if we have any more problems.

Ah, clever trick! Thanks.

I've used this branch to run the extra 50Mpc boxes which Josh identified as having better sizes. I don't think I can use it for the original 8 runs because it changes the format of the restart files and I don't want to restart from z=127.

Only one of those 8 runs had crashed so I've merged in the extra barrier after writing restarts (35eaa463) and restarted it without the fix from this merge request. Any of those 8 might fail again but I don't think they can corrupt their restart files at least.

The 50Mpc box that crashed has crashed again, so I do need to apply this fix to the older runs.

Can I make restarting work by moving runtime out of the engine struct? I was thinking of just having 'float engine_runtime' in engine.c, 'extern float engine_runtime' in engine.h and then changing e.runtime to engine_runtime elsewhere. Would that work as a temporary fix?

Yes, don't see why that wouldn't also work.

The jobs haven't crashed but I guess we want to have them run for a few more cycles to check that this was indeed it.

I take it we now think this is ready for merging?

For me, yes. Was just waiting for John's feedback if he has any.

The only crashes I've had this week were the runs which I hadn't updated with this fix, so it looks ok to me.

merged

mentioned in commit 04c82352

Thanks, accepting. Current restarters will need to freeze at the previous version.