Make file locations changeable upon restart
A lot of subgrid modules depend on external data files, e.g. the cooling tables. The path to these files is a parameter in the parameter file that gets read during the initialisation of the corresponding subgrid properties. These paths can be both relative and absolute, as long as the path is accessible from the working directory where SWIFT runs.
When a restart file gets dumped, what happens to the path is module dependent. Most (if not all) subgrid modules currently just dump the path to the restart file as it is. Upon restart, the path is then simply read back.
There are two problems with this approach:
- if a path is dumped and then read back, it cannot be changed upon restart
- it is basically impossible to know what files are necessary to restart a run without carefully inspecting the parameter file
Of course, the whole restart mechanism is meant as a way to temporarily stop an ongoing run in one location, so all of this should not matter. I have however upon multiple occasions had to move runs from one disk to another or even from one machine to another (sometimes this does work!) and then this can become very painful. If a path is provided as an absolute path in the parameter file, then it can become impossible to move the run to another machine without changing either the restart file or the code. And figuring out which files need to be transferred in order to make a restart possible can be very tedious if not all files are present in the working directory (I personally prefer making symbolic links to all external data in the working directory).
It would therefore be very good if we could provide a general mechanism to make it possible to change file paths upon restart. One possible solution is to make all subgrid modules read file path parameters again upon restart, as is already done for some other parameters. This would probably require a lot of refactoring. A second option (which I personally prefer) would be to add this mechanism directly to the restart mechanism itself:
- we provide two new functions,
restart_dump_file_path()
andrestart_read_file_path()
that can be called during regular restart writing/reading to handle file paths. - for every regular restart file, we could produce a second
.yml
file that simply lists all file paths that were dumped. For maximum convenience, these paths should be converted into absolute paths. - upon restart, we read the paths from this
.yml
file, making it possible to change them
In addition to making it possible and easy to change file paths, the file path .yml
file would then also expose all external data files in one convenient location, so that you would not need to read through the entire parameter file to find those.