SWIFTsim issueshttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/issues2023-11-30T16:58:48Zhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/issues/877Investigate whether MPI_Continuation can be useful2023-11-30T16:58:48ZMatthieu SchallerInvestigate whether MPI_Continuation can be usefulMaybe for the proxy exchange of cells that can't be thread parallelised?Maybe for the proxy exchange of cells that can't be thread parallelised?Matthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/issues/853Erasure encoded restart files.2023-03-28T15:33:17ZPeter W. DraperErasure encoded restart files.Fun idea from Alastair Basden. To protect against the loss of single OSTs, which can happen on the COSMA snap file stores,
why not write out the restart files using an erasure encoded technique. This is the way that RAID devices can
oper...Fun idea from Alastair Basden. To protect against the loss of single OSTs, which can happen on the COSMA snap file stores,
why not write out the restart files using an erasure encoded technique. This is the way that RAID devices can
operate, but working at the level of files.
So instead of writing one restart file, we write let's say 6 files and we arrange to erasure encode these so that the loss
of one file isn't fatal as the missing data can be reconstructed from the remaining 5.
Could be code out there that does this already, but clearly we would need a lot of effort to make sure this worked and
didn't impact performance too much. It would also have a 20% overhead, which is better than a simple duplicate.
Also note we would need to make sure each file was written to a different OST.Peter W. DraperPeter W. Draper2099-12-31https://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/issues/821Explore HDF5's new subfiling VFD interface2024-01-17T13:02:02ZMatthieu SchallerExplore HDF5's new subfiling VFD interfaceFrom version 1.13.2 hdf5 proposes a new interface allowing to split data over multiple files.
See https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.13/hdf5-1.13.2/src/hdf5-1.13.2-RELEASE.txt
Their promise:
```
By allowing a co...From version 1.13.2 hdf5 proposes a new interface allowing to split data over multiple files.
See https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.13/hdf5-1.13.2/src/hdf5-1.13.2-RELEASE.txt
Their promise:
```
By allowing a configurable stripe size, number of I/O concentrators and
method for selecting MPI ranks as I/O concentrators, the Subfiling VFD
aims to enable an HDF5 application to find a middle ground between the
single shared file and file-per-process approaches to parallel file I/O
for the particular machine the application is running on. In general, the
goal is to avoid some of the complexity of the file-per-process approach
while also minimizing the locking issues of the single shared file approach
on a parallel file system.
```
This sounds like something halfway between our current two i/o strategies and could be more generic
and more efficient.
One for the rainy days...Rob McGibbonRob McGibbonhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/issues/798Make file locations changeable upon restart2022-04-08T12:12:40ZBert VandenbrouckeMake file locations changeable upon restartA lot of subgrid modules depend on external data files, e.g. the cooling tables. The path to these files is a parameter in the parameter file that gets read during the initialisation of the corresponding subgrid properties. These paths c...A lot of subgrid modules depend on external data files, e.g. the cooling tables. The path to these files is a parameter in the parameter file that gets read during the initialisation of the corresponding subgrid properties. These paths can be both relative and absolute, as long as the path is accessible from the working directory where SWIFT runs.
When a restart file gets dumped, what happens to the path is module dependent. Most (if not all) subgrid modules currently just dump the path to the restart file as it is. Upon restart, the path is then simply read back.
There are two problems with this approach:
- if a path is dumped and then read back, it cannot be changed upon restart
- it is basically impossible to know what files are necessary to restart a run without carefully inspecting the parameter file
Of course, the whole restart mechanism is meant as a way to temporarily stop an ongoing run in one location, so all of this should not matter. I have however upon multiple occasions had to move runs from one disk to another or even from one machine to another (sometimes this does work!) and then this can become very painful. If a path is provided as an absolute path in the parameter file, then it can become impossible to move the run to another machine without changing either the restart file or the code. And figuring out which files need to be transferred in order to make a restart possible can be very tedious if not all files are present in the working directory (I personally prefer making symbolic links to all external data in the working directory).
It would therefore be very good if we could provide a general mechanism to make it possible to change file paths upon restart. One possible solution is to make all subgrid modules read file path parameters again upon restart, as is already done for some other parameters. This would probably require a lot of refactoring. A second option (which I personally prefer) would be to add this mechanism directly to the restart mechanism itself:
- we provide two new functions, `restart_dump_file_path()` and `restart_read_file_path()` that can be called during regular restart writing/reading to handle file paths.
- for every regular restart file, we could produce a second `.yml` file that simply lists all file paths that were dumped. For maximum convenience, these paths should be converted into absolute paths.
- upon restart, we read the paths from this `.yml` file, making it possible to change them
In addition to making it possible and easy to change file paths, the file path `.yml` file would then also expose all external data files in one convenient location, so that you would not need to read through the entire parameter file to find those.
@matthieu @pdraper Any thoughts on this?Bert VandenbrouckeBert Vandenbrouckehttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/issues/682Spawn new black holes instead of converting gas particles?2022-07-26T12:56:03ZYannick BahéSpawn new black holes instead of converting gas particles?Should we (optionally, at least) change the way in which black holes are seeded? At the moment, we convert an entire gas particle to a black hole which, however, initially has a true (subgrid) mass that is potentially much lower than the...Should we (optionally, at least) change the way in which black holes are seeded? At the moment, we convert an entire gas particle to a black hole which, however, initially has a true (subgrid) mass that is potentially much lower than the gas particle mass. Instead, we could spawn a new particle (analogous to the procedure for gas particle splitting) with initially zero dynamical mass and seed subgrid mass. With the new nibbling implementation, this particle would then immediately nibble mass from (all) its gas neighbours to adjust its dynamical mass to its subgrid mass.
Advantages:
- Arguably more realistic treatment of low-mass black holes (proper dynamical mass)
- Less instantaneous gas loss when a BH is seeded (at EAGLE std-res, we typically remove 12x as much gas mass as the seed BH subgrid mass)
- Arguably more "SPH-philosophically correct", by drawing seed mass from all SPH neighbours instead of just the one seeding particle.
- More flexibility in where the new BH is born, it does not have to be at the exact location of a gas particle (we have seen that this tends to cause, probably artificial, accretion bursts immediately after seeding)
- Would work equally well for BH seed masses lower or higher than the (typical) gas particle mass.
- Essentially makes dynamical == subgrid mass for all black holes, eliminates analysis confusion / errors (dynamical masses are the "standard" quantity, but less physically meaningful).
Disadvantages:
- Black hole particles with m_dyn << m_gas can cause artificial energy transfer to BHs. However, low-mass black holes are anyway subject to repositioning (although not necessarily with aggressive velocity threshold).
- No guarantee that the new BH has (sufficiently many) gas neighbours, may therefore not be able to grow its dynamical mass. One could argue that we should never seed BHs in such a situation in the first place...?
- Creates more particles -- but typically n_BH << n_gas, so small effect compared to particle increase through gas splitting.
- Would probably need some care to make sure that (temporarily) zero particle mass does not create surprises
- May have unintentional side effects
- Human time needed for implementation and testing
Just putting it out there for consideration / criticism, especially @matthieu, @jborrow, and @rgb.Improved black hole modelYannick BahéYannick Bahéhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/issues/381Threadpool reduction2017-10-31T11:11:49ZMatthieu SchallerThreadpool reductionFrom my wish list:
There are now two places in the code where we perform a reduction of quantities using the threadpool. One of them is performance critical as it takes place at the end of the time-step to gather the minimum over all (l...From my wish list:
There are now two places in the code where we perform a reduction of quantities using the threadpool. One of them is performance critical as it takes place at the end of the time-step to gather the minimum over all (local) top-level cells.
At the moment this is done in a rather ad-hoc way (see `engine_collect_end_of_step_mapper()`) by locking something (here the `space` that has nothing to do with this bit of code!) whenever the thread is done iterating over its chunk of data.
That seems both inelegant and inefficient to me.
Ideally we could have a mechanism to perform a reduction which only locks once per thread in the pool (or even less) and not once per chunk. And ideally this would be generic and not have to be explicitly written for each reduction.Pedro GonnetPedro Gonnet