SWIFTsim merge requestshttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests2024-03-27T21:42:35Zhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1886Update to the spin/jet AGN model, March 20242024-03-27T21:42:35ZFilip HuskoUpdate to the spin/jet AGN model, March 2024@matthieu Here is the update I was mentioning. It is overall not as bad as I thought in terms of overall size. Here is a summary of the main changes.
1. Update to how the final BH-BH merger spin is calculated. The numerical relativity p...@matthieu Here is the update I was mentioning. It is overall not as bad as I thought in terms of overall size. Here is a summary of the main changes.
1. Update to how the final BH-BH merger spin is calculated. The numerical relativity papers with fitting functions typically give a formula for the final spin magnitude, which I was using. I was also assuming that the final spin direction corresponds to the direction of (in vector form) J_spin_1 + J_spin_2 + L_orb (the total angular momentum of the two spins + orbital angular momentum). This bit is very wrong, since the orbital angular momentum in the simulations dominates over everything else. In the new approach, I use a vector equation for the final spin (which is also given by the papers). This gives both its magnitude and direction correctly, matching the numerical relativity simulations.
2. I added wind efficiencies in the thick and slim disc (low and super-Eddington rates). These are implemented simply as thermal feedback, alongside the 'radiative'/quasar channel used in the thin disc, with everything being exactly the same, except the efficiencies. Technically, the thin disc feedback mode is also a wind, so there really isn't any distinction here, it's just a matter of what launches the wind (gas/MHD/radiation pressure), but they are all thermalized on subgrid scales. I have added additional tracer fields to keep track of how much energy is dumped in the wind mode, as well as in radiation, so things can be separated/analyzed in postprocessing.
3. The definition of the Eddington ratio (for this model) has been changed to the spin-dependent one, to be more consistent with recent papers.
4. The accretion/feedback modes are now switched immediately upon the calculation of the new Eddington ratio. This means that how much energy is being injected now matches which mode the BH is actually in, but the length of the current time-step may be too short or too long to represent one heating/kicking event. I am not sure what would be the best way to take care of that. But switching modes immediately is at least energetically more conservative.
5. Some updates to the accretion efficiency. I have added a slim disc (super-Eddington) efficiency, as well as an option of an Eddington-ratio dependent accretion efficiency in the thick disc (as motivated by simulations).
6. Added update to how jet efficiencies and spindown are calculated in the super-Eddington regime, as based on new GRMHD simulations. These same simulations predict non-negligible effects at sub-Eddington rates, so there is an option to apply this also in that regime.
7. Modified the scaling of the jet velocity with BH mass to reflect what is being done for COLIBRE.
8. Added two new tracer fields for jet kicking events. When a gas particle is kicked, it now knows which BH kicked it, and which accretion/feedback mode the BH was in when it did so. This will be extremely useful for analyses later on, especially for creation of radio images, which I hope to do.
Some things to do before merging:
- update parameter files
- update documentationFilip HuskoFilip Huskohttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1881Mhd_canvas_ABC_flow2024-03-25T16:39:07ZNikyta ShchutskyiMhd_canvas_ABC_flowMHD canvas with ABC flow forcing and exampleMHD canvas with ABC flow forcing and exampleMatthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1880Consider MHD attributes of particles that have no neighbours2024-03-26T22:38:13ZOrestis KarapiperisConsider MHD attributes of particles that have no neighboursTreating MHD attributes of particles with no neighbours ; solution to switch off artificial resistivity not optimal, should be improved upon.Treating MHD attributes of particles with no neighbours ; solution to switch off artificial resistivity not optimal, should be improved upon.Matthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1879Draft: Zoom Task analysis tools2024-03-20T18:33:29ZWilliam RoperDraft: Zoom Task analysis toolsA draft for now but open for comment. See below for examples.
This PR implements writing the cell type, subtype and depth into the `thread_info_*` files. The former of these is implemented in `plot_tasks.py` such that a user can flag `-...A draft for now but open for comment. See below for examples.
This PR implements writing the cell type, subtype and depth into the `thread_info_*` files. The former of these is implemented in `plot_tasks.py` such that a user can flag `--celltype` to split tasks based on cell type.
I've also implemented `plot_tasks_subset.py` which allows you to filter based on cell type, subtype, and minimum depth.
There's some fairly serious code duplication across all of this but not sure if it's worth making a module from which functions can be imported, would require a bit of a rewrite.
I've also implemented a random HSV colour rather than the colour cycle but I'm not sold on it. It does work a bit better for large task numbers but it's still not particularly easy to identify different tasks based on colour when you have a lot.
Note: I've been testing this on the `zoom_maketasks` branch.Peter W. DraperPeter W. Draperhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1877Zoom 5: Introduce Zoom cell tree and multipole construction adaptations2024-03-26T00:17:34ZWilliam RoperZoom 5: Introduce Zoom cell tree and multipole construction adaptationsThis introduces some modifications to `space_split` to enable zoom simulations. These are predominantly:
- Dividing `space_split_mapper` threadpools into different calls over each cell grid.
- Construction of a void cell tree down to th...This introduces some modifications to `space_split` to enable zoom simulations. These are predominantly:
- Dividing `space_split_mapper` threadpools into different calls over each cell grid.
- Construction of a void cell tree down to the zoom cell level (and population of the void cell multipoles).
- Linking of zoom cells into the leaves of the void cell tree.Matthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1875Draft: Zoom simulation documentation2024-03-17T15:30:15ZWilliam RoperDraft: Zoom simulation documentationImplements the Zoom RTD documentation. This is a draft which will be added to as the Zoom merge progresses. I envision readying this and merging it before merging the `zoom_merge` branch into master when we hit a sufficient checkpoint.Implements the Zoom RTD documentation. This is a draft which will be added to as the Zoom merge progresses. I envision readying this and merging it before merging the `zoom_merge` branch into master when we hit a sufficient checkpoint.Matthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1865Fix rounding error bug in calculation of number of top-level cells2024-03-05T13:06:00ZKyle OmanFix rounding error bug in calculation of number of top-level cellsBecause a value of `0.99` was used to force rounding down (see changelog), trying to use a top-level cells count of ~100 could lead to rounding to +/-1 from expected value. This more adaptive approach should avoid this issue for arbitrar...Because a value of `0.99` was used to force rounding down (see changelog), trying to use a top-level cells count of ~100 could lead to rounding to +/-1 from expected value. This more adaptive approach should avoid this issue for arbitrarily large numbers of top-level cells (at least until Ncells^2 approaches the floating point limit).
There is a potential issue if trying to use one top-level cell. I think it might make sense to enforce a minimum value for the `tol` that I've introduced (could be the previously-used `0.99`, for instance, just needs to be less than 1). Thought I'd ask here before adding more logic, though.https://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1864Updates for testing Circularly Polarised Alfven Wave Test2024-03-06T17:07:53ZOrestis KarapiperisUpdates for testing Circularly Polarised Alfven Wave TestMatthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1848Draft: Added loop advection test, should run out of the box2024-03-27T21:48:02ZOrestis KarapiperisDraft: Added loop advection test, should run out of the boxMatthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1838Draft: Velocity dependent term in Dedner evolution2024-02-26T12:42:54ZOrestis KarapiperisDraft: Velocity dependent term in Dedner evolutionThis is still preliminary but perhaps interestingThis is still preliminary but perhaps interestingMatthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1826Draft: Magnetosonic speed in timescale used in updating the viscosity alphas2024-02-23T21:01:39ZOrestis KarapiperisDraft: Magnetosonic speed in timescale used in updating the viscosity alphasAlso changed hydro parameters to what works best for me in the shock tubes, and is consistent with the value suggested for hydro. Let me know what you think.Also changed hydro parameters to what works best for me in the shock tubes, and is consistent with the value suggested for hydro. Let me know what you think.Matthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1824Sink : bugs fixing2024-03-25T17:14:24ZDarwinSink : bugs fixingAll known bugs that lead to crashes are solved.All known bugs that lead to crashes are solved.https://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1777Draft: Streamlined MAGMA2 implementation2024-02-13T12:31:21ZMatthieu SchallerDraft: Streamlined MAGMA2 implementationThis is a reimplementation of the MAGMA2 branch with:
- Simpler symmetric matrix handling
- Only 1st order reconstruction of `v` and `u`.
- Only basic viscosity and diffusion terms as in the Rosswog 2020 implementation
Questions:
...This is a reimplementation of the MAGMA2 branch with:
- Simpler symmetric matrix handling
- Only 1st order reconstruction of `v` and `u`.
- Only basic viscosity and diffusion terms as in the Rosswog 2020 implementation
Questions:
- Do we need more advanced terms? SPHENIX-like? Or the Rosswog entropy-based viscosity switch?
- What implementation do we want? Traditional or Gasoline?
Todo:
- [ ] Document the matrix stuff
- [ ] 1D and 2D cases
- [ ] Scale-factors
- [ ] Strange corner cases (e.g. no ngb)
- [ ] Tests in galaxy settings
- [ ] Properly implement the symmetric functions.
- [ ] Fix documentation. It still mentions `Minimal` and `SPHENIX` in places.https://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1749Draft: Merge the moving mesh hydro scheme in master2024-03-27T14:16:36ZYolan UyttenhoveDraft: Merge the moving mesh hydro scheme in master### Overview
This adds a moving-mesh hydro scheme to SWIFT. To use it, SWIFT must be compiled with the option `--with-hydro=shadowswift` and run with `--hydro`. Most hydro-test work out of the box, but might need tweaking for optimal pe...### Overview
This adds a moving-mesh hydro scheme to SWIFT. To use it, SWIFT must be compiled with the option `--with-hydro=shadowswift` and run with `--hydro`. Most hydro-test work out of the box, but might need tweaking for optimal performance (see caveats).
The scheme uses a `drift` -> `geometry` -> `flux` -> `kick2` -> `gradient` -> `timestep` -> `kick1` structure for it's time-step:
- **Grid construction** is carried out as a `self` task with implicit `grid-sync` `pair` tasks ensuring the necessary synchronization with the neighboring cells whose particles are _read_ during construction.
- **Flux exchange** is always carried out by ordinary `self` and `pair` tasks since it must be done on the construction level in the tree (we need the Voronoi faces). Flux exchange is also manifestly conservative/symmetric and therefore `pair` tasks must be handled on both MPI nodes if necessary.
- For the **gradient interaction**, I have switched to a meshless WLS gradient scheme, meaning that the gradient calculation is a "normal" hydro loop which can be carried out by `self`, `pair`, `subself` and `subpair` interactions on any desired level, independent of the grid construction. Alternatively, the older WLS gradient computation which uses the Voronoi cells is also available and that method uses the same task structure of the flux interaction loop.
- The **limiter interaction** follows the same structure as the gradient loop. Meaning that it will be an ordinary hydro loop when the meshless gradients are used and a neighbour loop explicitely looping over the Voronoi faces else.
### Caveats
- Running a hydro-test as-is, will introduce scatter in the densities, because each particle has it's own volume determined by it's Voronoi cell. Exact ICs can be computed by using the volumes saved in the first snapshot and the desired densities to compute the appropriate masses for the particles.
- A vacuum must be explicitly represented by particles of 0 (or extremely low) density (e.g. in the Evrard collapse or Vacuum hydro tests). Exactly zero mass particles cause problems with gravity at the moment (e.g. cannot compute center of mass of cells containing only zero mass particles), but work fine in hydro only simulations.
- The timestep limiter works in most cases, but might use a wrong timestep for the flux calculation in some rare cases (see also Issue #832).
### Current status
This is an overview of what is/isn't working (yet):
- [x] Hydrodynamics (1D, 2D and 3D)
- [x] Gravity
- [x] Cosmological time-integration
- [ ] Zel'dovich pancake
- [x] Chemistry (See also issue #834 for detailed status)
- [x] cooling
- [ ] Star formation/stellar feedback
- [x] Non periodic boundary conditions: reflective, vacuum, open and inflow boundary conditions
- [x] (Fixed) boundary particles
- [x] MPI support
- [ ] Particle splitting/merging
### To-do
- [ ] Some functions are missing docstrings
- [x] Documentation on websiteYolan UyttenhoveYolan Uyttenhovehttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1735Draft: Add host weights to METIS partitioning2023-09-11T10:47:06ZPeter W. DraperDraft: Add host weights to METIS partitioningSee #856 first implementation of the basics. Largely untested!See #856 first implementation of the basics. Largely untested!Peter W. DraperPeter W. Draperhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1667Draft: Added Smoothed RT option to SPHM1RT2023-06-12T17:25:00ZTsang Keung ChanDraft: Added Smoothed RT option to SPHM1RTAdded a new injection scheme, namely smoothedRT:
> If ``smoothedRT`` is set to 1, we use the ``Smoothed RT`` scheme (see Section 4.4
> in doi:10.1093/mnras/stt1722). Instead of adding radiation energy and then decaying
> through therm...Added a new injection scheme, namely smoothedRT:
> If ``smoothedRT`` is set to 1, we use the ``Smoothed RT`` scheme (see Section 4.4
> in doi:10.1093/mnras/stt1722). Instead of adding radiation energy and then decaying
> through thermo-chemistry, we add source terms for radiation in the thermo-chemistry
> equations. ``Smoothed RT`` should be faster because radiation is not artifically
> boosted and decayed within the time-step. ``Smoothed RT`` should also be more accurate
> since it models the physical process more accurately -- radiation is continously
> injected and interacts with matter.
Test with Stromgren_3D singlebin and the result is satisfactory:
![output_singlebin_0050](/uploads/0e5ff686e1be7e53d7282d695d4af970/output_singlebin_0050.png)
![output_singlebin_0050-Stromgren3Dsinglebin](/uploads/69b438e78c319b2aeb7d7e47f8214a86/output_singlebin_0050-Stromgren3Dsinglebin.png)Matthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1665Draft: Add threadpool mappers for scheduler_reweight2023-09-11T10:48:37ZPeter W. DraperDraft: Add threadpool mappers for scheduler_reweightIs better than serial code (100ms compared to 500ms in my test), but has one issue, which is some weights will be less optimal than before as the accumulation of weights will not happen between threads. This is why we use a uniform chunk...Is better than serial code (100ms compared to 500ms in my test), but has one issue, which is some weights will be less optimal than before as the accumulation of weights will not happen between threads. This is why we use a uniform chunk to keep the work load from too many splits.
For any reasonable set of tasks this is a small issue, there is no effect measureable that I can see.Matthieu SchallerMatthieu Schallerhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1661Draft: Implement a memcpy() clone that uses the threadpool2023-09-11T10:45:49ZPeter W. DraperDraft: Implement a memcpy() clone that uses the threadpoolUsing multiple threads to copy data can make better use of the available memory bandwidth when multiple NUMA
regions are being used. This is an attempt to exploit that.
Experiments show that using it should be done only for quite large ...Using multiple threads to copy data can make better use of the available memory bandwidth when multiple NUMA
regions are being used. This is an attempt to exploit that.
Experiments show that using it should be done only for quite large data, so a heuristic is in place
to support that, which requires a number of 4k pages, at least 1 seems to be required, which makes sense,
but more are required in reality (COSMA8 AMD 128 core, 8 NUMA regions). Surprisingly even then using all the
threads is still not a guarantor of the best speed, so we also apply a limit to the number of threads
that will be used to 25% of the total, using more than 1 gives a good improvement which tails off
rapidly.
The actual gains here are modest, the largest come from particle splitting, but only when the buffers
need reallocating and replication, which is only used for testing.
So the question is, is this worth it...Peter W. DraperPeter W. Draperhttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1658Draft: Lanson&Vila (2008) timestep criterion for Gizmo2023-01-21T20:59:01ZBert VandenbrouckeDraft: Lanson&Vila (2008) timestep criterion for GizmoThis contains the original attempt to implement a better timestep criterion for Gizmo, based on Lanson&Vila (2008).
Needs further testing and refinement.This contains the original attempt to implement a better timestep criterion for Gizmo, based on Lanson&Vila (2008).
Needs further testing and refinement.Bert VandenbrouckeBert Vandenbrouckehttps://gitlab.cosma.dur.ac.uk/swift/swiftsim/-/merge_requests/1654Draft: NUMA aware pinning of queues and runners.2023-09-11T10:41:25ZPeter W. DraperDraft: NUMA aware pinning of queues and runners.Adds the capability to pin queues and runners within NUMA regions.
Adds queue selection by tasks on the basis of the NUMA region that holds the start of it's main data area.
Adds the spreading of swift allocated memory using larger in...Adds the capability to pin queues and runners within NUMA regions.
Adds queue selection by tasks on the basis of the NUMA region that holds the start of it's main data area.
Adds the spreading of swift allocated memory using larger interleave chunks (default for those is 4k).
On COSMA8 this shows speed improvements over the existing master, even with pinning and interleave.
(Based on the !1649 so we also have those improvements, now merged.)
Not sure how serious these changes are yet, as we need to add an additional argument to all the swift_free() calls so that the
memory spread can be undone, also the memory alignment is done using page boundaries (4k). Also requires that there is
a one to one correspondence between queues and runners as these are pinned to NUMA regions in pairs.Peter W. DraperPeter W. Draper