|
|
|
# Modules
|
|
|
|
|
|
|
|
### Cosma 8
|
|
|
|
|
|
|
|
```
|
|
|
|
module load intel_comp/2021.1.0 compiler
|
|
|
|
module load intel_mpi/2018
|
|
|
|
module load ucx/1.10.1
|
|
|
|
module load fftw/3.3.9epyc parallel_hdf5/1.10.6 parmetis/4.0.3-64bit gsl/2.5
|
|
|
|
```
|
|
|
|
|
|
|
|
(Alternatively, `intel_mpi/2020-update2` can be tried instead of intel 2018 but seems at least partially broken)
|
|
|
|
|
|
|
|
For users preferring, Open-MPI, we recommend
|
|
|
|
|
|
|
|
```
|
|
|
|
module load intel_comp/2021.1.0 compiler
|
|
|
|
module load openmpi/4.0.5
|
|
|
|
module load ucx/1.10.3
|
|
|
|
module load fftw/3.3.9epyc parallel_hdf5/1.10.6 parmetis/4.0.3-64bit gsl/2.5
|
|
|
|
```
|
|
|
|
|
|
|
|
Note: when running on a single node of cosma-8 (i.e. the non-MPI version of the code), add `--pool-threads=32` to the command line for a small extra oomph. Everything is similar to the rest of the cosma's.
|
|
|
|
|
|
|
|
### Cosma 5,6 & 7
|
|
|
|
|
|
|
|
Since October 2021, we recommend:
|
|
|
|
|
|
|
|
```
|
|
|
|
module load intel_comp/2021.1.0 compiler
|
|
|
|
module load intel_mpi/2018
|
|
|
|
module load ucx/1.10.3
|
|
|
|
module load fftw/3.3.9cosma7 # or fftw/3.3.9 on cosma 5 & 6
|
|
|
|
module load parallel_hdf5/1.10.6 parmetis/4.0.3-64bit gsl/2.5
|
|
|
|
```
|
|
|
|
|
|
|
|
Since October 2020, we recommend:
|
|
|
|
|
|
|
|
```
|
|
|
|
module load intel_comp/2020-update2
|
|
|
|
module load intel_mpi/2020-update2
|
|
|
|
module load ucx/1.8.1
|
|
|
|
module load parmetis/4.0.3-64bit
|
|
|
|
module load parallel_hdf5/1.10.6
|
|
|
|
module load fftw/3.3.8cosma7 # On cosma 5 or 6 use fftw/3.3.8 On cosma 8, use fftw/3.3.8epyc
|
|
|
|
module load gsl/2.5
|
|
|
|
module load llvm/10.0.1 # Only necessary if wanting to use the code formatting tool
|
|
|
|
```
|
|
|
|
|
|
|
|
(Note the 2019 version of Intel-MPI is broken and the 2020u2 might still be shaky)
|
|
|
|
|
|
|
|
|
|
|
|
We previously used (for records only):
|
|
|
|
|
|
|
|
```
|
|
|
|
intel_comp/2018 intel_mpi/2018 parallel_hdf5/1.10.3 fftw/3.3.7 gsl/2.4 parmetis/4.0.3
|
|
|
|
```
|
|
|
|
|
|
|
|
For users who prefer using OpenMPI on cosma7, we recommend:
|
|
|
|
```
|
|
|
|
module load intel_comp/2020-update2
|
|
|
|
module load openmpi/4.0.5
|
|
|
|
module load ucx/1.8.1
|
|
|
|
module load parmetis/4.0.3-64bit
|
|
|
|
module load parallel_hdf5/1.10.6
|
|
|
|
module load fftw/3.3.8cosma7
|
|
|
|
module load gsl/2.5
|
|
|
|
```
|
|
|
|
|
|
|
|
# Configuration
|
|
|
|
|
|
|
|
The following configuration options are the most common:
|
|
|
|
|
|
|
|
- Default (aka gravity + hydro):
|
|
|
|
`./configure --enable-ipo --with-tbbmalloc --with-parmetis`
|
|
|
|
|
|
|
|
- Full EAGLE:
|
|
|
|
`./configure --with-subgrid=EAGLE --with-hydro=sphenix --with-kernel=wendland-C2 --enable-ipo --with-tbbmalloc --with-parmetis`
|
|
|
|
|
|
|
|
|
|
|
|
- Full EAGLE-XL:
|
|
|
|
`./configure --with-subgrid=EAGLE-XL --with-hydro=sphenix --with-kernel=wendland-C2 --enable-ipo --with-tbbmalloc --with-parmetis`
|
|
|
|
|
|
|
|
- Debugging: Add `--enable-debug --enable-debugging-checks --disable-optimization`
|
|
|
|
|
|
|
|
|
|
|
|
# SLURM scripts
|
|
|
|
|
|
|
|
The parameters can be take from the corresponding example. A submission script for a single node job would look like:
|
|
|
|
|
|
|
|
```
|
|
|
|
#!/bin/bash -l
|
|
|
|
|
|
|
|
#SBATCH -J SWIFT_EAGLE_12
|
|
|
|
#SBATCH -N 1
|
|
|
|
#SBATCH -o job.%J.dump
|
|
|
|
#SBATCH -e job.%J.err
|
|
|
|
#SBATCH -p cosma7
|
|
|
|
#SBATCH -A dp004
|
|
|
|
#SBATCH --exclusive
|
|
|
|
#SBATCH --mail-type=ALL
|
|
|
|
#SBATCH --mail-user=your.email@your.job
|
|
|
|
#SBATCH -t 72:00:00
|
|
|
|
|
|
|
|
module load intel_comp/2020-update2 intel_mpi/2020-update2 ucx/1.8.1 parmetis/4.0.3-64bit parallel_hdf5/1.10.6 gsl/2.5 fftw/3.3.8cosma7
|
|
|
|
|
|
|
|
# Gravity only:
|
|
|
|
../../swift --pin --cosmology --self-gravity -v 1 --threads=28 eagle_12.yml
|
|
|
|
|
|
|
|
# Gravity + hydro:
|
|
|
|
../../swift --pin --cosmology --hydro --self-gravity -v 1 --threads=28 eagle_12.yml
|
|
|
|
|
|
|
|
# Full physics:
|
|
|
|
../../swift --pin --cosmology --eagle -v 1 --threads=28 eagle_12.yml
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
whilst for a multi-node job using Intel-MPI 2020, one would use:
|
|
|
|
|
|
|
|
```
|
|
|
|
#!/bin/bash -l
|
|
|
|
|
|
|
|
#SBATCH -J SWIFT_EAGLE_12
|
|
|
|
#SBATCH -N 8
|
|
|
|
#SBATCH --tasks-per-node=2
|
|
|
|
#SBATCH -o job.%J.dump
|
|
|
|
#SBATCH -e job.%J.err
|
|
|
|
#SBATCH -p cosma7
|
|
|
|
#SBATCH -A dp004
|
|
|
|
#SBATCH --exclusive
|
|
|
|
#SBATCH --mail-type=ALL
|
|
|
|
#SBATCH --mail-user=your.email@your.job
|
|
|
|
#SBATCH -t 72:00:00
|
|
|
|
|
|
|
|
module load intel_comp/2020-update2 intel_mpi/2020-update2 ucx/1.10.1 parmetis/4.0.3-64bit parallel_hdf5/1.10.6 gsl/2.5 fftw/3.3.8cosma7
|
|
|
|
|
|
|
|
# Gravity only:
|
|
|
|
mpirun -np 16 ../../swift --pin --cosmology --self-gravity -v 1 --threads=14 eagle_12.yml
|
|
|
|
|
|
|
|
# Gravity + hydro:
|
|
|
|
mpirun -np 16 ../../swift --pin --cosmology --hydro --self-gravity -v 1 --threads=14 eagle_12.yml
|
|
|
|
|
|
|
|
# Full physics:
|
|
|
|
mpirun -np 16 ../../swift --pin --cosmology --eagle -v 1 --threads=14 eagle_12.yml
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
### DINE
|
|
|
|
|
|
|
|
This is a number of AMD epyc nodes. Here is a SLURM submission script for a recent
|
|
|
|
Intel toolchain, the build details are the same as COSMA8.
|
|
|
|
```
|
|
|
|
#!/bin/bash -l
|
|
|
|
|
|
|
|
#SBATCH -J SWIFT-EAGLE-25
|
|
|
|
#SBATCH --ntasks=8
|
|
|
|
#SBATCH --cpus-per-task=32
|
|
|
|
#SBATCH -o job.%J.dump
|
|
|
|
#SBATCH -e job.%J.err
|
|
|
|
#SBATCH -p bluefield1
|
|
|
|
#SBATCH -A durham
|
|
|
|
#SBATCH --exclusive
|
|
|
|
#SBATCH -t 4:00:00
|
|
|
|
|
|
|
|
module purge
|
|
|
|
module load intel_comp/2022.1.2 compiler mpi
|
|
|
|
module load gsl/2.4 fftw/3.3.9epyc parmetis/4.0.3-64bit parallel_hdf5/1.12.0 ucx/1.13.0rc2
|
|
|
|
|
|
|
|
export I_MPI_PIN_DOMAIN=auto
|
|
|
|
|
|
|
|
mpirun -np $SLURM_NTASKS ../../../swift_mpi --hydro --self-gravity --stars -v 1 --threads=16 \
|
|
|
|
eagle_25.yml
|
|
|
|
|
|
|
|
exit
|
|
|
|
```
|
|
|
|
# Modules
|
|
|
|
|
|
|
|
### Cosma 8
|
|
|
|
|
|
|
|
```
|
|
|
|
module load intel_comp/2024.2.0 compiler-rt tbb compiler mpi
|
|
|
|
module load ucx/1.13.0rc2
|
|
|
|
module load parallel_hdf5/1.14.4
|
|
|
|
module load fftw/3.3.10
|
|
|
|
module load parmetis
|
|
|
|
module load gsl/2.5
|
|
|
|
module load sundials/5.1.0_c8_single
|
|
|
|
```
|
|
|
|
|
|
|
|
----------------------
|
|
|
|
|
|
|
|
Prior to July 2024 (for legacy):
|
|
|
|
|
|
|
|
### Cosma 8
|
|
|
|
|
|
|
|
```
|
|
|
|
module load intel_comp/2021.1.0 compiler
|
|
|
|
module load intel_mpi/2018
|
|
|
|
module load ucx/1.10.1
|
|
|
|
module load fftw/3.3.9epyc parallel_hdf5/1.10.6 parmetis/4.0.3-64bit gsl/2.5
|
|
|
|
```
|
|
|
|
|
|
|
|
(Alternatively, `intel_mpi/2020-update2` can be tried instead of intel 2018 but seems at least partially broken)
|
|
|
|
|
|
|
|
For users preferring, Open-MPI, we recommend
|
|
|
|
|
|
|
|
```
|
|
|
|
module load intel_comp/2021.1.0 compiler
|
|
|
|
module load openmpi/4.0.5
|
|
|
|
module load ucx/1.10.3
|
|
|
|
module load fftw/3.3.9epyc parallel_hdf5/1.10.6 parmetis/4.0.3-64bit gsl/2.5
|
|
|
|
```
|
|
|
|
|
|
|
|
Note: when running on a single node of cosma-8 (i.e. the non-MPI version of the code), add `--pool-threads=32` to the command line for a small extra oomph. Everything is similar to the rest of the cosma's.
|
|
|
|
|
|
|
|
### Cosma 5,6 & 7
|
|
|
|
|
|
|
|
Since October 2021, we recommend:
|
|
|
|
|
|
|
|
```
|
|
|
|
module load intel_comp/2021.1.0 compiler
|
|
|
|
module load intel_mpi/2018
|
|
|
|
module load ucx/1.10.3
|
|
|
|
module load fftw/3.3.9cosma7 # or fftw/3.3.9 on cosma 5 & 6
|
|
|
|
module load parallel_hdf5/1.10.6 parmetis/4.0.3-64bit gsl/2.5
|
|
|
|
```
|
|
|
|
|
|
|
|
Since October 2020, we recommend:
|
|
|
|
|
|
|
|
```
|
|
|
|
module load intel_comp/2020-update2
|
|
|
|
module load intel_mpi/2020-update2
|
|
|
|
module load ucx/1.8.1
|
|
|
|
module load parmetis/4.0.3-64bit
|
|
|
|
module load parallel_hdf5/1.10.6
|
|
|
|
module load fftw/3.3.8cosma7 # On cosma 5 or 6 use fftw/3.3.8 On cosma 8, use fftw/3.3.8epyc
|
|
|
|
module load gsl/2.5
|
|
|
|
module load llvm/10.0.1 # Only necessary if wanting to use the code formatting tool
|
|
|
|
```
|
|
|
|
|
|
|
|
(Note the 2019 version of Intel-MPI is broken and the 2020u2 might still be shaky)
|
|
|
|
|
|
|
|
|
|
|
|
We previously used (for records only):
|
|
|
|
|
|
|
|
```
|
|
|
|
intel_comp/2018 intel_mpi/2018 parallel_hdf5/1.10.3 fftw/3.3.7 gsl/2.4 parmetis/4.0.3
|
|
|
|
```
|
|
|
|
|
|
|
|
For users who prefer using OpenMPI on cosma7, we recommend:
|
|
|
|
```
|
|
|
|
module load intel_comp/2020-update2
|
|
|
|
module load openmpi/4.0.5
|
|
|
|
module load ucx/1.8.1
|
|
|
|
module load parmetis/4.0.3-64bit
|
|
|
|
module load parallel_hdf5/1.10.6
|
|
|
|
module load fftw/3.3.8cosma7
|
|
|
|
module load gsl/2.5
|
|
|
|
```
|
|
|
|
|
|
|
|
# Configuration
|
|
|
|
|
|
|
|
The following configuration options are the most common:
|
|
|
|
|
|
|
|
- Default (aka gravity + hydro):
|
|
|
|
`./configure --enable-ipo --with-tbbmalloc --with-parmetis`
|
|
|
|
|
|
|
|
- Full EAGLE:
|
|
|
|
`./configure --with-subgrid=EAGLE --with-hydro=sphenix --with-kernel=wendland-C2 --enable-ipo --with-tbbmalloc --with-parmetis`
|
|
|
|
|
|
|
|
|
|
|
|
- Full EAGLE-XL:
|
|
|
|
`./configure --with-subgrid=EAGLE-XL --with-hydro=sphenix --with-kernel=wendland-C2 --enable-ipo --with-tbbmalloc --with-parmetis`
|
|
|
|
|
|
|
|
- Debugging: Add `--enable-debug --enable-debugging-checks --disable-optimization`
|
|
|
|
|
|
|
|
|
|
|
|
# SLURM scripts
|
|
|
|
|
|
|
|
The parameters can be take from the corresponding example. A submission script for a single node job would look like:
|
|
|
|
|
|
|
|
```
|
|
|
|
#!/bin/bash -l
|
|
|
|
|
|
|
|
#SBATCH -J SWIFT_EAGLE_12
|
|
|
|
#SBATCH -N 1
|
|
|
|
#SBATCH -o job.%J.dump
|
|
|
|
#SBATCH -e job.%J.err
|
|
|
|
#SBATCH -p cosma7
|
|
|
|
#SBATCH -A dp004
|
|
|
|
#SBATCH --exclusive
|
|
|
|
#SBATCH --mail-type=ALL
|
|
|
|
#SBATCH --mail-user=your.email@your.job
|
|
|
|
#SBATCH -t 72:00:00
|
|
|
|
|
|
|
|
module load intel_comp/2020-update2 intel_mpi/2020-update2 ucx/1.8.1 parmetis/4.0.3-64bit parallel_hdf5/1.10.6 gsl/2.5 fftw/3.3.8cosma7
|
|
|
|
|
|
|
|
# Gravity only:
|
|
|
|
../../swift --pin --cosmology --self-gravity -v 1 --threads=28 eagle_12.yml
|
|
|
|
|
|
|
|
# Gravity + hydro:
|
|
|
|
../../swift --pin --cosmology --hydro --self-gravity -v 1 --threads=28 eagle_12.yml
|
|
|
|
|
|
|
|
# Full physics:
|
|
|
|
../../swift --pin --cosmology --eagle -v 1 --threads=28 eagle_12.yml
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
whilst for a multi-node job using Intel-MPI 2020, one would use:
|
|
|
|
|
|
|
|
```
|
|
|
|
#!/bin/bash -l
|
|
|
|
|
|
|
|
#SBATCH -J SWIFT_EAGLE_12
|
|
|
|
#SBATCH -N 8
|
|
|
|
#SBATCH --tasks-per-node=2
|
|
|
|
#SBATCH -o job.%J.dump
|
|
|
|
#SBATCH -e job.%J.err
|
|
|
|
#SBATCH -p cosma7
|
|
|
|
#SBATCH -A dp004
|
|
|
|
#SBATCH --exclusive
|
|
|
|
#SBATCH --mail-type=ALL
|
|
|
|
#SBATCH --mail-user=your.email@your.job
|
|
|
|
#SBATCH -t 72:00:00
|
|
|
|
|
|
|
|
module load intel_comp/2020-update2 intel_mpi/2020-update2 ucx/1.10.1 parmetis/4.0.3-64bit parallel_hdf5/1.10.6 gsl/2.5 fftw/3.3.8cosma7
|
|
|
|
|
|
|
|
# Gravity only:
|
|
|
|
mpirun -np 16 ../../swift --pin --cosmology --self-gravity -v 1 --threads=14 eagle_12.yml
|
|
|
|
|
|
|
|
# Gravity + hydro:
|
|
|
|
mpirun -np 16 ../../swift --pin --cosmology --hydro --self-gravity -v 1 --threads=14 eagle_12.yml
|
|
|
|
|
|
|
|
# Full physics:
|
|
|
|
mpirun -np 16 ../../swift --pin --cosmology --eagle -v 1 --threads=14 eagle_12.yml
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
### DINE
|
|
|
|
|
|
|
|
This is a number of AMD epyc nodes. Here is a SLURM submission script for a recent
|
|
|
|
Intel toolchain, the build details are the same as COSMA8.
|
|
|
|
```
|
|
|
|
#!/bin/bash -l
|
|
|
|
|
|
|
|
#SBATCH -J SWIFT-EAGLE-25
|
|
|
|
#SBATCH --ntasks=8
|
|
|
|
#SBATCH --cpus-per-task=32
|
|
|
|
#SBATCH -o job.%J.dump
|
|
|
|
#SBATCH -e job.%J.err
|
|
|
|
#SBATCH -p bluefield1
|
|
|
|
#SBATCH -A durham
|
|
|
|
#SBATCH --exclusive
|
|
|
|
#SBATCH -t 4:00:00
|
|
|
|
|
|
|
|
module purge
|
|
|
|
module load intel_comp/2022.1.2 compiler mpi
|
|
|
|
module load gsl/2.4 fftw/3.3.9epyc parmetis/4.0.3-64bit parallel_hdf5/1.12.0 ucx/1.13.0rc2
|
|
|
|
|
|
|
|
export I_MPI_PIN_DOMAIN=auto
|
|
|
|
|
|
|
|
mpirun -np $SLURM_NTASKS ../../../swift_mpi --hydro --self-gravity --stars -v 1 --threads=16 \
|
|
|
|
eagle_25.yml
|
|
|
|
|
|
|
|
exit
|
|
|
|
```
|
|
|
|
Note that `I_MPI_PIN_DOMAIN` is needed for pinning. |
|
|
\ No newline at end of file |
