Modules
Cosma 8
module load intel_comp/2024.2.0 compiler-rt tbb compiler mpi
module load ucx/1.17.0
module load parallel_hdf5/1.14.4
module load fftw/3.3.10
module load parmetis/4.0.3-64bit
module load gsl/2.5
module load sundials/5.8.0_c8_singleThe thread pinning for MPI is achieved by setting:
export I_MPI_PIN_DOMAIN=autoin the submission script.
Alternatively, using Open-MPI:
module load intel_comp/2024.2.0 compiler-rt tbb compiler
module load openmpi/5.0.3
module load fftw/3.3.10
module load gsl
module load parmetis
module load parallel_hdf5/1.14.4
module load sundials/5.8.0_c8_singleThe thread pinning for MPI is achieved by setting:
mpirun --map-by socket:PE=32 -np MY_NUMBER_OF_MPI_RANKS swift_mpi --my-fancy-swift-options(for 4 ranks per node with 32 threads each)
Installing Spack for SWIFT on Cosma's Grace-Hopper Nodes
- Install spack to cosma apps directory
- Follow the Spack tutorial
- You should end up with a spack install in e.g.
/cosma/apps/<project>/<username>/spack - You will run out of disk space if you install Spack in your home directory
- Running Spack's
setup-env.shwill fail due to Cosma providing an incompatible Python module by default. So unload Python first:
$ cat ~/load_spack_aarch64.sh
module unload python
source ~/spack/share/spack/setup-env.sh- After sourcing the Spack setup file you should have access to the
spackcommand. Create a Spack environment just for SWIFT. You can call your environment whatever you like. Here I call itswift.spack env create swift spack env activate swift - Install required dependencies:
spack install --add hdf5 - You should now be able to compile and run swift as normal.
Misc tips for SWIFT on Cosma Grace-Hopper
Sometimes Spack struggles to load the right library environment variables. You may need to compile with:
LIBRARY_PATH=$LD_LIBRARY_PATH ./configure ...Cosma 7
Prior to July 2024 (for legacy):
Cosma 8
module load intel_comp/2021.1.0 compiler
module load intel_mpi/2018
module load ucx/1.10.1
module load fftw/3.3.9epyc parallel_hdf5/1.10.6 parmetis/4.0.3-64bit gsl/2.5(Alternatively, intel_mpi/2020-update2 can be tried instead of intel 2018 but seems at least partially broken)
For users preferring, Open-MPI, we recommend
module load intel_comp/2021.1.0 compiler
module load openmpi/4.0.5
module load ucx/1.10.3
module load fftw/3.3.9epyc parallel_hdf5/1.10.6 parmetis/4.0.3-64bit gsl/2.5 Note: when running on a single node of cosma-8 (i.e. the non-MPI version of the code), add --pool-threads=32 to the command line for a small extra oomph. Everything is similar to the rest of the cosma's.
Cosma 5,6 & 7
Since October 2021, we recommend:
module load intel_comp/2021.1.0 compiler
module load intel_mpi/2018
module load ucx/1.10.3
module load fftw/3.3.9cosma7 # or fftw/3.3.9 on cosma 5 & 6
module load parallel_hdf5/1.10.6 parmetis/4.0.3-64bit gsl/2.5Since October 2020, we recommend:
module load intel_comp/2020-update2
module load intel_mpi/2020-update2
module load ucx/1.8.1
module load parmetis/4.0.3-64bit
module load parallel_hdf5/1.10.6
module load fftw/3.3.8cosma7 # On cosma 5 or 6 use fftw/3.3.8 On cosma 8, use fftw/3.3.8epyc
module load gsl/2.5
module load llvm/10.0.1 # Only necessary if wanting to use the code formatting tool(Note the 2019 version of Intel-MPI is broken and the 2020u2 might still be shaky)
We previously used (for records only):
intel_comp/2018 intel_mpi/2018 parallel_hdf5/1.10.3 fftw/3.3.7 gsl/2.4 parmetis/4.0.3For users who prefer using OpenMPI on cosma7, we recommend:
module load intel_comp/2020-update2
module load openmpi/4.0.5
module load ucx/1.8.1
module load parmetis/4.0.3-64bit
module load parallel_hdf5/1.10.6
module load fftw/3.3.8cosma7
module load gsl/2.5 Configuration
The following configuration options are the most common:
-
Default (aka gravity + hydro):
./configure --enable-ipo --with-tbbmalloc --with-parmetis -
Full EAGLE:
./configure --with-subgrid=EAGLE --with-hydro=sphenix --with-kernel=wendland-C2 --enable-ipo --with-tbbmalloc --with-parmetis -
Full EAGLE-XL:
./configure --with-subgrid=EAGLE-XL --with-hydro=sphenix --with-kernel=wendland-C2 --enable-ipo --with-tbbmalloc --with-parmetis -
Debugging: Add
--enable-debug --enable-debugging-checks --disable-optimization
SLURM scripts
The parameters can be take from the corresponding example. A submission script for a single node job would look like:
#!/bin/bash -l
#SBATCH -J SWIFT_EAGLE_12
#SBATCH -N 1
#SBATCH -o job.%J.dump
#SBATCH -e job.%J.err
#SBATCH -p cosma7
#SBATCH -A dp004
#SBATCH --exclusive
#SBATCH --mail-type=ALL
#SBATCH --mail-user=your.email@your.job
#SBATCH -t 72:00:00
module load intel_comp/2020-update2 intel_mpi/2020-update2 ucx/1.8.1 parmetis/4.0.3-64bit parallel_hdf5/1.10.6 gsl/2.5 fftw/3.3.8cosma7
# Gravity only:
../../swift --pin --cosmology --self-gravity -v 1 --threads=28 eagle_12.yml
# Gravity + hydro:
../../swift --pin --cosmology --hydro --self-gravity -v 1 --threads=28 eagle_12.yml
# Full physics:
../../swift --pin --cosmology --eagle -v 1 --threads=28 eagle_12.yml
whilst for a multi-node job using Intel-MPI 2020, one would use:
#!/bin/bash -l
#SBATCH -J SWIFT_EAGLE_12
#SBATCH -N 8
#SBATCH --tasks-per-node=2
#SBATCH -o job.%J.dump
#SBATCH -e job.%J.err
#SBATCH -p cosma7
#SBATCH -A dp004
#SBATCH --exclusive
#SBATCH --mail-type=ALL
#SBATCH --mail-user=your.email@your.job
#SBATCH -t 72:00:00
module load intel_comp/2020-update2 intel_mpi/2020-update2 ucx/1.10.1 parmetis/4.0.3-64bit parallel_hdf5/1.10.6 gsl/2.5 fftw/3.3.8cosma7
# Gravity only:
mpirun -np 16 ../../swift --pin --cosmology --self-gravity -v 1 --threads=14 eagle_12.yml
# Gravity + hydro:
mpirun -np 16 ../../swift --pin --cosmology --hydro --self-gravity -v 1 --threads=14 eagle_12.yml
# Full physics:
mpirun -np 16 ../../swift --pin --cosmology --eagle -v 1 --threads=14 eagle_12.yml
DINE
This is a number of AMD epyc nodes. Here is a SLURM submission script for a recent Intel toolchain, the build details are the same as COSMA8.
#!/bin/bash -l
#SBATCH -J SWIFT-EAGLE-25
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=32
#SBATCH -o job.%J.dump
#SBATCH -e job.%J.err
#SBATCH -p bluefield1
#SBATCH -A durham
#SBATCH --exclusive
#SBATCH -t 4:00:00
module purge
module load intel_comp/2022.1.2 compiler mpi
module load gsl/2.4 fftw/3.3.9epyc parmetis/4.0.3-64bit parallel_hdf5/1.12.0 ucx/1.13.0rc2
export I_MPI_PIN_DOMAIN=auto
mpirun -np $SLURM_NTASKS ../../../swift_mpi --hydro --self-gravity --stars -v 1 --threads=16 \
eagle_25.yml
exitNote that I_MPI_PIN_DOMAIN is needed for pinning.