Fritz parallel cluster (NHR+Tier3)

SSH public host keys of fritz.nhr.fau.de (as of 11/2021)

ssh-dss AAAAB3NzaC1kc3MAAACBAJj7qaKbgbgtUCB8iXOzCY8Vs0ZbdnlZsNDdKZct3vOzt2B4js2yBEGs0Fsmvjy88ro33TeI1JjMsnNP6T4wmeNPIGLkUvtX2fPcLL3c9WbVkeOf3R2b5VMRIJ+l3rVgwvHihBJFcgaAQO/mB75hdtzY6Pk5cReVYR2uidD3HkKtAAAAFQC1cIYQ0jUUXnJeeMG/t/8muhFaxQAAAIBrGWK0GhIAFu9cSy/pVXiJP7rJrSvsDGGdZ1HgB7GEjeN4O9ZFZS+vj2m4ij0LjOVtulf9LDJR56fA+3Jpjcu32/L7IPCm5/nqJd7An9/xt8D+tUPhOZfRugol9f6tV/oDRI3Y7rMDjChpjpkuN9bP2vshveHLlA0WB9Lqdgu2fgAAAIBKS/RFirbOnuP38OJ6mTXLeSlNsLEs+zW+vHhL5a08MXrAUQHYUwZplH2bNQpMyeRH55UoRJC0XDHpJzW8yafcwpO6k7uL1CWi3Gnhya9EbX2GIe8cYRrhYhcO+0M8UrVXKksVVWyAfkXZsIjTQCEcsCNhl0no5xS0/yOB6b6WzQ== fritz.nhr.fau.de
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBxc984fY6EkjlFQBFTtu/9X9EolCSz1OHNzaa8VWBj5TxV9GF8RTBXJ6why2AdK3dVrv+Qyko+X5vsMMflEiRc= fritz.nhr.fau.de
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILqv3FDYom0c4HgfCzLw9Ts2PE0GYqWaaOrM9EfQxvTI fritz.nhr.fau.de
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLCrYuIIKhA+F4hktnR4VKAZ44J6CWIMfC9mCei80YZ0294kFwpYQg3RYRLHSyL6XqLgxaZN2kFm0V0NpUpEYSP2V9eWpuyfeB6a3M1I8yy4rDagHgWuHYuX2fSm8uwfndnJJ6hV1xfZuoZrZJIMkdy8qBl4y1cxn8G6CS1KEFkeJp7wMuIdIruFbJa5eQXVgAxaqKPQYRldpK8c1OAByfQv9cBXF53cNZhtlwkUes6/PqNyU1aIodfahdYh6mxn/4Rzy+NMD0YS066P8xWP1n+bsTBpZ51pH7qTIiW1yRKFmAeFvkWVnS6N5qIwJnzB3J7DRUue9h1EhW4HCo6CEX3GCOt0kuV4ax0JgYO/Lz0cUTdDcgkWOpVtQ+WyLNUech+TsOREn19QjaK9QRriBOvBcNCnbpBXHZSqfOYGB6uggkVjyDPI6S5pclt544ie6pklAOSzrha5CLnzD4U8oVuqhFHteO39qXpbvxkUDuNsDf9t8K5fmWgCXXWtJSVhE= fritz.nhr.fau.de

fingerprints of the SSH public host keys of fritz.nhr.fau.de (as of 11/2021)

1024 SHA256:1jZOatgzkjn7G3b1/K48T+cQ8MUB60oU7j+CK5FWId8 fritz.nhr.fau.de (DSA)
256  SHA256:2NzGq1vTO//RNtVLEGrp5yMyPHmtAw6nguSxcFUHHWU fritz.nhr.fau.de (ECDSA)
256  SHA256:5km4SnsTbyBG6gX1y11imEEU8QKP8EbrqFNPce1eEU4 fritz.nhr.fau.de (ED25519)
3072 SHA256:p7+HzbUyVSjYh2hx2nQcrIuWnZZKJhnNUoI9kz+Q4yw fritz.nhr.fau.de (RSA)

SSH public host keys of cshpc.rrze.fau.de (as of 11/2021)

ssh-dss AAAAB3NzaC1kc3MAAACBAO2L8+7bhJm7OvvJMcdGSJ5/EaxvX5RRzE9RrB8fx5H69ObkqC6Baope4rOS9/+2gtnm8Q3gZ5QkostCiKT/Wex0kQQUmKn3fx6bmtExLq8YwqoRXRmNTjBIuyZuZH9w/XFK36MP63p/8h7KZXvkAzSRmNVKWzlsAg5AcTpLSs3ZAAAAFQCD0574+lRlF0WONMSuWeQDRFM4vwAAAIEAz1nRhBHZY+bFMZKMjuRnVzEddOWB/3iWEpJyOuyQWDEWYhAOEjB2hAId5Qsf+bNhscAyeKgJRNwn2KQMA2kX3O2zcfSdpSAGEgtTONX93XKkfh6JseTiFWos9Glyd04jlWzMbwjdpWvwlZjmvPI3ATsv7bcwHji3uA75PznVUikAAACBANjcvCxlW1Rjo92s7KwpismWfcpVqY7n5LxHfKRVqhr7vg/TIhs+rAK1XF/AWxyn8MHt0qlWxnEkbBoKIO5EFTvxCpHUR4TcHCx/Xkmtgeq5jWZ3Ja2bGBC3b47bHHNdDJLU2ttXysWorTXCoSYH82jr7kgP5EV+nPgwDhIMscpk cshpc.rrze.fau.de
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNVzp97t3CxlHtUiJ5ULqc/KLLH+Zw85RhmyZqCGXwxBroT+iK1Quo1jmG6kCgjeIMit9xQAHWjS/rxrlI10GIw= cshpc.rrze.fau.de
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPSIFF3lv2wTa2IQqmLZs+5Onz1DEug8krSrWM3aCDRU cshpc.rrze.fau.de
1024 35 135989634870042614980757742097308821255254102542653975453162649702179684202242220882431712465065778248253859082063925854525619976733650686605102826383502107993967196649405937335020370409719760342694143074628619457902426899384188195801203193251135968431827547590638365453993548743041030790174687920459410070371 cshpc.rrze.fau.de
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAs0wFVn1PN3DGcUtd/JHsa6s1DFOAu+Djc1ARQklFSYmxdx5GNQMvS2+SZFFa5Rcw+foAP9Ks46hWLo9mOjTV9AwJdOcSu/YWAhh+TUOLMNowpAEKj1i7L1Iz9M1yrUQsXcqDscwepB9TSSO0pSJAyrbuGMY7cK8m6//2mf7WSxc= cshpc.rrze.fau.de

fingerprints of the SSH public host keys of cshpc.rrze.fau.de (as of 11/2021)

1024 SHA256:A82eA7py46zE/TrSTCRYnJSW7LZXY16oOBxstJF3jxU cshpc.rrze.fau.de (DSA)
256  SHA256:wFaDywle3yJvygQ4ZAPDsi/iSBTaF6Uoo0i0z727aJU cshpc.rrze.fau.de (ECDSA)
256  SHA256:is52MRsxMgxHFn58o0ZUh8vCzIuE2gYanmhrxdy0rC4 cshpc.rrze.fau.de (ED25519)
1024 SHA256:Za1mKhTRFDXUwn7nhPsWc7py9a6OHqS2jin01LJC3ro cshpc.rrze.fau.de (RSA)

Intel One API is installed in the “Free User” edition via Spack.

The modules intel (and the Spack internal intel-oneapi-compilers) provides the legacy Intel compilers icc, icpc, and ifort as well as the new LLVM-based ones (icx, icpx, dpcpp, ifx).

Recommended compiler flags are:  -O3 -xHost

If you want to enable full-width AVX512 SIMD support you have to additionally set the flag: -qopt-zmm-usage=high

The modules intelmpi (and the Spack internal intel-oneapi-mpi) provides Intel MPI. To use the legacy Intel compilers with Intel MPI you just have to use the appropriate wrappers with the Intel compiler names, i.e. mpiicc, mpiicpc, mpiifort. To use the new LLVM-based Intel compilers with Intel MPI you have to specify them explicitly, i.e use mpiicc -cc=icx, mpiicpc -cxx=icpx, or mpiifort -fc=ifx. The execution of mpicc, mpicxx, and mpif90 results in using the GNU compilers.

The modules mkland tbb(and the Spack internal intel-oneapi-mkl, and intel-oneapi-tbb) provide Intel MKL and TBB. Use Intel’s MKL link line advisor to figure out the appropriate command line for linking with MKL. The Intel MKL also includes drop-in wrappers for FFTW3.

Further Intel tools may be added in the future.

The GPU compilers are available in the version coming with the operating system (currently 8.5.0) as well as modules (currently versions 9.4, 10.3, and 11.2).

Recommended compiler flags are: -O3 -xHost

If you want to enable full-width AVX512 SIMD support you have to additionally set the flag: -qopt-zmm-usage=high

Open MPI is the default MPI for the Fritz cluster. Usage of srun instead of mpirun is recommended.

Open MPI is built using Spack:

• with the compiler mentioned in the module name; the corresponding compiler will be loaded as dependency when the Open MPI modules is loaded
• without support for thread-multiple
• with fabrics=ucx
• with support for Slurm as scheduler (and internal PMIx of Open MPI)

Do not rely on the Python installation from the operating system. Use our python modules instead. These installations will be updated in place from time to time. We can add further packages from the Miniconda distribution as needed.

You can modify the Python environment as follows:

Set the location where pip and conda install packages to $WORK, see Python and Jupyter for details. By default packages will be installed in $HOME, which has limited capacity.

Extend the base environment
$ pip install --user <packages>'

Create a new one of your own
$ conda create -n <environment_name> <packages>'

Clone and modify this environment

$ conda create --name myclone --clone base
$ conda install --name myclone new_package

See also https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html.

When using gdb -p <pid> (or the equivalent attach <pid>  command in gdb) to attach to a process running in a SLURM job, you might encounter errors or warnings related to executable and library files than cannot be opened. Such issues will also prevent symbols from being resolved correctly, making debugging really difficult.

The reason that this happens is that processes in a SLURM job get a slightly different view of file system mounts (using a so-called namespace). When you want to attach GDB to a running process and use SSH to log into the node where the process is running, the gdb  process will not be in the same namespace, causing GDB to have issues to directly access the binary (and its libraries) you’re trying to debug.

The workaround is to use a slightly different method for attaching to the process:

1. $ gdb <executable>
2. (gdb) set sysroot /
3. (gdb) attach <pid>

(Thanks to our colleagues at SURFsara for figuring this out!)

Arm DDT is a powerful parallel debugger. NHR@FAU holds a license for 32 processes.

Amber usually delivers the most economic performance using GPGPUs. Thus, the Alex GPGPU cluster might be a better choice.

We provide Gromacs versions without and with PLUMED. Gromacs (and PLUMED) are built using Spack.

Gromacs often delivers the most economic performance if GPGPUs are used. Thus the Alex GPGPU cluster might be a better choice.

If running on Fritz, it is mandatory in most cased to optimizes the number of PME processes experimentally. “pme_tune” REQUIRES FURTHER WORK AS A NON-MPI BINARY HAS TO BE USED

The modules lammps/20211027-gcc11.2.0-ompi-mkl has been compiled using Gcc-11.2.0, Open MPI 4.1.1, and Intel OneAPI MKL using

• -DBUILD_SHARED_LIBS:BOOL=ON -DLAMMPS_EXCEPTIONS:BOOL=OFF -DBUILD_MPI=ON -DBUILD_OMP:BOOL=ON -DPKG_OPENMP=ON -DPKG_GPU=OFF -DBUILD_LIB=ON -DWITH_JPEG:BOOL=ON -DWITH_PNG:BOOL=ON -DWITH_FFMPEG:BOOL=ON -DPKG_ASPHERE=ON -DPKG_BODY=ON -DPKG_CLASS2=ON -DPKG_COLLOID=ON -DPKG_COMPRESS=ON -DPKG_CORESHELL=ON -DPKG_DIPOLE=ON -DPKG_GRANULAR=ON -DPKG_KSPACE=ON -DPKG_KOKKOS=ON -DPKG_LATTE=ON -DPKG_MANYBODY=ON -DPKG_MC=ON -DPKG_MEAM=OFF -DPKG_MISC=ON -DPKG_MLIAP=OFF -DPKG_MOLECULE=ON -DPKG_MPIIO=ON -DPKG_OPT=OFF -DPKG_PERI=ON -DPKG_POEMS=ON -DPKG_PYTHON=ON -DPKG_QEQ=ON -DPKG_REPLICA=ON -DPKG_RIGID=ON -DPKG_SHOCK=ON -DPKG_SNAP=ON -DPKG_SPIN=ON -DPKG_SRD=ON -DPKG_USER-ATC=ON -DPKG_USER-ADIOS=OFF -DPKG_USER-AWPMD=OFF -DPKG_USER-BOCS=OFF -DPKG_USER-CGSDK=OFF -DPKG_USER-COLVARS=OFF -DPKG_USER-DIFFRACTION=OFF -DPKG_USER-DPD=OFF -DPKG_USER-DRUDE=OFF -DPKG_USER-EFF=OFF -DPKG_USER-FEP=OFF -DPKG_USER-H5MD=ON -DPKG_USER-LB=ON -DPKG_USER-MANIFOLD=OFF -DPKG_USER-MEAMC=ON -DPKG_USER-MESODPD=OFF -DPKG_USER-MESONT=OFF -DPKG_USER-MGPT=OFF -DPKG_USER-MISC=ON -DPKG_USER-MOFFF=OFF -DPKG_USER-NETCDF=ON -DPKG_USER-OMP=ON -DPKG_USER-PHONON=OFF -DPKG_USER-PLUMED=OFF -DPKG_USER-PTM=OFF -DPKG_USER-QTB=OFF -DPKG_USER-REACTION=OFF -DPKG_USER-REAXC=ON -DPKG_USER-SDPD=OFF -DPKG_USER-SMD=OFF -DPKG_USER-SMTBQ=OFF -DPKG_USER-SPH=OFF -DPKG_USER-TALLY=OFF -DPKG_USER-UEF=OFF -DPKG_USER-YAFF=OFF -DPKG_VORONOI=ON -DPKG_KIM=ON -DFFT=MKL -DEXTERNAL_KOKKOS=ON

Run module avail lammps to see all currently installed Lammps modules. Allocate an interactive job and run mpirun -np 1 lmp -help to see which Lammps packages have been included in a specific build.

TODO – no module yet

At the moment, we provide the official pre-built Linux-x86_64-multicore binary.

At the moment we provide VASP 5.4.x and VASP 6.3.x modules to eligible users.

The module vasp6/6.3.0-hybrid-intel-impi-AVX2-with-addons includes DFTD4, libbeef, and sol_compat/VASPsol.

Interactive jobs can be requested by using salloc instead of sbatch and specifying the respective options on the command line.

The following will give you an interactive shell on one node for one hour:

salloc -N 1 --partition=singlenode --time=01:00:00

Interactive jobs can be requested by using salloc instead of sbatch and specifying the respective options on the command line.

The following will give you four nodes with an interactive shell on the first node for one hour:

salloc -N 4 --partition=multinode --time=01:00:00

In this example, the executable will be run on one node, using 72 MPI processes, i.e. one per physical core.

#!/bin/bash -l
#SBATCH --nodes=1 
#SBATCH --ntasks-per-node=72
#SBATCH --partition=singlenode
#SBATCH --time=01:00:00
#SBATCH --export=NONE

unset SLURM_EXPORT_ENV
module load XXX 

srun ./mpi_application

In this example, the executable will be run using 72 OpenMP threads (i.e. one per physical core) for a total job walltime of 1 hour.

For more efficient computation, OpenMP threads should be pinned to the compute cores. This can be achieved by the following environment variables: OMP_PLACES=cores, OMP_PROC_BIND=true. For more information, see e.g. the HPC Wiki.

#!/bin/bash -l
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=72
#SBATCH --partition=singlenode
#SBATCH --time=01:00:00
#SBATCH --export=NONE

unset SLURM_EXPORT_ENV 
module load XXX 

# set number of threads to requested cpus-per-task
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK 

./openmpi_application

In this example, the executable will be run using 2 MPI processes with  36 OpenMP threads (i.e. one per physical core) for a total job walltime of 1 hour.

For more efficient computation, OpenMP threads should be pinned to the compute cores. This can be achieved by the following environment variables: OMP_PLACES=cores, OMP_PROC_BIND=true. For more information, see e.g. the HPC Wiki.

#!/bin/bash -l
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=36
#SBATCH --partition=singlenode
#SBATCH --time=1:00:00
#SBATCH --export=NONE

unset SLURM_EXPORT_ENV 
module load XXX 

# cpus-per-task has to be set again for srun
export SRUN_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK

# set number of threads to requested cpus-per-task  
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK 

srun ./hybrid_application

In this example, the executable will be run on four nodes, using 72 MPI processes per node, i.e. one per physical core.

#!/bin/bash -l
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=72
#SBATCH --partition=multinode
#SBATCH --time=1:0:0
#SBATCH --export=NONE

unset SLURM_EXPORT_ENV 
module load XXX 

srun ./mpi_application

In this example, the executable will be run using on four nodes with 2 MPI processes per node and 36 OpenMP threads each (i.e. one per physical core) for a total job walltime of 1 hour.

For more efficient computation, OpenMP threads should be pinned to the compute cores. This can be achieved by the following environment variables: OMP_PLACES=cores, OMP_PROC_BIND=true. For more information, see e.g. the HPC Wiki.

#!/bin/bash -l
#SBATCH --partition=multinode
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=36
#SBATCH --time=01:00:00
#SBATCH --export=NONE

unset SLURM_EXPORT_ENV 
module load XXX 

# cpus-per-task has to be set again for srun
export SRUN_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK 

# set number of threads to requested cpus-per-task  
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK 

srun ./hybrid_application