Special applications, and tips & tricks

Many different applications are used by scientists from the various fields on the HPC systems at NHR@FAU. The HPC team at NHR@FAU does not have any experience with most of the applications. Nevertheless, we try to collect some useful information and tips&tricks for key applications on the following pages. The pages will not teach you how to use the applications but collect information which is specific to the usage on the HPC systems of NHR@FAU.

Central installation of software

As the parallel computers of NHR@FAU are operated stateless, all system software has to reside in a RAM disk, i.e. in main memory. Therefore, only limited packages from the Linux distribution can be installed on the compute nodes, and the compute nodes only contain a subset of the packages installed on the login nodes. Most (application) software but also libraries therefore will be installed in /apps and made available as modules. Multiple versions of a single software can be provided in that way, too.

As a general rule: software will be installed centrally

    1. if there are multiple groups which benefit from the software, or
    2. if the software is very easy to install.

In both cases NHR@FAU will more and more request that a least one group acts as mentor for the software, i.e. provides NHR@FAU with simple input for validation of the installation, but moreover, that group has to provide limited support for other groups to get started with the software.

NHR@FAU is installing moving to installing more software using Spack. Users may “chain” our central Spack installations to complement it with their own additions. Loading the module user-spack; will set up the necessary configs for you and add the command spack. However, NHR@FAU cannot provide help for your installations.

Notes on specific software

Molecular dynamics for chemistry, life science, and material science

  • Amber & AmberTools – suite of biomolecular simulation programs. Here, the term “Amber” does not refer to the set of molecular mechanical force fields for the simulation of biomolecules but to the package of molecular simulation programs consisting of the AmberTools (sander and many more) and Amber (pmemd).
  • DL_POLY / DL_POLY_CLASSIC – general-purpose classical molecular dynamics simulation software.
    DL_POLY with its fine grain IO can easily kill $FASTTMP. RRZE will kill / throttle any jobs which abuse the parallel file system.
  • GROMACS – versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles. It is primarily designed for biochemical molecules like proteins, lipids and nucleic acids that have a lot of complicated bonded interactions, but since GROMACS is extremely fast at calculating the non-bonded interactions (that usually dominate simulations) many groups are also using it for research on non-biological systems, e.g. polymers.
  • IMD – software package for classical molecular dynamics simulations. Several types of interactions are supported and a rich choice of simulation options is available. There is no restriction on the number of particle types.
  • LAMMPS – classical molecular dynamics code with a focus on materials modeling. LAMMPS has potentials for solid-state materials (metals, semiconductors) and soft matter (biomolecules, polymers) and coarse-grained or mesoscopic systems. It can be used to model atoms or, more generically, as a parallel particle simulator at the atomic, meso, or continuum scale.
  • NAMD – a molecular dynamics program designed for parallel computation. Full and efficient treatment of electrostatic and van der Waals interactions are provided via the Particle Mesh Ewald algorithm.

Quantum chemistry

  • CP2K – quantum chemistry and solid state physics program package to perform atomistic simulations of solid state, liquid, molecular, periodic, material, crystal, and biological systems.
  • CPMD – parallelized plane wave / pseudopotential implementation of Density Functional Theory, particularly designed for ab-initio molecular dynamics.
  • Molpro – a comprehensive system of ab initio programs for advanced molecular electronic structure calculations.
  • OpenMolcas – a quantum chemistry software package using the multiconfigurational approach to the electronic structure.
  • ORCA – an ab initio quantum chemistry program package that contains modern electronic structure methods including density functional theory, many-body perturbation, coupled cluster, multireference methods, and semi-empirical quantum chemistry methods. Its main field of application is larger molecules, transition metal complexes, and their spectroscopic properties.
  • Quantum Espresso – integrated suite of open-source computer codes for electronic-structure calculations and materials modeling at the nanoscale. It is based on density-functional theory, plane waves, and pseudopotentials.
  • TURBOMOLE – program package for ab initio electronic structure calculations.
  • VASP – Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modeling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.
    Not yet installed centrally – but contact us if you have an own license but would like to use a RRZE provided binary.

Computational fluid dynamics (CFD), multiphysics and FE, climatology and glaciology

  • ANSYS CFX – commercial CFD software tool with a special focus on turbomachinery, such as pumps, fans, compressors and gas and hydraulic turbines.
  • ANSYS Fluent – commercial CFD to model flow, turbulence, heat transfer and reactions in all sorts of applications.
  • ANSYS Mechanical – commercial finite element analysis (FEA) tool, to solve complex mechanical problems.
  • Simcenter STAR-CCM+ – commercial software tool for CFD and more generally computational continuum mechanics (by CD-adapco or nowadays Siemens PLM).
  • OpenFOAM.org/OpenFOAM.com – leading free, open-source software for computational fluid dynamics but also an extensive C++ toolbox for the development of customized numerical solvers, and pre-/post-processing utilities for the solution of continuum mechanics problems.
  • Abaqus – software suite for finite element analysis and computer-aided engineering.
    Currently no (known) active users on the HPC systems.
  • COMSOL Multiphysics – general-purpose simulation software for modeling designs, devices, and processes in all fields of engineering, manufacturing, and scientific research.
  • Elmer/Ice – open-source Finite Element software for ice sheet, glaciers, and ice flow modeling.
    Avoid MATC expressions in the SIF input.
  • WRF – Weather Research and Forecasting (WRF) is a next-generation mesoscale numerical weather prediction system designed for both atmospheric research and operational forecasting applications.

Mathematical and statistical software

  • Matlab – numerical computing environment and proprietary programming language developed by MathWorks
  • Python and Jupyter – Python is a versatile scripting language (often used by data scientists); Jupyter is a sort of interactive web-based user environment
  • R and R Studio – “Microsoft R Open” is installed on Woody. It is an enhanced distribution of R. R Studio is a HTML5-based IDE for R.
    Additional packages available from MRAN; can be installed centrally upon request. You are also able to install packages on your own in user space.

Machine learning and big data

  • cuDNN in version 7 is installed on TinyGPU and the GPU nodes of Emmy since autumn 2019 as Nvidia relaxed the license conditions.
    There is no need to load a module as the include files and libraries are in /usr/include and /usr/lib[64].
  • TensorFlow & co
    RRZE currently does not provide any central installation for two three reasons: (a) the software changes very rapidly; (b) NVIDIA imposes very strict conditions on cuDNN – you can only get a personal license; (c) no group volunteered as mentor so far.
  • OpenPose – multi-person gesture detection (i.e. human body, hand, facial, and foot keypoints)
    A detailed description on how to get OpenPose installed on RRZE’s HPC systems can be found in http://peter-uhrig.de/openpose-with-nvcaffe-in-a-singularity-container-with-support-for-multiple-architectures/

Physics

  • software for the astrophysics community is maintained by ECAP
  • several packages for quantum computing are collected in the quantumtools module.

Virtualization and containers

  • Singularity in version 3.x or Apptainer is available on all HPC systems. Thus, you can run your own environment in a docker-like fashion. More information about the general usage of Singularity available in this HPC Cafe presentation.
    Note: When importing containers e.g. from dockerhub set the environment variable SINGULARITY_CACHEDIR (and APPTAINER_CACHEDIR respectively) to save space in $HOME as it defaults to ~/.singularity/docker/ and ~/.singularity/metadata/.
    Building singularity contains on the HPC systems themselves may work in the meantime thanks to user namespaces.
    Our colleagues from GWDG put together a nice intro to Singularity – we link it here although many details will not work 1:1 on our system: https://gitlab-ce.gwdg.de/nhr-container/container-workshop-jupyter

 

HPC tools and software development

  • DDT – parallel debugger.
  • TotalView – parallel debugger. See in HPC-Wiki.
  • LIKWID – tool suite for performance-oriented programmers and administrators. The term LIKWID stands for ‘Like I know what I do’. See in HPC-Wiki and on Github .
  • Intel Trace Collector / Analyzer – tool for profiling and checking the correctness of MPI communication. A short how-to is available here or in the HPC-Wiki.
  • Intel Vtune & friends (advisor, inspector, performance_snapshots, vtune_amplifier) – tools for detailed performance profiling. See HPC-Wiki for Vtune and Intel Advisor.
    Only available on special machines – contact hpc-support@fau.de for details.
  • Continuous Integration / Gitlab CI – Continuous Integration allows you to run a pipeline of scripts to build, test, and validate the code changes before merging them into the main branch.
  • Working with NVIDIA GPUs