FAQ

Categories: Usage | Software | Hardware | General information

Usage

Alex Cluster

How can I access the new clusters Alex and Fritz?

FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there.

If you do not have an HPC account, please follow our instructions on “Getting started with HPC“.

External scientists have to submit a NHR proposal to get access.

How can I request an interactive job on Alex?

Interactive jobs can be requested by using salloc and specifying the respective options on the command line.

The following will give you an interactive shell on one of the A40 nodes for one hour:
salloc --gres=gpu:a40:1 --partition=a40 --time=01:00:00

Note that settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job!

This and more information can be found in our documention on Alex.

How can I run my job on the cluster?

To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available.

Please do not run your jobs on the cluster frontends!

The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there.

Please consult our documentation for details about Batch Processing.

We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

Why is my code not using the GPU on Alex?

CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc.

Please also have a look at our documentation “Working with NVIDIA GPUs“.

Basic HPC Knowledge

What is a parallel file system?

In a parallel file system (PFS), data is distributed not only across several disks but also multiple servers in order to increase the data access bandwidth. Most PFS’s are connected to the high-speed network of a cluster, and aggregated bandwidths in the TByte/s range are not uncommon. High bandwidth can, however, only be obtained with large files and streaming access.

For information on how to use a parallel file system on our clusters, please read our documentation on “Parallel file systems $FASTTMP“.

What is SMT (also known as hyperthreading)?

Simultaneous multi-threading (SMT) allows a CPU core to run more than one software thread at the same time. These “hardware threads” a.k.a. “virtual cores” share almost all resources. The purpose of this feature is to make better use of the execution units within the core. It is rather hard to predict the benefit of SMT for real applications, so the best strategy is to try it using a well-designed, realistic benchmark case.

What is thread or process affinity?

Modern multicore systems have a strong topology, i.e., groups of hardware threads share different resources such as cores, caches, and memory interfaces. Many performance features of parallel programs depend on where their threads and processes are running in the machine. This makes it vital to bind these threads and processes to hardware threads so that variability is reduced and resources are balanced.

Why should I care about file systems?

Not only may efficient file operations speed up your own code (if file I/O is what you must do); they will also reduce the burden on shared file servers and thus leave more performance headroom for other users of the resource. Hence, it is a matter of thoughtfulness to optimize file accesses even if your performance gain is marginal.

Batch System

How can I request an interactive job on Alex?

Interactive jobs can be requested by using salloc and specifying the respective options on the command line.

The following will give you an interactive shell on one of the A40 nodes for one hour:
salloc --gres=gpu:a40:1 --partition=a40 --time=01:00:00

Note that settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job!

This and more information can be found in our documention on Alex.

How can I request an interactive job on Fritz?

Interactive jobs can be requested by using salloc and specifying the respective options on the command line.

The following will give you an interactive shell on one node for one hour:
salloc -N 1 --partition=singlenode --time=01:00:00

The following will give you four nodes with an interactive shell on the first node for one hour:
salloc -N 4 --partition=multinode --time=01:00:00

Settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job!

This and more information can be found in our documentation on Fritz.

How can I request an interactive job on TinyGPU?

Interactive Slurm Shell (RTX2080Ti, RTX3080, V100 and A100 nodes only)

To generate an interactive Slurm shell on one of the compute nodes, the following command has to be issued on the woody frontend:
salloc.tinygpu --gres=gpu:1 --time=00:30:00

This will give you an interactive shell for 30 minutes on one of the nodes, allocating 1 GPU and the respective number of CPU cores. There, you can then for example compile your code or do test runs of your binary. For MPI-parallel binaries, use sruninstead of mpirun.

Please note that sallocautomatically exports the environment of your shell on the login node to your interactive job. This can cause problems if you have loaded any modules due to the version differences between the woody frontend and the TinyGPU compute nodes. To mitigate this, purge all loaded modules via module purge before issuing the salloc command.

This and more information can be found in our documentation about TinyGPU.

How can I request an interactive job on Woody-NG?

Interactive jobs can be requested by using salloc and specifying the respective options on the command line.

The following will give you an interactive shell on one node with one core dedicated to you for one hour:
salloc -n 1 --time=01:00:00

Settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job!

This and more information can be found in our documentation about Woody-NG.

How can I run my job on the cluster?

To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available.

Please do not run your jobs on the cluster frontends!

The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there.

Please consult our documentation for details about Batch Processing.

We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.

CUDA

Why is my code not running on the GPU in TinyGPU?

Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version).

Please have look at our documentation about “Working with NVIDIA GPUs“.

Why is my code not using the GPU on Alex?

CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc.

Please also have a look at our documentation “Working with NVIDIA GPUs“.

Cluster Access

How can I access the cluster frontends?

Almost all HPC systems at NHR@FAU use private IP addresses that can only be accessed directly from within the FAU.There are three options for logging in to the clusters from outside the university:

  1. Use a VPN (Virtual Private Network) connection.
  2. Use IPv6. The cluster frontends have world-visible IPv6 addresses.
  3. Use our “dialog server” cshpc.rrze.fau.de. The dialog server is the only HPC machine with a public IPv4 address. cshpc is a Linux system that is open to all HPC accounts. From this machine, you can log into any NHR@FAU system. A more complete description can be found on our documentation pages.

Whichever option you choose, you need to use SSH. Please consult our extensive SSH documentation pages for details.

 

How can I access the new clusters Alex and Fritz?

FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there.

If you do not have an HPC account, please follow our instructions on “Getting started with HPC“.

External scientists have to submit a NHR proposal to get access.

How can I get access to HPC systems?

Getting an HPC account

Depending on the status, there are different protocols to get an HPC account:

  • NHR users from outside FAU; See the page on NHR application rules for up-to-date information on allocating resources of NHR@FAU.
    Also check the pages on the NHR@FAU HPC-Portal Usage /New digital workflow for HPC accounts.
  • FAU staff and students (except for lectures): use the HPC application form. Details on how to fill the form are given below. Basic usage of the HPC systems typically is free of charge for FAU researchers for publicly funded research. For compute needs beyond the free basic usage see the page on NHR application rules for preliminary information on allocating resources of NHR@FAU.
  • Lectures of FAU with need for HPC access: there is a simplified protocol to get HPC accounts for all students of your course. Lecturer have to approach HPC support with a list of the IdM accounts of all students of the course and  the course name. Late registrations of additional students are not possible. Thus, be sure to collect all IdM accounts before sending the list to RRZE.
  • Block courses with external participants: Lecturer have to approach HPC support at least one week in advance with title and date of the course, and the expected number of participants. Such accounts cannot be valid for more than one week.

The HPC application form for FAU within HPC4FAU

You can get the application form here: HPC application form. Applications always have to be approved by your local chair / institute – we do not serve private persons. If you have any questions regarding the application, please contact your local IT contact person at the chair / institute.

You need to fill out the application form, print it, sign it and let it be stamped with the Chair or Institute seal.

Once it is ready, you can bring it by the RRZE Service Desk or send it via Email, or internal mail.

Please visit our documentation about Getting started with HPC for more information.

How can I run my job on the cluster?

To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available.

Please do not run your jobs on the cluster frontends!

The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there.

Please consult our documentation for details about Batch Processing.

We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

FileSystems/Data Storage

How can I leverage node-local storage on TinyGPU to increase job performance?

Each node has at least 880 GB of local SSD capacity for temporary files under $TMPDIR.

The directory $TMPDIR will be deleted automatically as soon as the user has no jobs running on the node any more.

Data to be kept can be copied to a cluster-wide volume at the end of the job.

Please also read our documentation on “File Systems“.

What is a parallel file system?

In a parallel file system (PFS), data is distributed not only across several disks but also multiple servers in order to increase the data access bandwidth. Most PFS’s are connected to the high-speed network of a cluster, and aggregated bandwidths in the TByte/s range are not uncommon. High bandwidth can, however, only be obtained with large files and streaming access.

For information on how to use a parallel file system on our clusters, please read our documentation on “Parallel file systems $FASTTMP“.

Where can I store my data?

Your home directory is accessible via $HOME. Each user gets a standard quota of 50 Gigabytes and quota extensions are not possible.

Additional storage is accessible via $HPCVAULT. Here, the default quota for each user is 500 Gigabytes.

The recommended work directory is accessible via $WORK. The standard quota for each user is 500 Gigabytes.

All three directories ($HOME, $HPCVAULT and $WORK) are available throughout our HPC systems.

We recommend you use the aforementioned variables in your jobscripts and not rely on the specific paths as this may change over time, i.e. when directories are relocated to a different NFS server.

Job-specific storage (either located in main memory [RAM disk] or, if available, local HDD / SDD) is accessible via $TMPDIR and always node-local. Size differs between clusters and is only available during job lifetime. Data is flushed after the job finishes!

Some of our clusters have a local parallel filesystem for high performance short-term storage that is accessible via $FASTTMP. These filesystems are specific to the clusters and not available on other clusters. This type of storage is not suitable for programs such as MD simulations that have quite high output rates!

Please also have a look into our documentation on “File Systems“.

Why should I care about file systems?

Not only may efficient file operations speed up your own code (if file I/O is what you must do); they will also reduce the burden on shared file servers and thus leave more performance headroom for other users of the resource. Hence, it is a matter of thoughtfulness to optimize file accesses even if your performance gain is marginal.

Why the need for several file systems?

Different file systems have different features; for example, a central NFS server has massive bytes for the buck but limited data bandwidth, while a parallel file system is much faster but smaller and usually available to one cluster only. A node-local SSD, one the other hand, has the advantage of very low latency but it cannot be accessed from outside a compute node.

For further information, please consult our documentation on “File systems“.

Fritz Cluster

How can I access the new clusters Alex and Fritz?

FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there.

If you do not have an HPC account, please follow our instructions on “Getting started with HPC“.

External scientists have to submit a NHR proposal to get access.

How can I request an interactive job on Fritz?

Interactive jobs can be requested by using salloc and specifying the respective options on the command line.

The following will give you an interactive shell on one node for one hour:
salloc -N 1 --partition=singlenode --time=01:00:00

The following will give you four nodes with an interactive shell on the first node for one hour:
salloc -N 4 --partition=multinode --time=01:00:00

Settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job!

This and more information can be found in our documentation on Fritz.

How can I run my job on the cluster?

To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available.

Please do not run your jobs on the cluster frontends!

The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there.

Please consult our documentation for details about Batch Processing.

We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

GPU usage

How can I run my job on the cluster?

To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available.

Please do not run your jobs on the cluster frontends!

The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there.

Please consult our documentation for details about Batch Processing.

We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.

Why is my code not running on the GPU in TinyGPU?

Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version).

Please have look at our documentation about “Working with NVIDIA GPUs“.

Why is my code not using the GPU on Alex?

CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc.

Please also have a look at our documentation “Working with NVIDIA GPUs“.

Login

How can I access the cluster frontends?

Almost all HPC systems at NHR@FAU use private IP addresses that can only be accessed directly from within the FAU.There are three options for logging in to the clusters from outside the university:

  1. Use a VPN (Virtual Private Network) connection.
  2. Use IPv6. The cluster frontends have world-visible IPv6 addresses.
  3. Use our “dialog server” cshpc.rrze.fau.de. The dialog server is the only HPC machine with a public IPv4 address. cshpc is a Linux system that is open to all HPC accounts. From this machine, you can log into any NHR@FAU system. A more complete description can be found on our documentation pages.

Whichever option you choose, you need to use SSH. Please consult our extensive SSH documentation pages for details.

 

How can I run my job on the cluster?

To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available.

Please do not run your jobs on the cluster frontends!

The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there.

Please consult our documentation for details about Batch Processing.

We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.

Software environment

How can I run my job on the cluster?

To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available.

Please do not run your jobs on the cluster frontends!

The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there.

Please consult our documentation for details about Batch Processing.

We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

Why is my code not running on the GPU in TinyGPU?

Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version).

Please have look at our documentation about “Working with NVIDIA GPUs“.

Why is my code not using the GPU on Alex?

CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc.

Please also have a look at our documentation “Working with NVIDIA GPUs“.

TinyGPU Cluster

How can I leverage node-local storage on TinyGPU to increase job performance?

Each node has at least 880 GB of local SSD capacity for temporary files under $TMPDIR.

The directory $TMPDIR will be deleted automatically as soon as the user has no jobs running on the node any more.

Data to be kept can be copied to a cluster-wide volume at the end of the job.

Please also read our documentation on “File Systems“.

How can I request an interactive job on TinyGPU?

Interactive Slurm Shell (RTX2080Ti, RTX3080, V100 and A100 nodes only)

To generate an interactive Slurm shell on one of the compute nodes, the following command has to be issued on the woody frontend:
salloc.tinygpu --gres=gpu:1 --time=00:30:00

This will give you an interactive shell for 30 minutes on one of the nodes, allocating 1 GPU and the respective number of CPU cores. There, you can then for example compile your code or do test runs of your binary. For MPI-parallel binaries, use sruninstead of mpirun.

Please note that sallocautomatically exports the environment of your shell on the login node to your interactive job. This can cause problems if you have loaded any modules due to the version differences between the woody frontend and the TinyGPU compute nodes. To mitigate this, purge all loaded modules via module purge before issuing the salloc command.

This and more information can be found in our documentation about TinyGPU.

How can I run my job on the cluster?

To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available.

Please do not run your jobs on the cluster frontends!

The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there.

Please consult our documentation for details about Batch Processing.

We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

Why is my code not running on the GPU in TinyGPU?

Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version).

Please have look at our documentation about “Working with NVIDIA GPUs“.

Woody Cluster

How can I request an interactive job on Woody-NG?

Interactive jobs can be requested by using salloc and specifying the respective options on the command line.

The following will give you an interactive shell on one node with one core dedicated to you for one hour:
salloc -n 1 --time=01:00:00

Settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job!

This and more information can be found in our documentation about Woody-NG.

How can I run my job on the cluster?

To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available.

Please do not run your jobs on the cluster frontends!

The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there.

Please consult our documentation for details about Batch Processing.

We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

How can I access the new clusters Alex and Fritz?
FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there. If you do not have an HPC account, please follow our instructions on “Getting started with HPC“. External scientists have to submit a NHR proposal to get access.
How can I request an interactive job on Alex?
Interactive jobs can be requested by using salloc and specifying the respective options on the command line. The following will give you an interactive shell on one of the A40 nodes for one hour: salloc --gres=gpu:a40:1 --partition=a40 --time=01:00:00 Note that settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job! This and more information can be found in our documention on Alex.
How can I run my job on the cluster?
To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available. Please do not run your jobs on the cluster frontends! The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. Please consult our documentation for details about Batch Processing. We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
Why is my code not using the GPU on Alex?
CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc. Please also have a look at our documentation “Working with NVIDIA GPUs“.
What is a parallel file system?
In a parallel file system (PFS), data is distributed not only across several disks but also multiple servers in order to increase the data access bandwidth. Most PFS’s are connected to the high-speed network of a cluster, and aggregated bandwidths in the TByte/s range are not uncommon. High bandwidth can, however, only be obtained with large files and streaming access. For information on how to use a parallel file system on our clusters, please read our documentation on “Parallel file systems $FASTTMP“.
What is SMT (also known as hyperthreading)?
Simultaneous multi-threading (SMT) allows a CPU core to run more than one software thread at the same time. These “hardware threads” a.k.a. “virtual cores” share almost all resources. The purpose of this feature is to make better use of the execution units within the core. It is rather hard to predict the benefit of SMT for real applications, so the best strategy is to try it using a well-designed, realistic benchmark case.
What is thread or process affinity?
Modern multicore systems have a strong topology, i.e., groups of hardware threads share different resources such as cores, caches, and memory interfaces. Many performance features of parallel programs depend on where their threads and processes are running in the machine. This makes it vital to bind these threads and processes to hardware threads so that variability is reduced and resources are balanced.
Why should I care about file systems?
Not only may efficient file operations speed up your own code (if file I/O is what you must do); they will also reduce the burden on shared file servers and thus leave more performance headroom for other users of the resource. Hence, it is a matter of thoughtfulness to optimize file accesses even if your performance gain is marginal.
How can I request an interactive job on Alex?
Interactive jobs can be requested by using salloc and specifying the respective options on the command line. The following will give you an interactive shell on one of the A40 nodes for one hour: salloc --gres=gpu:a40:1 --partition=a40 --time=01:00:00 Note that settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job! This and more information can be found in our documention on Alex.
How can I request an interactive job on Fritz?
Interactive jobs can be requested by using salloc and specifying the respective options on the command line. The following will give you an interactive shell on one node for one hour: salloc -N 1 --partition=singlenode --time=01:00:00 The following will give you four nodes with an interactive shell on the first node for one hour: salloc -N 4 --partition=multinode --time=01:00:00 Settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job! This and more information can be found in our documentation on Fritz.
How can I request an interactive job on TinyGPU?
Interactive Slurm Shell (RTX2080Ti, RTX3080, V100 and A100 nodes only) To generate an interactive Slurm shell on one of the compute nodes, the following command has to be issued on the woody frontend: salloc.tinygpu --gres=gpu:1 --time=00:30:00 This will give you an interactive shell for 30 minutes on one of the nodes, allocating 1 GPU and the respective number of CPU cores. There, you can then for example compile your code or do test runs of your binary. For MPI-parallel binaries, use sruninstead of mpirun. Please note that sallocautomatically exports the environment of your shell on the login node to your interactive job. This can cause problems if you have loaded any modules due to the version differences between the woody frontend and the TinyGPU compute nodes. To mitigate this, purge all loaded modules via module purge before issuing the salloc command. This and more information can be found in our documentation about TinyGPU.
How can I request an interactive job on Woody-NG?
Interactive jobs can be requested by using salloc and specifying the respective options on the command line. The following will give you an interactive shell on one node with one core dedicated to you for one hour: salloc -n 1 --time=01:00:00 Settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job! This and more information can be found in our documentation about Woody-NG.
How can I run my job on the cluster?
To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available. Please do not run your jobs on the cluster frontends! The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. Please consult our documentation for details about Batch Processing. We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.
Why is my code not running on the GPU in TinyGPU?
Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version). Please have look at our documentation about “Working with NVIDIA GPUs“.
Why is my code not using the GPU on Alex?
CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc. Please also have a look at our documentation “Working with NVIDIA GPUs“.
How can I access the cluster frontends?
Almost all HPC systems at NHR@FAU use private IP addresses that can only be accessed directly from within the FAU.There are three options for logging in to the clusters from outside the university: Use a VPN (Virtual Private Network) connection. Use IPv6. The cluster frontends have world-visible IPv6 addresses. Use our “dialog server” cshpc.rrze.fau.de. The dialog server is the only HPC machine with a public IPv4 address. cshpc is a Linux system that is open to all HPC accounts. From this machine, you can log into any NHR@FAU system. A more complete description can be found on our documentation pages. Whichever option you choose, you need to use SSH. Please consult our extensive SSH documentation pages for details.  
How can I access the new clusters Alex and Fritz?
FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there. If you do not have an HPC account, please follow our instructions on “Getting started with HPC“. External scientists have to submit a NHR proposal to get access.
How can I get access to HPC systems?
Getting an HPC account Depending on the status, there are different protocols to get an HPC account: NHR users from outside FAU; See the page on NHR application rules for up-to-date information on allocating resources of NHR@FAU. Also check the pages on the NHR@FAU HPC-Portal Usage /New digital workflow for HPC accounts. FAU staff and students (except for lectures): use the HPC application form. Details on how to fill the form are given below. Basic usage of the HPC systems typically is free of charge for FAU researchers for publicly funded research. For compute needs beyond the free basic usage see the page on NHR application rules for preliminary information on allocating resources of NHR@FAU. Lectures of FAU with need for HPC access: there is a simplified protocol to get HPC accounts for all students of your course. Lecturer have to approach HPC support with a list of the IdM accounts of all students of the course and  the course name. Late registrations of additional students are not possible. Thus, be sure to collect all IdM accounts before sending the list to RRZE. Block courses with external participants: Lecturer have to approach HPC support at least one week in advance with title and date of the course, and the expected number of participants. Such accounts cannot be valid for more than one week. The HPC application form for FAU within HPC4FAU You can get the application form here: HPC application form. Applications always have to be approved by your local chair / institute – we do not serve private persons. If you have any questions regarding the application, please contact your local IT contact person at the chair / institute. You need to fill out the application form, print it, sign it and let it be stamped with the Chair or Institute seal. Once it is ready, you can bring it by the RRZE Service Desk or send it via Email, or internal mail. Please visit our documentation about Getting started with HPC for more information.
How can I run my job on the cluster?
To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available. Please do not run your jobs on the cluster frontends! The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. Please consult our documentation for details about Batch Processing. We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
How can I leverage node-local storage on TinyGPU to increase job performance?
Each node has at least 880 GB of local SSD capacity for temporary files under $TMPDIR. The directory $TMPDIR will be deleted automatically as soon as the user has no jobs running on the node any more. Data to be kept can be copied to a cluster-wide volume at the end of the job. Please also read our documentation on “File Systems“.
What is a parallel file system?
In a parallel file system (PFS), data is distributed not only across several disks but also multiple servers in order to increase the data access bandwidth. Most PFS’s are connected to the high-speed network of a cluster, and aggregated bandwidths in the TByte/s range are not uncommon. High bandwidth can, however, only be obtained with large files and streaming access. For information on how to use a parallel file system on our clusters, please read our documentation on “Parallel file systems $FASTTMP“.
Where can I store my data?
Your home directory is accessible via $HOME. Each user gets a standard quota of 50 Gigabytes and quota extensions are not possible. Additional storage is accessible via $HPCVAULT. Here, the default quota for each user is 500 Gigabytes. The recommended work directory is accessible via $WORK. The standard quota for each user is 500 Gigabytes. All three directories ($HOME, $HPCVAULT and $WORK) are available throughout our HPC systems. We recommend you use the aforementioned variables in your jobscripts and not rely on the specific paths as this may change over time, i.e. when directories are relocated to a different NFS server. Job-specific storage (either located in main memory [RAM disk] or, if available, local HDD / SDD) is accessible via $TMPDIR and always node-local. Size differs between clusters and is only available during job lifetime. Data is flushed after the job finishes! Some of our clusters have a local parallel filesystem for high performance short-term storage that is accessible via $FASTTMP. These filesystems are specific to the clusters and not available on other clusters. This type of storage is not suitable for programs such as MD simulations that have quite high output rates! Please also have a look into our documentation on “File Systems“.
Why should I care about file systems?
Not only may efficient file operations speed up your own code (if file I/O is what you must do); they will also reduce the burden on shared file servers and thus leave more performance headroom for other users of the resource. Hence, it is a matter of thoughtfulness to optimize file accesses even if your performance gain is marginal.
Why the need for several file systems?
Different file systems have different features; for example, a central NFS server has massive bytes for the buck but limited data bandwidth, while a parallel file system is much faster but smaller and usually available to one cluster only. A node-local SSD, one the other hand, has the advantage of very low latency but it cannot be accessed from outside a compute node. For further information, please consult our documentation on “File systems“.
How can I access the new clusters Alex and Fritz?
FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there. If you do not have an HPC account, please follow our instructions on “Getting started with HPC“. External scientists have to submit a NHR proposal to get access.
How can I request an interactive job on Fritz?
Interactive jobs can be requested by using salloc and specifying the respective options on the command line. The following will give you an interactive shell on one node for one hour: salloc -N 1 --partition=singlenode --time=01:00:00 The following will give you four nodes with an interactive shell on the first node for one hour: salloc -N 4 --partition=multinode --time=01:00:00 Settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job! This and more information can be found in our documentation on Fritz.
How can I run my job on the cluster?
To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available. Please do not run your jobs on the cluster frontends! The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. Please consult our documentation for details about Batch Processing. We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
How can I run my job on the cluster?
To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available. Please do not run your jobs on the cluster frontends! The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. Please consult our documentation for details about Batch Processing. We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.
Why is my code not running on the GPU in TinyGPU?
Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version). Please have look at our documentation about “Working with NVIDIA GPUs“.
Why is my code not using the GPU on Alex?
CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc. Please also have a look at our documentation “Working with NVIDIA GPUs“.
How can I access the cluster frontends?
Almost all HPC systems at NHR@FAU use private IP addresses that can only be accessed directly from within the FAU.There are three options for logging in to the clusters from outside the university: Use a VPN (Virtual Private Network) connection. Use IPv6. The cluster frontends have world-visible IPv6 addresses. Use our “dialog server” cshpc.rrze.fau.de. The dialog server is the only HPC machine with a public IPv4 address. cshpc is a Linux system that is open to all HPC accounts. From this machine, you can log into any NHR@FAU system. A more complete description can be found on our documentation pages. Whichever option you choose, you need to use SSH. Please consult our extensive SSH documentation pages for details.  
How can I run my job on the cluster?
To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available. Please do not run your jobs on the cluster frontends! The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. Please consult our documentation for details about Batch Processing. We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.
How can I run my job on the cluster?
To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available. Please do not run your jobs on the cluster frontends! The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. Please consult our documentation for details about Batch Processing. We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
Why is my code not running on the GPU in TinyGPU?
Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version). Please have look at our documentation about “Working with NVIDIA GPUs“.
Why is my code not using the GPU on Alex?
CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc. Please also have a look at our documentation “Working with NVIDIA GPUs“.
How can I leverage node-local storage on TinyGPU to increase job performance?
Each node has at least 880 GB of local SSD capacity for temporary files under $TMPDIR. The directory $TMPDIR will be deleted automatically as soon as the user has no jobs running on the node any more. Data to be kept can be copied to a cluster-wide volume at the end of the job. Please also read our documentation on “File Systems“.
How can I request an interactive job on TinyGPU?
Interactive Slurm Shell (RTX2080Ti, RTX3080, V100 and A100 nodes only) To generate an interactive Slurm shell on one of the compute nodes, the following command has to be issued on the woody frontend: salloc.tinygpu --gres=gpu:1 --time=00:30:00 This will give you an interactive shell for 30 minutes on one of the nodes, allocating 1 GPU and the respective number of CPU cores. There, you can then for example compile your code or do test runs of your binary. For MPI-parallel binaries, use sruninstead of mpirun. Please note that sallocautomatically exports the environment of your shell on the login node to your interactive job. This can cause problems if you have loaded any modules due to the version differences between the woody frontend and the TinyGPU compute nodes. To mitigate this, purge all loaded modules via module purge before issuing the salloc command. This and more information can be found in our documentation about TinyGPU.
How can I run my job on the cluster?
To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available. Please do not run your jobs on the cluster frontends! The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. Please consult our documentation for details about Batch Processing. We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
Why is my code not running on the GPU in TinyGPU?
Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version). Please have look at our documentation about “Working with NVIDIA GPUs“.
How can I request an interactive job on Woody-NG?
Interactive jobs can be requested by using salloc and specifying the respective options on the command line. The following will give you an interactive shell on one node with one core dedicated to you for one hour: salloc -n 1 --time=01:00:00 Settings from the calling shell (e.g. loaded module paths) will be inherited by the interactive job! This and more information can be found in our documentation about Woody-NG.
How can I run my job on the cluster?
To submit a job to one of our cluster, you first have to login to a cluster frontend. The compute nodes are not directly accessible and we have a batch system running that handles the queuing of jobs into different partitions (depending on the needed resources, e.g. runtime) and sorting according to some priority scheme. A job will run when the required resources become available. Please do not run your jobs on the cluster frontends! The login nodes are not suitable for computational work, since they are shared among all users. We do not allow MPI-parallel applications on the frontends and short parallel test runs must be performed using batch jobs. It is possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. Please consult our documentation for details about Batch Processing. We also provide general job script examples for parallel jobs and GPU jobs; however, we have also prepared more specific job scripts for applications that our users frequently run on the clusters.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

Software

Alex Cluster

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

Why is my code not using the GPU on Alex?

CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc.

Please also have a look at our documentation “Working with NVIDIA GPUs“.

CUDA

Why is my code not running on the GPU in TinyGPU?

Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version).

Please have look at our documentation about “Working with NVIDIA GPUs“.

Why is my code not using the GPU on Alex?

CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc.

Please also have a look at our documentation “Working with NVIDIA GPUs“.

Cluster Access

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

Continuous X

What is Continuous Benchmarking (CB)?

CB can be seen as a variant of CT, where not only functionality but also performance is tested in order to avoid regressions, i.e., unwanted performance degradation due to code changes.

Please also see our documentation on “Continuous Integration / Gitlab Cx“.

What is Continuous Deployment (CD)?

CD is the automatic deployment of the software coming out of the other Cx processes. This can be the installation on a particular system, rolling out a revision within a whole organization, pushing installation packages to public repositories, etc.

Please also see our documentation on “Continuous Integration / Gitlab Cx“.

What is Continuous Integration (CI)?

Continuous Integration is the practice of automatically integrating code changes into a software project. It relies on a code repository that supports automated building and testing. Often, CI also involves setting up a build system from scratch, including all dependencies.

Please also see our documentation on “Continuous Integration / Gitlab Cx“.

What is Continuous Testing (CT)?

It is the practice of executing automated tests as an integral part of the software development process. It tries to make sure that no functionality is lost and no errors are introduced during development.

Please also see our documentation on “Continuous Integration / Gitlab Cx“.

Fritz Cluster

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

GPU usage

Why is my code not running on the GPU in TinyGPU?

Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version).

Please have look at our documentation about “Working with NVIDIA GPUs“.

Why is my code not using the GPU on Alex?

CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc.

Please also have a look at our documentation “Working with NVIDIA GPUs“.

Software environment

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

Why is my code not running on the GPU in TinyGPU?

Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version).

Please have look at our documentation about “Working with NVIDIA GPUs“.

Why is my code not using the GPU on Alex?

CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc.

Please also have a look at our documentation “Working with NVIDIA GPUs“.

TinyGPU Cluster

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

Why is my code not running on the GPU in TinyGPU?

Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version).

Please have look at our documentation about “Working with NVIDIA GPUs“.

Woody Cluster

The software I need is not installed. What can I do now?

On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found.

To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later.

For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“.

Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager.

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
Why is my code not using the GPU on Alex?
CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc. Please also have a look at our documentation “Working with NVIDIA GPUs“.
Why is my code not running on the GPU in TinyGPU?
Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version). Please have look at our documentation about “Working with NVIDIA GPUs“.
Why is my code not using the GPU on Alex?
CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc. Please also have a look at our documentation “Working with NVIDIA GPUs“.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
What is Continuous Benchmarking (CB)?
CB can be seen as a variant of CT, where not only functionality but also performance is tested in order to avoid regressions, i.e., unwanted performance degradation due to code changes. Please also see our documentation on “Continuous Integration / Gitlab Cx“.
What is Continuous Deployment (CD)?
CD is the automatic deployment of the software coming out of the other Cx processes. This can be the installation on a particular system, rolling out a revision within a whole organization, pushing installation packages to public repositories, etc. Please also see our documentation on “Continuous Integration / Gitlab Cx“.
What is Continuous Integration (CI)?
Continuous Integration is the practice of automatically integrating code changes into a software project. It relies on a code repository that supports automated building and testing. Often, CI also involves setting up a build system from scratch, including all dependencies. Please also see our documentation on “Continuous Integration / Gitlab Cx“.
What is Continuous Testing (CT)?
It is the practice of executing automated tests as an integral part of the software development process. It tries to make sure that no functionality is lost and no errors are introduced during development. Please also see our documentation on “Continuous Integration / Gitlab Cx“.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
Why is my code not running on the GPU in TinyGPU?
Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version). Please have look at our documentation about “Working with NVIDIA GPUs“.
Why is my code not using the GPU on Alex?
CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc. Please also have a look at our documentation “Working with NVIDIA GPUs“.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
Why is my code not running on the GPU in TinyGPU?
Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version). Please have look at our documentation about “Working with NVIDIA GPUs“.
Why is my code not using the GPU on Alex?
CUDA is not installed as part of the OS – you have to load a cuda module for your binaries to find libcublas, libcudnn, etc. Please also have a look at our documentation “Working with NVIDIA GPUs“.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.
Why is my code not running on the GPU in TinyGPU?
Do not try to build a GPU-enabled Tensorflow, pytorch, … on the Woody login nodes. That will fail as the Woody login nodes do not have Nvidia software installed (moreover, most TinyGPU nodes run a newer OS version). Please have look at our documentation about “Working with NVIDIA GPUs“.
The software I need is not installed. What can I do now?
On all HPC systems, established tools for software development (compilers, editors, …), libraries, and selected applications are available. For many of these applications, it is necessary to set special environment variables, so that e.g. search paths are correct or license servers can be found. To ease selection of and switching between different versions of software packages, all HPC systems use the modules system (cf. modules.sourceforge.net). It allows to conveniently load the necessary configurations for different programs or different versions of the same program an, if necessary, unload them again later. For information on how to use the modules system, please have a look into the respective section in our documentation about “Software environment“. Even more packages will become visible once one of the 000-all-spack-pkgs modules has been loaded. Most of the software is installed using “Spack“ as enhanced HPC package manager. Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK). You can also bring your own environment in a container using Singularity. However, building Singularity containers on the HPC systems themselves is not supported (as that would require root access). The Infiniband drivers from the host are not mounted into your container. All filesystems will also be available by default in the container. In certain use cases it might be a good idea to avoid bind-mounting your normal $HOME directory with all its “dot directories” into the container by explicitly specifying a different directory, e.g. -H $HOME/my-container-home.

Hardware

Basic HPC Knowledge

Is it true that Arm processors are now competitive in HPC?

“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.

What is a parallel file system?

In a parallel file system (PFS), data is distributed not only across several disks but also multiple servers in order to increase the data access bandwidth. Most PFS’s are connected to the high-speed network of a cluster, and aggregated bandwidths in the TByte/s range are not uncommon. High bandwidth can, however, only be obtained with large files and streaming access.

For information on how to use a parallel file system on our clusters, please read our documentation on “Parallel file systems $FASTTMP“.

What is a vector computer?

A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.

Why should I care about file systems?

Not only may efficient file operations speed up your own code (if file I/O is what you must do); they will also reduce the burden on shared file servers and thus leave more performance headroom for other users of the resource. Hence, it is a matter of thoughtfulness to optimize file accesses even if your performance gain is marginal.

FileSystems/Data Storage

How can I leverage node-local storage on TinyGPU to increase job performance?

Each node has at least 880 GB of local SSD capacity for temporary files under $TMPDIR.

The directory $TMPDIR will be deleted automatically as soon as the user has no jobs running on the node any more.

Data to be kept can be copied to a cluster-wide volume at the end of the job.

Please also read our documentation on “File Systems“.

What is a parallel file system?

In a parallel file system (PFS), data is distributed not only across several disks but also multiple servers in order to increase the data access bandwidth. Most PFS’s are connected to the high-speed network of a cluster, and aggregated bandwidths in the TByte/s range are not uncommon. High bandwidth can, however, only be obtained with large files and streaming access.

For information on how to use a parallel file system on our clusters, please read our documentation on “Parallel file systems $FASTTMP“.

What is file system metadata?

Metadata comprises all the bookkeeping information in a file system: file sizes, permissions, modification and access times, etc. A workload that, e.g., opens and closes files in rapid succession leads to frequent metadata accesses, putting a lot of strain on any file server infrastructure. This is why a small number of users with “toxic” workload can slow down file operations to a crawl for everyone. Note also that especially parallel file systems are ill-suited for metadata-heavy operations.

Where can I store my data?

Your home directory is accessible via $HOME. Each user gets a standard quota of 50 Gigabytes and quota extensions are not possible.

Additional storage is accessible via $HPCVAULT. Here, the default quota for each user is 500 Gigabytes.

The recommended work directory is accessible via $WORK. The standard quota for each user is 500 Gigabytes.

All three directories ($HOME, $HPCVAULT and $WORK) are available throughout our HPC systems.

We recommend you use the aforementioned variables in your jobscripts and not rely on the specific paths as this may change over time, i.e. when directories are relocated to a different NFS server.

Job-specific storage (either located in main memory [RAM disk] or, if available, local HDD / SDD) is accessible via $TMPDIR and always node-local. Size differs between clusters and is only available during job lifetime. Data is flushed after the job finishes!

Some of our clusters have a local parallel filesystem for high performance short-term storage that is accessible via $FASTTMP. These filesystems are specific to the clusters and not available on other clusters. This type of storage is not suitable for programs such as MD simulations that have quite high output rates!

Please also have a look into our documentation on “File Systems“.

Why should I care about file systems?

Not only may efficient file operations speed up your own code (if file I/O is what you must do); they will also reduce the burden on shared file servers and thus leave more performance headroom for other users of the resource. Hence, it is a matter of thoughtfulness to optimize file accesses even if your performance gain is marginal.

Why the need for several file systems?

Different file systems have different features; for example, a central NFS server has massive bytes for the buck but limited data bandwidth, while a parallel file system is much faster but smaller and usually available to one cluster only. A node-local SSD, one the other hand, has the advantage of very low latency but it cannot be accessed from outside a compute node.

For further information, please consult our documentation on “File systems“.

Miscellaneous

Is it true that Arm processors are now competitive in HPC?

“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.

What is a vector computer?

A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.

Test Cluster

Is it true that Arm processors are now competitive in HPC?

“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.

What is a vector computer?

A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.

TinyGPU Cluster

How can I leverage node-local storage on TinyGPU to increase job performance?

Each node has at least 880 GB of local SSD capacity for temporary files under $TMPDIR.

The directory $TMPDIR will be deleted automatically as soon as the user has no jobs running on the node any more.

Data to be kept can be copied to a cluster-wide volume at the end of the job.

Please also read our documentation on “File Systems“.

Is it true that Arm processors are now competitive in HPC?
“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.
What is a parallel file system?
In a parallel file system (PFS), data is distributed not only across several disks but also multiple servers in order to increase the data access bandwidth. Most PFS’s are connected to the high-speed network of a cluster, and aggregated bandwidths in the TByte/s range are not uncommon. High bandwidth can, however, only be obtained with large files and streaming access. For information on how to use a parallel file system on our clusters, please read our documentation on “Parallel file systems $FASTTMP“.
What is a vector computer?
A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.
Why should I care about file systems?
Not only may efficient file operations speed up your own code (if file I/O is what you must do); they will also reduce the burden on shared file servers and thus leave more performance headroom for other users of the resource. Hence, it is a matter of thoughtfulness to optimize file accesses even if your performance gain is marginal.
How can I leverage node-local storage on TinyGPU to increase job performance?
Each node has at least 880 GB of local SSD capacity for temporary files under $TMPDIR. The directory $TMPDIR will be deleted automatically as soon as the user has no jobs running on the node any more. Data to be kept can be copied to a cluster-wide volume at the end of the job. Please also read our documentation on “File Systems“.
What is a parallel file system?
In a parallel file system (PFS), data is distributed not only across several disks but also multiple servers in order to increase the data access bandwidth. Most PFS’s are connected to the high-speed network of a cluster, and aggregated bandwidths in the TByte/s range are not uncommon. High bandwidth can, however, only be obtained with large files and streaming access. For information on how to use a parallel file system on our clusters, please read our documentation on “Parallel file systems $FASTTMP“.
What is file system metadata?
Metadata comprises all the bookkeeping information in a file system: file sizes, permissions, modification and access times, etc. A workload that, e.g., opens and closes files in rapid succession leads to frequent metadata accesses, putting a lot of strain on any file server infrastructure. This is why a small number of users with “toxic” workload can slow down file operations to a crawl for everyone. Note also that especially parallel file systems are ill-suited for metadata-heavy operations.
Where can I store my data?
Your home directory is accessible via $HOME. Each user gets a standard quota of 50 Gigabytes and quota extensions are not possible. Additional storage is accessible via $HPCVAULT. Here, the default quota for each user is 500 Gigabytes. The recommended work directory is accessible via $WORK. The standard quota for each user is 500 Gigabytes. All three directories ($HOME, $HPCVAULT and $WORK) are available throughout our HPC systems. We recommend you use the aforementioned variables in your jobscripts and not rely on the specific paths as this may change over time, i.e. when directories are relocated to a different NFS server. Job-specific storage (either located in main memory [RAM disk] or, if available, local HDD / SDD) is accessible via $TMPDIR and always node-local. Size differs between clusters and is only available during job lifetime. Data is flushed after the job finishes! Some of our clusters have a local parallel filesystem for high performance short-term storage that is accessible via $FASTTMP. These filesystems are specific to the clusters and not available on other clusters. This type of storage is not suitable for programs such as MD simulations that have quite high output rates! Please also have a look into our documentation on “File Systems“.
Why should I care about file systems?
Not only may efficient file operations speed up your own code (if file I/O is what you must do); they will also reduce the burden on shared file servers and thus leave more performance headroom for other users of the resource. Hence, it is a matter of thoughtfulness to optimize file accesses even if your performance gain is marginal.
Why the need for several file systems?
Different file systems have different features; for example, a central NFS server has massive bytes for the buck but limited data bandwidth, while a parallel file system is much faster but smaller and usually available to one cluster only. A node-local SSD, one the other hand, has the advantage of very low latency but it cannot be accessed from outside a compute node. For further information, please consult our documentation on “File systems“.
Is it true that Arm processors are now competitive in HPC?
“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.
What is a vector computer?
A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.
Is it true that Arm processors are now competitive in HPC?
“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.
What is a vector computer?
A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.
How can I leverage node-local storage on TinyGPU to increase job performance?
Each node has at least 880 GB of local SSD capacity for temporary files under $TMPDIR. The directory $TMPDIR will be deleted automatically as soon as the user has no jobs running on the node any more. Data to be kept can be copied to a cluster-wide volume at the end of the job. Please also read our documentation on “File Systems“.

General information

Alex Cluster

How can I access the new clusters Alex and Fritz?

FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there.

If you do not have an HPC account, please follow our instructions on “Getting started with HPC“.

External scientists have to submit a NHR proposal to get access.

Basic HPC Knowledge

Is it true that Arm processors are now competitive in HPC?

“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.

What is a vector computer?

A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.

What is SMT (also known as hyperthreading)?

Simultaneous multi-threading (SMT) allows a CPU core to run more than one software thread at the same time. These “hardware threads” a.k.a. “virtual cores” share almost all resources. The purpose of this feature is to make better use of the execution units within the core. It is rather hard to predict the benefit of SMT for real applications, so the best strategy is to try it using a well-designed, realistic benchmark case.

What is thread or process affinity?

Modern multicore systems have a strong topology, i.e., groups of hardware threads share different resources such as cores, caches, and memory interfaces. Many performance features of parallel programs depend on where their threads and processes are running in the machine. This makes it vital to bind these threads and processes to hardware threads so that variability is reduced and resources are balanced.

Cluster Access

How can I access the cluster frontends?

Almost all HPC systems at NHR@FAU use private IP addresses that can only be accessed directly from within the FAU.There are three options for logging in to the clusters from outside the university:

  1. Use a VPN (Virtual Private Network) connection.
  2. Use IPv6. The cluster frontends have world-visible IPv6 addresses.
  3. Use our “dialog server” cshpc.rrze.fau.de. The dialog server is the only HPC machine with a public IPv4 address. cshpc is a Linux system that is open to all HPC accounts. From this machine, you can log into any NHR@FAU system. A more complete description can be found on our documentation pages.

Whichever option you choose, you need to use SSH. Please consult our extensive SSH documentation pages for details.

 

How can I access the new clusters Alex and Fritz?

FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there.

If you do not have an HPC account, please follow our instructions on “Getting started with HPC“.

External scientists have to submit a NHR proposal to get access.

Contact

How can I contact the HPC team?

An informal and low-threshold way to talk to members of the HPC team is our regular HPC Café. The HPC Café takes place every second Tuesday of the month at 4:00 p.m. in seminar room 2.049 at RRZE, Martensstr.1, 91058 Erlangen. Due to the Covid-19 pandemic, the HPC Café was replaced by an online consultation hour since early 2020. Details are published on the HPC Café website.

Note: Currently we mostly work from home and thus cannot be reached via our office phone numbers. We can arrange virtual appointments by Zoom, MS Teams, or BigBlueButton. You may also call RRZE’s HelpDesk (+49-9131-85-29955) and leave a message for us—but you probably get a faster response by sending us an e-mail (hpc-support@fau.de).

FileSystems/Data Storage

Where can I store my data?

Your home directory is accessible via $HOME. Each user gets a standard quota of 50 Gigabytes and quota extensions are not possible.

Additional storage is accessible via $HPCVAULT. Here, the default quota for each user is 500 Gigabytes.

The recommended work directory is accessible via $WORK. The standard quota for each user is 500 Gigabytes.

All three directories ($HOME, $HPCVAULT and $WORK) are available throughout our HPC systems.

We recommend you use the aforementioned variables in your jobscripts and not rely on the specific paths as this may change over time, i.e. when directories are relocated to a different NFS server.

Job-specific storage (either located in main memory [RAM disk] or, if available, local HDD / SDD) is accessible via $TMPDIR and always node-local. Size differs between clusters and is only available during job lifetime. Data is flushed after the job finishes!

Some of our clusters have a local parallel filesystem for high performance short-term storage that is accessible via $FASTTMP. These filesystems are specific to the clusters and not available on other clusters. This type of storage is not suitable for programs such as MD simulations that have quite high output rates!

Please also have a look into our documentation on “File Systems“.

Fritz Cluster

How can I access the new clusters Alex and Fritz?

FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there.

If you do not have an HPC account, please follow our instructions on “Getting started with HPC“.

External scientists have to submit a NHR proposal to get access.

Login

How can I access the cluster frontends?

Almost all HPC systems at NHR@FAU use private IP addresses that can only be accessed directly from within the FAU.There are three options for logging in to the clusters from outside the university:

  1. Use a VPN (Virtual Private Network) connection.
  2. Use IPv6. The cluster frontends have world-visible IPv6 addresses.
  3. Use our “dialog server” cshpc.rrze.fau.de. The dialog server is the only HPC machine with a public IPv4 address. cshpc is a Linux system that is open to all HPC accounts. From this machine, you can log into any NHR@FAU system. A more complete description can be found on our documentation pages.

Whichever option you choose, you need to use SSH. Please consult our extensive SSH documentation pages for details.

 

How can I change my HPC password?

If you applied for the HPC account using the paper forms: Please log in to www.idm.fau.de with your IdM account and change the HPC password under Services.

Please note that it generally takes a few hours until password changes are known on the HPC systems, the change will not happen at the same time on all clusters.

If you got your HPC account through the new HPC portal, there is no password for the HPC account at all. Authorization at the HPC portal is done using SSO (provided by your home institute) and login to the HPC systems is by SSH keys (which have to be uploaded through the HPC portal). Information on how to upload SSH keys to the HPC portal can be found in our documentation on the HPC portal.

Miscellaneous

I heard that RISC-V is “the new thing”.

RISC-V is just a modern, open instruction set architecture (ISA) that does not have the licensing issues of Arm. However, the underlying processor architecture will mainly determine the performance of code. So far, competitive RISC-V designs are nowhere to be seen in HPC, but this may change in the future.

Is it true that Arm processors are now competitive in HPC?

“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.

What about the Apple M1?

It’s positively impressive, in terms of memory bandwidth as well as the architecture of its memory hierarchy. Current models still lack the peak performance needed to be competitive with x86 server CPUs, however.

What is a vector computer?

A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.

Password

How can I change my HPC password?

If you applied for the HPC account using the paper forms: Please log in to www.idm.fau.de with your IdM account and change the HPC password under Services.

Please note that it generally takes a few hours until password changes are known on the HPC systems, the change will not happen at the same time on all clusters.

If you got your HPC account through the new HPC portal, there is no password for the HPC account at all. Authorization at the HPC portal is done using SSO (provided by your home institute) and login to the HPC systems is by SSH keys (which have to be uploaded through the HPC portal). Information on how to upload SSH keys to the HPC portal can be found in our documentation on the HPC portal.

Test Cluster

Is it true that Arm processors are now competitive in HPC?

“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.

What is a vector computer?

A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.

How can I access the new clusters Alex and Fritz?
FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there. If you do not have an HPC account, please follow our instructions on “Getting started with HPC“. External scientists have to submit a NHR proposal to get access.
Is it true that Arm processors are now competitive in HPC?
“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.
What is a vector computer?
A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.
What is SMT (also known as hyperthreading)?
Simultaneous multi-threading (SMT) allows a CPU core to run more than one software thread at the same time. These “hardware threads” a.k.a. “virtual cores” share almost all resources. The purpose of this feature is to make better use of the execution units within the core. It is rather hard to predict the benefit of SMT for real applications, so the best strategy is to try it using a well-designed, realistic benchmark case.
What is thread or process affinity?
Modern multicore systems have a strong topology, i.e., groups of hardware threads share different resources such as cores, caches, and memory interfaces. Many performance features of parallel programs depend on where their threads and processes are running in the machine. This makes it vital to bind these threads and processes to hardware threads so that variability is reduced and resources are balanced.
How can I access the cluster frontends?
Almost all HPC systems at NHR@FAU use private IP addresses that can only be accessed directly from within the FAU.There are three options for logging in to the clusters from outside the university: Use a VPN (Virtual Private Network) connection. Use IPv6. The cluster frontends have world-visible IPv6 addresses. Use our “dialog server” cshpc.rrze.fau.de. The dialog server is the only HPC machine with a public IPv4 address. cshpc is a Linux system that is open to all HPC accounts. From this machine, you can log into any NHR@FAU system. A more complete description can be found on our documentation pages. Whichever option you choose, you need to use SSH. Please consult our extensive SSH documentation pages for details.  
How can I access the new clusters Alex and Fritz?
FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there. If you do not have an HPC account, please follow our instructions on “Getting started with HPC“. External scientists have to submit a NHR proposal to get access.
How can I contact the HPC team?
An informal and low-threshold way to talk to members of the HPC team is our regular HPC Café. The HPC Café takes place every second Tuesday of the month at 4:00 p.m. in seminar room 2.049 at RRZE, Martensstr.1, 91058 Erlangen. Due to the Covid-19 pandemic, the HPC Café was replaced by an online consultation hour since early 2020. Details are published on the HPC Café website. Note: Currently we mostly work from home and thus cannot be reached via our office phone numbers. We can arrange virtual appointments by Zoom, MS Teams, or BigBlueButton. You may also call RRZE’s HelpDesk (+49-9131-85-29955) and leave a message for us—but you probably get a faster response by sending us an e-mail (hpc-support@fau.de).
Where can I store my data?
Your home directory is accessible via $HOME. Each user gets a standard quota of 50 Gigabytes and quota extensions are not possible. Additional storage is accessible via $HPCVAULT. Here, the default quota for each user is 500 Gigabytes. The recommended work directory is accessible via $WORK. The standard quota for each user is 500 Gigabytes. All three directories ($HOME, $HPCVAULT and $WORK) are available throughout our HPC systems. We recommend you use the aforementioned variables in your jobscripts and not rely on the specific paths as this may change over time, i.e. when directories are relocated to a different NFS server. Job-specific storage (either located in main memory [RAM disk] or, if available, local HDD / SDD) is accessible via $TMPDIR and always node-local. Size differs between clusters and is only available during job lifetime. Data is flushed after the job finishes! Some of our clusters have a local parallel filesystem for high performance short-term storage that is accessible via $FASTTMP. These filesystems are specific to the clusters and not available on other clusters. This type of storage is not suitable for programs such as MD simulations that have quite high output rates! Please also have a look into our documentation on “File Systems“.
How can I access the new clusters Alex and Fritz?
FAU staff and students who already have an HPC account can request access to Alex here: https://hpc.fau.de/tier3-access-to-alex/ and access to Fritz here: https://hpc.fau.de/tier3-access-to-fritz/. Access is restricted to  projects with extended demands, thus, not feasible on TinyGPU or Woody/Meggie, but still below the NHR thresholds. You have to prove that and provide a short description of what you want to do there. If you do not have an HPC account, please follow our instructions on “Getting started with HPC“. External scientists have to submit a NHR proposal to get access.
How can I access the cluster frontends?
Almost all HPC systems at NHR@FAU use private IP addresses that can only be accessed directly from within the FAU.There are three options for logging in to the clusters from outside the university: Use a VPN (Virtual Private Network) connection. Use IPv6. The cluster frontends have world-visible IPv6 addresses. Use our “dialog server” cshpc.rrze.fau.de. The dialog server is the only HPC machine with a public IPv4 address. cshpc is a Linux system that is open to all HPC accounts. From this machine, you can log into any NHR@FAU system. A more complete description can be found on our documentation pages. Whichever option you choose, you need to use SSH. Please consult our extensive SSH documentation pages for details.  
How can I change my HPC password?
If you applied for the HPC account using the paper forms: Please log in to www.idm.fau.de with your IdM account and change the HPC password under Services. Please note that it generally takes a few hours until password changes are known on the HPC systems, the change will not happen at the same time on all clusters. If you got your HPC account through the new HPC portal, there is no password for the HPC account at all. Authorization at the HPC portal is done using SSO (provided by your home institute) and login to the HPC systems is by SSH keys (which have to be uploaded through the HPC portal). Information on how to upload SSH keys to the HPC portal can be found in our documentation on the HPC portal.
I heard that RISC-V is “the new thing”.
RISC-V is just a modern, open instruction set architecture (ISA) that does not have the licensing issues of Arm. However, the underlying processor architecture will mainly determine the performance of code. So far, competitive RISC-V designs are nowhere to be seen in HPC, but this may change in the future.
Is it true that Arm processors are now competitive in HPC?
“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.
What about the Apple M1?
It’s positively impressive, in terms of memory bandwidth as well as the architecture of its memory hierarchy. Current models still lack the peak performance needed to be competitive with x86 server CPUs, however.
What is a vector computer?
A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.
How can I change my HPC password?
If you applied for the HPC account using the paper forms: Please log in to www.idm.fau.de with your IdM account and change the HPC password under Services. Please note that it generally takes a few hours until password changes are known on the HPC systems, the change will not happen at the same time on all clusters. If you got your HPC account through the new HPC portal, there is no password for the HPC account at all. Authorization at the HPC portal is done using SSO (provided by your home institute) and login to the HPC systems is by SSH keys (which have to be uploaded through the HPC portal). Information on how to upload SSH keys to the HPC portal can be found in our documentation on the HPC portal.
Is it true that Arm processors are now competitive in HPC?
“Arm CPU” just means that it uses an instruction set architecture (ISA) licensed from Arm, but the hardware implementation can vary a lot. There is a plethora of Arm-based designs, some from Arm and many from other vendors. Many target the low-power/embedded market, but there are some which have entered the HPC area. Prominent examples are the Fujitsu A64FX and the Marvell ThunderX2. A TX2 system is available in the NHR test and benchmark cluster.
What is a vector computer?
A vector computer has an ISA and CPU architecture that enable efficient operations on array data. This goes under the name of Single Instruction Multiple Data (SIMD). SIMD features have proliferated in commodity CPUs as well, but a true vector CPU has features that make it more efficient, such as large vector lengths (e.g., 256 elements) and a high memory bandwidth (e.g., 1.5 Tbyte/s). Currently, only NEC offers a true vector processor, the SX-Aurora Tsubasa. A node with two Tsubasa cards is available in the NHR test and benchmark cluster.