Getting started with HPC

This guide will give you a short overview of the most important aspects of running applications on the HPC systems. For more in-depth information, please refer to the linked documentation.

If you have any questions that you like to ask in person, you can come to our HPC Café. On every second Tuesday of the month, we offer a short talk about a specific topic and you can also get a hands-on introduction to our systems. And of course, there is coffee and cake!

Getting an HPC account

Depending on the status, there are different protocols to get an HPC account:

  • NHR users from outside FAU; See the page on NHR application rules for up-to-date information on allocating resources of NHR@FAU.
    Also check the pages on the NHR@FAU HPC-Portal Usage /New digital workflow for HPC accounts.
  • FAU staff and students (except for lectures): use the HPC application form. Details on how to fill out the form are given below. Basic usage of the HPC systems typically is free of charge for FAU researchers for publicly funded research. For compute needs beyond the free basic usage see the page on NHR application rules for preliminary information on allocating resources of NHR@FAU.
  • Lectures of FAU with need for HPC access: there is a simplified protocol to get HPC accounts for all students of your course. Lecturers have to approach HPC support with a list of the IdM accounts of all students of the course and the course name. Late registrations of additional students are not possible. Thus, be sure to collect all IdM accounts before sending the list to RRZE.
  • Block courses with external participants: Lecturers have to approach HPC support at least one week in advance with title and date of the course, and the expected number of participants. Such accounts cannot be valid for more than one week.

The HPC application form for FAU within HPC4FAU

You can get the application form here: HPC application form. Applications always have to be approved by your local chair / institute – we do not serve private persons. If you have any questions regarding the application, please contact your local IT contact person at the chair / institute.

You need to fill out the application form, print it, sign it and let it be stamped with the Chair or Institute seal.

Once it is ready, you can bring it by the RRZE Service Desk or send it via Email, or internal mail.

As a Corona simplification, the chair’s seal can be omitted if the application is forwarded to RRZE by the known local IT contact person.

Fill out the following metadata fields if applying for the first time:

For the System requirement section, a rough estimate of the resources you expect to need is sufficient. Just tell us how many nodes/CPUs your simulations will need (typical job size), the expected runtime per simulation, and the approximate number of simulation runs you are planning (overall requested computation time).  If you are unsure or need help do not hesitate to contact us. Also note that you have to provide an expiration date, usually the duration of your specific project. The duration can also be coupled to your affiliation (employee, PhD student, etc.).

For the lower part of the form, the following information is required:

 

Please note that the more detailed project information is only necessary in case a new RRZE customer ID is requested.

Changing/setting a password for your HPC Account

Once your application is processed, you will be notified by email.
Before a login to our HPC systems is possible a password needs to be set in IDM.
It will always take several hours for a password change in IDM to effect all systems and not at the same time for the different clusters.

Select “Data overview” in the left-hand menu bar.

Scroll down to your HPC account and left-click the gear icon next to “username”.

HPC Clusters

By default, all clusters use Linux operating systems with text-mode only. Basic knowledge of file handling, scripting, editing, etc. under Linux is therefore required.

NHR@FAU operates diverse HPC systems that are tailored to different use cases. Thus, the choice of the appropriate cluster always is essential even if your account will work on most of the systems:

  • single-core or single node (throughput) jobs: Woody
  • multi-node MPI-parallel jobs: Meggie, Fritz
  • GPU jobs: TinyGPU, Alex
    most of the nodes in TinyGPU have been financed by individual groups; therefore, access restrictions/throttling policies may apply.
  • large main memory requirement: TinyFat
    the modern Broadwell-based nodes have been financed by an individual group; therefore, access restrictions/throttling policies may apply.

Also, see the table on the main HPC page. If you’re unsure about which systems to use, feel free to contact the HPC group.

Connecting to HPC systems

To log into the HPC front ends, you have to connect via an SSH (SecureShell) client. Windows users can either use the SSH functionality included in recent Windows 10 versions (via command or Powershell)  or a third-party client like for example PuTTY or MobaXterm. Under Linux and Mac, native OpenSSH functionality is available. From within the university network, you can connect using the following command:

ssh USERNAME@CLUSTERNAME.nhr.fau.de

In this case, USERNAME is your HPC user name and CLUSTERNAME is the name of the cluster you want to log into, e.g. woody. If you want to access TinyFat, or TinyGPU, you have to connect to tinyx.

If you want to access the clusters from outside the university network, you have to connect to the dialogserver first :

ssh USERNAME@cshpc.rrze.fau.de

You can then ssh to the cluster front ends from there. As an alternative, you can also use VPN to access the clusters directly.

For more in-depth information about how to use SSH, refer to SSH – Secure Shell access to HPC systems.

Working with data

Different file systems are accessible from the clusters. Due to their different properties, some might be more suited for the required task than others. The first three classes of directories are available on all HPC systems:

  • $HOME: standard home directory at login, available under /home/hpc
    • small quota (50 GB) and 500k inodes – cannot be increased
    • backup: regular, additional fine-grained snapshots
    • storage of important files only
  • $WORK: general-purpose work directory
    The recommended work directory is $WORK. Its destination may point to different file servers and file systems:
    • available under /home/woody
      • standard quota 500GB
      • no backup
      • can be used for input/output files and for small files
    • available under /home/saturn or /home/titan, both are for shareholders only!
      • group quota according to payment (typically 25+ TB)
      • no backup
      • can be used for input/output files and for small  files
    • available under /home/atuin, for NHR projects only!
      • user/group quota according to NHR grant (typically 10+ TB)
      • no backup
      • can be used for input/output files and for small  file
  • $HPCVAULT: available under /home/vault
    • standard quota 500GB for files and quota on the number of files/directories
    • backup: regular, additional snapshots
    • high-quality, mid- to long-term storage of larger files
  • Parallel file systems $FASTTMP:
    • local to Meggie/Fritz, cannot be accessed from outside these systems
    • no backup, no quota for data volume, but high watermark deletion and limits on the number of files/directories
    • short-term storage, only for high-performance parallel I/O, no ASCII files

For all filesystems, your personal folder is located in your group directory, for example for $HOME at /home/hpc/GROUPNAME/USERNAME. You can also use the environment variables to access the folders directly.

File system quota

Nearly all file systems impose quotas on the data volume and/or the number of files or directories. These quotas may be set per user or per group. There is a distinction between hard quota, which is the absolute upper limit that cannot be exceeded, and soft quota, which can be exceeded temporarily for a certain grace period (7 days). After that time, it turns into a hard quota. You will be notified automatically if you exceed your personal quota on any file system. You can look up your used quota by typing quota -s or shownicerquota.pl on any cluster front end.

Shareholders can look up information on their group quota in text files available as /home/{saturn,titan,janus,atuin}/quota/GROUPNAME.txt.

Data transfer

Under Linux and Mac, scp and rsync are the preferred ways to copy data from and to a remote machine. Under Windows, either the Linux subsystem,  scp via Command/PowerShell, or additional tools like WinSCP can be used. For large files, we recommend using scp or rsyncbecause they are usually much faster than other transfer protocols. rsync also provides more extensive functionality, e.g. skipping files that already exist or resuming file transfers.

Available Software

The standard Linux packages are installed on the cluster front ends. On the compute nodes, usually, much less software is available.

The majority of software is provided by RRZE via the modules system. It contains a variety of compilers, libraries, open and commercial software. A module has to be loaded explicitly to become usable. All module commands affect the current shell only. The available modules may differ between the clusters.

The available modules can be listed via module avail. Modules are loaded via module load <modulename> and unloaded via module unload <modulename>. The currently loaded modules are displayed by module list.  The module commands can usually be used unmodified in any type of Slurm job script.

Some modules cannot be loaded together. In some cases, such a conflict is detected automatically during the load command, in which case an error message is printed and no modifications are made. Modules can depend on other modules so that these are loaded automatically when you load the module. As an example, when you load the OpenMPI module, the corresponding compiler modules will be loaded automatically:

  
$ module load openmpi/4.1.3-gcc11.2.0
$ module list
  Currently Loaded Modulefiles:
  1) gcc/11.2.0 <aL> 2) openmpi/4.1.3-gcc11.2.0

Compiling parallel applications

For compiling your MPI parallel application, you have to explicitly load the necessary modules. For example when using the Intel compiler and Intel MPI,  use module load intel/<version> and module load intelmpi/<version>. When the GNU compiler should be used instead,  use module load gcc/<version> and module load intelmpi/<version>. The selection of the compiler occurs by the wrapper name, e.g. mpicc = gcc, mpiicc = Intel; mpif90 = gfortran; mpiifort = Intel.

In case of OpenMPI, the module is dependent on the compiler mentioned in the module name, e.g. openmpi/4.1.3-gcc11.2.0 depends on the gcc compiler. The corresponding compiler module will be loaded automatically as a dependency with the OpenMPI module.

Prior to running your code, you have to load the same modules as for compiling the program, e.g. inside your Slurm job script.

More details on running parallel applications can be found here.

Running Jobs

The cluster front ends can be used for interactive work like editing input files or compiling your application. The amount of time each of your applications is running is restricted by system limits, e.g., after 1 hour of CPU time, your run will be killed. Front ends are shared among all users, so be considerate of which applications you run. Please do not run applications with large computational or memory requirements on the front ends, since this may interfere with the work of other users. MPI parallel jobs are generally not allowed on the front ends at RRZE.

Batch system

Compute nodes cannot be accessed directly. Compute resources have to be requested by resource manager software, the so-called batch system. All user jobs except short serial test runs must be submitted to the cluster through this batch system. This is done by creating a job script, that contains all the commands you want to run and also the requested resources like the number of compute nodes and runtime. The submitted jobs are routed into a number of queues (depending on the needed resources, e.g. runtime) and sorted according to some priority scheme. A job will run when the required resources become available. The output of the job is written into a file in your submit directory.

The clusters use a software called Slurm as the batch system. Please refer to the linked general documentation for details on the required commands and example scripts. There might be cluster-specific differences in the batch system, which are detailed in the respective cluster documentation.

It is also possible to submit interactive batch jobs that, when started, open a shell on one of the assigned compute nodes and let you run interactive programs there. This is especially useful for testing or applications which cannot be run on the front ends due to higher computational requirements.

Job status

Use the command sinfo on the respective cluster frontend node to get the current status of the cluster nodes (idle, mixed, allocated). This information can be useful to assess the current workload of the cluster, which also influences the queuing time of your job. Keep in mind, however, that there might be some resource or priority limitations that prevent your job from running even if nodes are available.

You can use squeue to check the status of your jobs. If your job is not starting straight away, you can see the reason for the delay in the column NODELIST(REASON). An explanation of these reasons is available here.

When you log into the cluster frontend node, it will also show the Message of the Day (MOTD), where changes in the configuration, maintenance times, and other disruptions in service will be announced.

Good practices

Try to use the appropriate amount of parallelism. Since most workloads are not highly scalable, it is not always better to use more cores for your application. It can be beneficial to run scaling experiments to figure out the “sweet spot” of your application.

Check the results of your job regularly to prevent a waste of computational resources. You can also check if your job actually uses the allocated nodes in an efficient way and if it runs with the expected performance. On all clusters, it is also possible to access performance data of your running and finished jobs, including e.g. memory used, floating point rate, and usage of the (parallel) file system. To review this data, log into https://monitoring.nhr.fau.de/ using your HPC account. For NHR users, log into the HPC Portal and follow the links.

Use the appropriate file system for your calculations.  Doing tiny-size, high-frequency  I/O on a parallel file may overload the metadata servers. When data becomes obsolete, delete it, especially on the parallel file systems ($FASTTMP). No quota limitations apply there, but if a certain level is reached, a high-watermark deletion will be executed, which will affect the old files of all users. Data that should be stored mid- to long-term should be moved to $HPCVAULT.

If you have a problem with your application that you cannot solve yourself, report it to the HPC-support using your FAU mail address. This will immediately open a helpdesk ticket and someone will get back to you. Please provide as much detail as possible so we know where to look, including user name, cluster name, jobID, file system, time of the event, etc..

Additional materials for beginners