Woody throughput cluster (Tier3)

FAU’s Woody cluster is a high-performance compute resource for throughput workloads. Woody is a Tier3 resource serving FAU’s basic needs. Therefore, NHR accounts are not enabled by default.

Woody is the next generation throughput cluster and became operational in summer 2022.

• 2 public (and 2 dedicated) front end nodes each with with two Intel Xeon Gold 6342 CPUs (“Icelake”, 2x 24 cores, HT disabled, 2.80 GHz base frequency), 512 GB RAM, and 100 GbE connection to RRZE’s network backbone.
• 64 compute nodes (w12xx/w13xx nodes) with one Intel Xeon E3-1240 v5 CPU (“Skylake”, 4 cores, HT disabled, 3.50 GHz base frequency), 32 GB RAM, 1 TB HDD
  a total of 256 cores from 04/2016 and 01/2017
• 112 compute nodes (w14xx/w15xx nodes) with one Intel Xeon E3-1240 v6 CPU (“Kaby Lake”, 4 cores, HT disabled, 3.70 GHz base frequency), 32 GB RAM, 960 GB SDD
  a total of 448 cores from Q3/2019
• 110 compute nodes (w22xx/w23xx/w24xx/w25xx) with two Intel Xeon Gold 6326 CPUs (“Icelake”, 2x 16 cores, HT disabled,  2.90 GHz base frequency), 256 GB RAM, 1.8 TB SSD
  a total of 3.520 cores from Q2/2022 and Q2/2023.
  40 nodes are financed by ECAP and dedicated to them. 40 nodes additional are financed by and belong to a project from the Physics department.
  

This website shows information regarding the following topics:

• Access, User Environment, File Systems
  • Access to the machine
  • File Systems
  • Batch Processing
• Further Information

Access, User Environment, and File Systems

Access to the machine

Users can connect to woody.nhr.fau.de (keep the “nhr” instead of “rrze” in mind!) and will be randomly routed to one of the two front ends. All systems in the cluster, including the front ends, have private IPv4 addresses in the 10.28.244.0/22 and IPv6 addresses in the 2001:638:a000:4801::/64 range. They can normally only be accessed directly from within the FAU networks. There is one exception: If your internet connection supports IPv6, you can directly ssh to the front  ends (but not to the compute nodes). Otherwise, if you need access from outside of FAU, you usually have to connect for example to the dialog server cshpc.rrze.fau.de first and then ssh to woody.nhr.fau.de from there.

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFBe6F2hyQkJp19HVlZJnc3d/SiSfMRx4LDXAGnhFKXi4sBMWbMxsAFQPT5ZnlTRsPbhMJJeJLGxycQg/pGboyI= woody.nhr.fau.de
		ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIw6ALXTLpIx+JJG3D917bkTlA2J4vXZGXDIysOV+ZiP woody.nhr.fau.de
		ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL6ZBgdcFMU1RjtDpBGjtTovgQZqT/Bd0QWfxzgYvqzUJohiTIAk5YPNq5zygi5yVfO7NphURKSCYn/9XPVa1bBEoVpVAFa5hX1aN1nKynL0Ao2FIlfp/eO9jN0sH4FFdkEoMyIPdYrbsaXULQafREZi2J3dZhhx4+hItP9euAGQCGbdpmZ0f6o+PBleQ6jdILpGhz8Iw/+eMLkqul7okGhiUP9IZqFtJnUW2Smqlgty3qI5eK4TdXDUV17Uvqk4YJFpwyLjXuOCantL+jadtYtEpYtlHW5ZLz33LKkMnnpb4k3ZSJ9QprI8i9/RFRtQlarz7KCnRjbVaI94k51wzV3JIFasdEx2NXDxBC8ZVoi/fWYyt2gsMV+4UB6BSo31xsI3RhsTfpAtUGF0Mt4XV/XuSyMTcX1u6qaSZKCOOVWkp75h+1FROHKlTPtMnLLc71WohDKk/snLpOS736SKSLVoxA+t+21+SiMkFcZa+mX7GySQtHz42ahTS3nMfQsMc= woody-ng woody.nhr.fau.de

256  SHA256:sf45uAzebm6at31zW3/e/T6nyd+sdCSBR9wgeu8ZOuc woody.nhr.fau.de (ECDSA)
		256  SHA256:GUA916wyrc1WFjUT9FcqoJU3aNaHyb40QwJfbtXG5UI woody.nhr.fau.de (ED25519)
		3072 SHA256:jpQpiCp3FAVLOap+f1QNdL52NskusxEdhowyUru5gM0 woody.nhr.fau.de (RSA)

ssh-dss AAAAB3NzaC1kc3MAAACBAO2L8+7bhJm7OvvJMcdGSJ5/EaxvX5RRzE9RrB8fx5H69ObkqC6Baope4rOS9/+2gtnm8Q3gZ5QkostCiKT/Wex0kQQUmKn3fx6bmtExLq8YwqoRXRmNTjBIuyZuZH9w/XFK36MP63p/8h7KZXvkAzSRmNVKWzlsAg5AcTpLSs3ZAAAAFQCD0574+lRlF0WONMSuWeQDRFM4vwAAAIEAz1nRhBHZY+bFMZKMjuRnVzEddOWB/3iWEpJyOuyQWDEWYhAOEjB2hAId5Qsf+bNhscAyeKgJRNwn2KQMA2kX3O2zcfSdpSAGEgtTONX93XKkfh6JseTiFWos9Glyd04jlWzMbwjdpWvwlZjmvPI3ATsv7bcwHji3uA75PznVUikAAACBANjcvCxlW1Rjo92s7KwpismWfcpVqY7n5LxHfKRVqhr7vg/TIhs+rAK1XF/AWxyn8MHt0qlWxnEkbBoKIO5EFTvxCpHUR4TcHCx/Xkmtgeq5jWZ3Ja2bGBC3b47bHHNdDJLU2ttXysWorTXCoSYH82jr7kgP5EV+nPgwDhIMscpk cshpc.rrze.fau.de
		ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNVzp97t3CxlHtUiJ5ULqc/KLLH+Zw85RhmyZqCGXwxBroT+iK1Quo1jmG6kCgjeIMit9xQAHWjS/rxrlI10GIw= cshpc.rrze.fau.de
		ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPSIFF3lv2wTa2IQqmLZs+5Onz1DEug8krSrWM3aCDRU cshpc.rrze.fau.de
		1024 35 135989634870042614980757742097308821255254102542653975453162649702179684202242220882431712465065778248253859082063925854525619976733650686605102826383502107993967196649405937335020370409719760342694143074628619457902426899384188195801203193251135968431827547590638365453993548743041030790174687920459410070371 cshpc.rrze.fau.de
		ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAs0wFVn1PN3DGcUtd/JHsa6s1DFOAu+Djc1ARQklFSYmxdx5GNQMvS2+SZFFa5Rcw+foAP9Ks46hWLo9mOjTV9AwJdOcSu/YWAhh+TUOLMNowpAEKj1i7L1Iz9M1yrUQsXcqDscwepB9TSSO0pSJAyrbuGMY7cK8m6//2mf7WSxc= cshpc.rrze.fau.de

1024 SHA256:A82eA7py46zE/TrSTCRYnJSW7LZXY16oOBxstJF3jxU cshpc.rrze.fau.de (DSA)
		256  SHA256:wFaDywle3yJvygQ4ZAPDsi/iSBTaF6Uoo0i0z727aJU cshpc.rrze.fau.de (ECDSA)
		256  SHA256:is52MRsxMgxHFn58o0ZUh8vCzIuE2gYanmhrxdy0rC4 cshpc.rrze.fau.de (ED25519)
		1024 SHA256:Za1mKhTRFDXUwn7nhPsWc7py9a6OHqS2jin01LJC3ro cshpc.rrze.fau.de (RSA)

Feel free to compile software in the versions and with the options you need yourself. This is perfectly fine, yet support for self-installed software cannot be granted. We only can provide software centrally which is of importance for multiple groups. If you want to use Spack for compiling additional software, you can load our user-spack module to make use of the packages we already build with Spack if the concretization matches instead of starting from scratch. Once user-spack is loaded, the command spack will be available (as alias), you will inherit the pre-sets we defined for certain packages (e.g. Open MPI to work with Slurm), but you’ll install everything into your own directories ($WORK/USER-SPACK).

You can also bring your own environment in a container using Singularity/Apptainer.

File Systems

The following table summarizes the available file systems and their features. It is only an excerpt from the description of the HPC file system.

Use $TMPDIR for the (limited) local storage which gets cleaned up once the job is finished.

Batch processing

As with all production clusters at RRZE, resources are controlled through a batch system. The front ends can be used for compiling and very short serial test runs, but everything else has to go through the batch system to the cluster.

Woody uses SLURM as a batch system. Please see our general batch system description for further details.

• The granularity of batch allocations are individual cores – not complete nodes, i.e.
  • nodes can be shared but your cores and your share of the main memory are exclusive
  • memory bandwidth, network bandwidth, and the local HDD/storage are shared.
• You always get 7.75 GB of main memory per requested core.
• Multi-node jobs are not possible as Woody is a throughput resource.

If you submit jobs, then by default you can get any type of node: Icelake, Skylake, or Kabylake based. They differ in their total number of cores and main memory. However, with every type of node you will automatically get 7.75GB per requested core. Due to differences in processor generation, the speed of the CPUs can be different, which means that job runtimes may vary.

It is also possible to request certain types of nodes from the batch system. This has two mayor use cases besides the obvious “benchmarking”. Some applications can benefit from AVX512, which is only available on the Icelake nodes. Moreover, the Icelake nodes have more main memory and CPU cores in total. You request a certain node type by adding the option --constraint=... to the sbatch/salloc command. In general, the following node types are available:

salloc -n 1 --time=01:00:00

#!/bin/bash -l
		#SBATCH --nodes=1 
		#SBATCH --ntasks-per-node=4
		#SBATCH --time=01:00:00
		#SBATCH --export=NONE
		
		unset SLURM_EXPORT_ENV
		module load XXX 
		
		srun ./mpi_application

#!/bin/bash -l
		#SBATCH --nodes=1
		#SBATCH --ntasks-per-node=1
		#SBATCH --cpus-per-task=4
		#SBATCH --time=01:00:00
		#SBATCH --export=NONE
		
		unset SLURM_EXPORT_ENV 
		module load XXX 
		
		# set number of threads to requested cpus-per-task
		export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK 
		./openmpi_application

Further Information

The performance of the new Icelake nodes is only better in the High Performance Linkpack (HPL) if AVX512 instructions are used!

Saturating the node in throughput mode, the memory bandwidth per (four) core(s) is only slightly higher.