OS-Upgrade of Alex from AlmaLinux8 to AlmaLinux9 – COMPLETED

Throughout the month of May, Alex will be gradually upgraded from AlmaLinux 8 to AlmaLinux 9. By now, the transition is fully completed.

In addition to the operating system itself, most development and application software is updated to current versions, old versions will usually no longer be available unless there is a good reason for that.

Note on your custom-built software

In most cases, you will also need to rebuild your custom-built software as the OS and the centrally installed software change significantly.

Login nodes only accept SSH public keys registered in the HPC portal

Like all newly setup systems since 2024, now also the Alex login nodes only accept SSH public keys registered in the HPC portal, i.e. the login nodes ignore ~/.ssh/authorized_keys!

Upload needed SSH keys to the HPC portal. They will be known on the HPC systems within 2 hours. As distribution of the public keys is an asynchronous process, different systems learn them at different times and csnhr might know them earlier than the Alex login nodes. Thus, be patient and wait full 2 hours before complaining.

Testing the AlmaLinux9 environment

To already test the new software environment now, proceed as follows:

UNTIL 2026-06-23: On the Alex login nodes run ssh -p 22922 alex which will take you to a virtual login node with the new software stack (but currently still the old Linux kernel — but that should not matter in almost all cases).
UPDATE 2026-05-23: Login to atest01 for the moment to get to a physical login node with the new software stack and the new AlmaLinux9 Linux kernel.
(You can of course also extend your local ssh_config to include the port and directly connect to the virtual login node.)
UPDATE 2026-05-26: The login nodes now use the new software stack and the new AlmaLinux9 Linux kernel. Until end of May, an old login node can be reached as atest02.
UPDATE 2026-06-11: The old login node as atest02 has been removed to make alex2 again a physical login node..
Some nodes have already been migrated. More nodes will be migrated step be step.
You can request these already migrated nodes by adding an additional -C el9 (or --constraint=el9) to your Slurm commands.
If your Slurm commands already contain a constraint, you must connect all constraints with &; giving multiple individual -C options will not work.
~~UPDATE 2026-05-26: the default has been set to -C el9. Until end of may, old nodes still can be requested using an additional -C el8 (or --constraint=el8)~~
UPDATE 2026-06-09: there are no compute nodes with -C el8 anymore.

Feedback

Please send feedback to hpc-support@fau.de.

Planned schedule (as of May 22nd)

Provided no major issues are reported in the first two weeks of May, the physical login nodes will be reinstalled with AlmaLinux9 on May 26 in calendar week 21 (May 18–22). During re-installation of the login nodes, there will be limited connection to Alex for ~~some hours~~ short time.

At the same time, the default for jobs will be changed to the new AlmaLinux9 environment by automatically adding el9 to all jobs without an own OS constraint.

Until the end of May, some compute nodes will still be available via -C el8 with AlmaLinux 8 and the previous software environment. However, login nodes running AlmaLinux 8 will likely no longer be available after this date.

By June 1 at the latest, all compute nodes are scheduled to be migrated to AlmaLinux 9, meaning the option to access the old environment via -C el8 will no longer be available.

Further updates

This post will be updated regularly as needed.

~~Number of already migrated nodes (as of May 13):~~

~~8x A40 (still 38 nodes with AlmaLinux8)~~
~~3x A100/40GB (still 17 nodes with AlmaLinux8)~~
~~3x A100/80GB (still 15 nodes with AlmaLinux8)~~

Intermittent unavailability of alex1

alex1 has been unavailable from time to time on the afternoon of Saturday, May 23.

alex2 has been available all the time.

All logins to alex1 or alex2 will now temporarily end up at alex2.
The AlmaLinux9 login node temporarily can be reached as atest01.

AlmaLinux9 is the default since May 26th @ 07:00!

The login nodes have been upgraded to AlmaLinux9 on May 26th @ 07:00.
All logins to alex1 or alex2 will temporarily end up at alex1.

The default for new Slurm submissions has also been changed to -C el9 on May 26th @ 07:00.

Until end of May, jobs still can request and be run on the remaining AlmaLinux8 nodes by manually specifying -C el8.
An old AlmaLinux8 login node can be reached as atest02 until end of May, too.

el8 jobs still waiting in the queue after end of May will be purged.

FAQs & news

Finding hidden modules
module spider [pattern] can be used to find hidden modules.
VASP with online license check
The new VASP binaries are compiled with an online license check. Follow the instructions given on module load vasp. The license certificate has to be renewed every 3 months.
Capacity quota for /anvme workspaces introduced
/anvme is regularly running out of space. We therefore have to introduce capacity limits on the group and user level in addition to the already existing inode limits.
Longer waiting times
Longer waiting times are expected as Alex is temporarily split into two clusters (still AlmaLinux8 and already AlmaLinux9). Thus, the number of nodes of a specific flavor is reduced, automatically resulting in longer waiting times.
~~As people probably take some time adapting to AlmaLinux9, the early bird has good chances to get empty AlmaLinux9 nodes despite their initially limited number.~~
UPDATE 2026-06-27: in the meantime about 2/3 of the nodes have been update to AlmaLinux9.
Number of already migrated nodes (as of May 23):
18x A40 (still 26 nodes to be migrated to AlmaLinux9)
9x A100/40GB (still 11 nodes to be migrated to AlmaLinux9)
7x A100/80GB (still 11 nodes to be migrated to AlmaLinux9)
2026-05-26 @ 07:00: AlmaLinux9 is the new default!
2026-05-27 @ 12:00: issues with the “workspace commands” have just been fixed. Reconnecting to the login node may be required for the change to become effective for you.
2026-05-27: atest02 did not have access to $HOME and other NFS directories – fixed since 16:30
2026-05-27: SSH logins to the new login node failed for some accounts (they got a password prompt instead of their SSH key being accepted). The reasons are not clear yet. We hope it was a temporary issue only.
Number of already migrated nodes (as of May 27):
35x A40 (still 9 nodes to be migrated to AlmaLinux9)
15x A100/40GB (still 5 nodes to be migrated to AlmaLinux9)
13x A100/80GB (still 5 nodes to be migrated to AlmaLinux9)
2026-05-28: a note has been added on the top of this page that only SSH public keys registered in the HPC portal are accepted by the login nodes
2026-05-30: the openmpi/4.1.8-* modules have been recreated due to issues with the Slurm integration
Number of already migrated nodes (as of May 31):
43x A40 (only 1 node left for migration to AlmaLinux9)
19x A100/40GB (only 1 node left for migration to AlmaLinux9)
all 18 A100/80GB nodes done
2026-06-01: there’s a performance regression in Plumed-2.10.0. We recommend to stick with Plumed-2.8.3 or Plumed-2.9.4. If you depend on the new features of Plumed-2.10.0 contact us for improved execution settings.
2026-06-01: Jobs with -C el8 can no longer be submitted.
2026-06-02: updates to openmpi/5.0.8-* modules and Slurm to make srun compatible with OpenMPI-5’s pmix.
2026-06-11: atest02 (old EL8 login node) has been removed
2026-06-11: still pending jobs with -C el8 have been purged from the queue as announced early. As there are no nodes with EL8 available anymore, the jobs could not have be run anymore.

The transition to AlmaLinux9 is fully completed.

2026-06-15: the UCX library has been updated to improve performance of MPI-based applications. No user-side changes are required to benefit from the updated UCX library.

Last update: 2026-06-15 - 7:12 PM