Spack package manager
Spack package manager
Some users require special software for their HPC runs which is not provided by the computing center. In order to provide this special software, the HPC systems at NHR@FAU provide the Spack package manager in the module environment under the name user-spack
.
The benefits when using Spack:
- Easy to maintain separate versions of the same package
- Separate installations when using different versioned or built dependencies
The benefits also have their downside. You partly have to very specific what to install with which dependencies.
The main Spack documentation (for the latest version) can be found here.
Basics
Load the user-spack
module:
$ module avail user-spack
------------------- /apps/modules/data/via-spack -------------------
user-spack/0.18.1
$ module load user-spack
All software will be installed below $WORK/USER-SPACK
.
List available compilers
The NHR@FAU team provides some predefined compilers which are mostly available as distinct modules or are part of the base system installation. They are only loosely related to the packages used inside Spack, so the list of compilers does not change when you load a module before.
$ spack compilers
==> Available compilers
-- dpcpp almalinux8-x86_64 --------------------------------------
dpcpp@2022.1.0 dpcpp@2021.4.0
-- gcc almalinux8-x86_64 ----------------------------------------
gcc@12.1.0 gcc@11.2.0 gcc@8.5.0
-- oneapi almalinux8-x86_64 -------------------------------------
oneapi@2022.1.0 oneapi@2021.4.0
List available packages
$ spack list netcdf
==> 7 packages.
netcdf-c netcdf-cxx netcdf-cxx4 netcdf-fortran parallel-netcdf py-h5netcdf py-netcdf4
List installed packages
$ spack find openmpi
==> 4 installed packages
-- linux-almalinux8-zen / gcc@8.4.1 -----------------------------
openmpi@3.1.6
-- linux-almalinux8-zen / intel@2021.4.0 ------------------------
openmpi@3.1.6
-- linux-almalinux8-zen / oneapi@2021.4.0 -----------------------
openmpi@3.1.6
-- linux-centos7-haswell / gcc@4.8.5 ----------------------------
openmpi@4.1.2
Get information about a package
$ spack info numactl
AutotoolsPackage: numactl
Description:
NUMA support for Linux
Homepage: https://github.com/numactl/numactl
Preferred version:
2.0.14 https://github.com/numactl/numactl/archive/v2.0.14.tar.gz
Safe versions:
2.0.14 https://github.com/numactl/numactl/archive/v2.0.14.tar.gz
2.0.12 https://github.com/numactl/numactl/archive/v2.0.12.tar.gz
2.0.11 https://github.com/numactl/numactl/archive/v2.0.11.tar.gz
Deprecated versions:
None
Variants:
None
Build Dependencies:
autoconf automake gnuconfig libtool m4
Link Dependencies:
None
Run Dependencies:
None
Get available versions for a package
$ spack versions kokkos
==> Safe versions (already checksummed):
develop master 3.6.00 3.5.00 3.4.01 3.4.00 3.3.01 3.3.00 3.2.01 3.2.00 3.1.01 3.1.00 3.0.00
==> Remote versions (not yet checksummed):
3.6.01
Install a package
The most complicated part is the specification of the package for installation. Spack uses a special syntax.
See Simple package installation, Custom versions & configurations and Customize dependencies in the Spack docs for more information.
Check what will be installed
$ spack spec libvdwxc
Input spec
--------------------------------
libvdwxc
Concretized
--------------------------------
libvdwxc@0.4.0%gcc@8.5.0+mpi~pfft arch=linux-almalinux8-x86_64
^fftw@3.3.10%gcc@8.5.0+mpi~openmp~pfft_patches precision=double,float arch=linux-almalinux8-x86_64
^openmpi@3.1.6%gcc@8.5.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker+pmi+romio+rsh~singularity+static+vt+wrapper-rpath fabrics=psm2 schedulers=slurm arch=linux-almalinux8-x86_64
^hwloc@1.11.13%gcc@8.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared patches=d1d94a4 arch=linux-almalinux8-x86_64
^libpciaccess@0.16%gcc@8.5.0 arch=linux-almalinux8-x86_64
^libtool@2.4.7%gcc@8.5.0 arch=linux-almalinux8-x86_64
^m4@1.4.19%gcc@8.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-almalinux8-x86_64
^libsigsegv@2.13%gcc@8.5.0 arch=linux-almalinux8-x86_64
^pkgconf@1.8.0%gcc@8.5.0 arch=linux-almalinux8-x86_64
^util-macros@1.19.3%gcc@8.5.0 arch=linux-almalinux8-x86_64
^libxml2@2.9.13%gcc@8.5.0~python arch=linux-almalinux8-x86_64
^libiconv@1.16%gcc@8.5.0 libs=shared,static arch=linux-almalinux8-x86_64
^xz@5.2.5%gcc@8.5.0~pic libs=shared,static arch=linux-almalinux8-x86_64
^zlib@1.2.12%gcc@8.5.0+optimize+pic+shared patches=0d38234 arch=linux-almalinux8-x86_64
^ncurses@6.2%gcc@8.5.0~symlinks+termlib abi=none arch=linux-almalinux8-x86_64
^numactl@2.0.14%gcc@8.5.0 patches=4e1d78c,62fc8a8,ff37630 arch=linux-almalinux8-x86_64
^autoconf@2.69%gcc@8.5.0 patches=35c4492,7793209,a49dd5b arch=linux-almalinux8-x86_64
^perl@5.34.1%gcc@8.5.0+cpanm+shared+threads arch=linux-almalinux8-x86_64
^berkeley-db@18.1.40%gcc@8.5.0+cxx~docs+stl patches=b231fcc arch=linux-almalinux8-x86_64
^bzip2@1.0.8%gcc@8.5.0~debug~pic+shared arch=linux-almalinux8-x86_64
^diffutils@3.8%gcc@8.5.0 arch=linux-almalinux8-x86_64
^gdbm@1.19%gcc@8.5.0 arch=linux-almalinux8-x86_64
^readline@8.1%gcc@8.5.0 arch=linux-almalinux8-x86_64
^automake@1.16.5%gcc@8.5.0 arch=linux-almalinux8-x86_64
^opa-psm2@11.2.206%gcc@8.5.0+avx2 arch=linux-almalinux8-x86_64
^openssh@8.0p1%gcc@8.5.0 arch=linux-almalinux8-x86_64
^slurm@21.08.3-1%gcc@8.5.0~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc arch=linux-almalinux8-x86_64
There you see the whole package tree including the used compilers, the feature flags and the architecture used.
Example from the Spack docs
# Install hdf5 and link it with specific versions of openmpi and hwloc
$ spack install hdf5@1.10.1 %gcc@4.7.3 +debug ^openmpi+cuda fabrics=auto ^hwloc+gl
This installs the hdf5
package in version 1.10.1
with the gcc
compiler in version 4.7.3
and enables the debug
flag.
As dependencies, it uses openmpi
with cuda
feature enabled and the fabrics
variable set to auto
and hwloc
with the gl
feature enabled.
Downloading sources & packages
On some compute systems, it might be required to use a proxy server for accessing the internet.
$ export http_proxy=http://proxy:80
$ export https_proxy=http://proxy:80
It takes some time
Spack itself is not the fastest piece of software and all operations at the computing centers are on remote file systems. Those file systems commonly don’t like the traffic which happens when building software.
Moreover, Spack installs all dependencies down to the lowest level for each compiler&deps combination. So you might see builds of base packages like zlib
, perl
, ncurses
, etc. The NHR@FAU team tries to provide a basic set of common packages to avoid to many builds of the same dependency over and over again.
Some builds fail
Due to the high flexibility of packages and versions coupled together by Spack and the various compilers, it might be that a build fails. It often helps to add a dependency with ^
compiled with gcc
. For the netcdf-fortran
package in the next paragraph, the ^numactl@2.0.14%gcc
dependency was required otherwise some dependency of OpenMPI failed with missing <numa.h>
.
Using an installed package
All installed packages (and installed dependencies) should be visible as modules after installation (see module avail
). If not, try unloading the user-spack
module and load it again. If it is still not visible, try logging in again in a new session.
$ module load user-spack
$ spack install netcdf-fortran%oneapi@2022.1.0 \
^netcdf-c%oneapi@2022.1.0 \
^openmpi@4.1.3%oneapi@2022.1.0 \
^cmake@3.23.1 \
^numactl@2.0.14%gcc
$ module avail netcdf-fortran
------------------- /home/saturn/unrz/unrzXXX/USER-SPACK/share/spack/modules/linux-almalinux8-x86_64 -------------------
netcdf-fortran/4.5.4-oneapi2022.1.0-openmpi-3bmi7ym
You can use the software versions installed through Spack like normal modules. The module files are managed by Spack but it has its own mechanism to setup the proper environment, it does not load the modules corresponding to the environment.
$ module load netcdf-fortran/4.5.4-oneapi2022.1.0-openmpi-3bmi7ym
$ nf-config
The module avail
command shows all installed packages, also the base packages like zlib
, perl
, ncurses
, etc. You can load them separately, but you do not need to load them explicitly to use netcdf-fortran
(in this case).
module avail output after installing a package through Spack
$ module avail
------------------- /home/saturn/unrz/unrzXXX/USER-SPACK/share/spack/modules/linux-almalinux8-x86_64 -------------------
autoconf/2.69-gcc8.4.1-g4skt4y libbsd/0.11.3-gcc8.4.1-flcvhjw numactl/2.0.14-gcc12.1.0-elphffx
autoconf/2.69-gcc12.1.0-zp6lni4 libbsd/0.11.5-oneapi2022.1.0-csnexv2 numactl/2.0.14-oneapi2022.1.0-2rlabzl
autoconf/2.69-oneapi2022.1.0-ogngd4o libevent/2.1.12-oneapi2022.1.0-urjh4xz opa-psm2/11.2.206-gcc12.1.0-7aw5fpm
automake/1.16.3-gcc8.4.1-2mgywnr libiconv/1.16-gcc8.4.1-wk2vhbw opari2/2.0.6-gcc8.4.1-4vdn6tg
automake/1.16.5-oneapi2022.1.0-v63ckpi libiconv/1.16-oneapi2022.1.0-caghp63 opari2/2.0.6-intel2021.4.0-34c744c
berkeley-db/18.1.40-gcc8.4.1-egliouh libidn2/2.3.0-gcc8.4.1-rggatlp openmpi/4.1.3-oneapi2022.1.0-44wu2xf
berkeley-db/18.1.40-oneapi2022.1.0-emyiyb2 libidn2/2.3.0-oneapi2022.1.0-tjr7ymg otf2/2.3-gcc8.4.1-rjoiusx
bzip2/1.0.8-gcc8.4.1-ahvvlrw libmd/1.0.3-gcc8.4.1-kqgrzgj otf2/2.3-intel2021.4.0-inao6hn
bzip2/1.0.8-oneapi2022.1.0-md3kfxz libmd/1.0.4-oneapi2022.1.0-zeltuf3 papi/6.0.0.1-gcc8.4.1-pqalrnb
cmake/3.23.1-oneapi2022.1.0-afaxwof libpciaccess/0.16-oneapi2022.1.0-iyi6tbt papi/6.0.0.1-intel2021.4.0-zvpbzqb
[...]
To remove them from the environment, just unload the module
$ module unload netcdf-fortran
What about spack load
, spack unload
and spack env
These subcommands of Spack are part of a builtin module system. It is currently not usable at the HPC systems of NHR@FAU.
Dr. Thomas Zeiser
Zentrum für Nationales Hochleistungsrechnen Erlangen (NHR@FAU)
c/o Regionales Rechenzentrum Erlangen (RRZE)
- Phone number: +49913185-28737
- Email: thomas.zeiser@fau.de
Thomas Gruber
Erlangen National High Performance Computing Center
Software & Tools Division
- Phone number: +49 9131 85-28911
- Email: thomas.gruber@fau.de