VASP
Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modeling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.
Availability / Target HPC systems
VASP requires an individual license.
Notes
- Parallelization and optimal performance:
- (try to) always use full nodes (ppn=20 for Emmy/Meggie)
- NCORE=5/10 & PPN=20 results in optimal performance in almost all cases, in general NCORE should be a divisor of PPN
- OpenMP parallelization is supposed to supersede NCORE
- use kpar if possible
- do not use hyperthreads on Emmy!
- Compilation:
- use -Davoidalloc
- use Intel toolchain and MKL
- in case of very large jobs with high memory requirements add ‘ -heap-arrays 64’ to Fortran flags before compilation (only possible for Intel ifort)
- Filesystems:
- Occasionally VASP user reported failing I/O on Meggie’s $FASTTMP (/lxfs), this might be a problem with Lustre and Fortran I/O. Please try to use the fix described here: https://github.com/RRZE-HPC/getcwd-autoretry-preload
- Since VASP does not do parallel MPI I/O, $WOODYHOME, /home/vault or purchaseable NFS storage (/home/titan /home/saturn) is more appropriate than /lxfs
- For medium sized jobs, even node local /dev/shm/ might be an option
- Walltime limit:
- VASP can only be gracefully stopped by creating the file “STOPCAR” https://www.vasp.at/wiki/index.php/STOPCAR automatic creation is shown in the example scripts below
Sample job scripts
#!/bin/bash -l #PBS -lnodes=1:ppn=40,walltime=10:00:00 #PBS -N my-VASP #PBS -j eo #enter submit directory cd $PBS_O_WORKDIR #define executeable: VASP=/path-to-your-vasp-installation/vasp #load modules module load intel64 #set PPN and pinning export PPN=20 export I_MPI_PIN=enable #set stacksize to unlimited ulimit -s unlimited #create STOPCAR with LSTOP 1800s before reaching walltimelimit lstop=1800 #create STOPCAR with LABORT 600s before reaching walltimelimit labort=600 #timer for STOP = .TRUE. let SLEEPTIME1=$PBS_WALLTIME-$lstop #timer for LABORT = .TRUE. let SLEEPTIME2=$PBS_WALLTIME-$labort echo "lstop in $SLEEPTIME1 seconds" echo "labort in $SLEEPTIME2 seconds" (sleep ${SLEEPTIME1} ; echo "LSTOP = .TRUE." > STOPCAR) & lstoppid=!$ (sleep ${SLEEPTIME2} ; echo "LABORT = .TRUE." > STOPCAR) & labortpid=!$ mpirun -ppn $PPN $VASP pkill -p $lstoppid pkill -p $labortpid
#! /bin/bash -l # #SBATCH --nodes=4 #SBATCH --tasks-per-node=20 #SBATCH --time=24:00:00 #SBATCH --job-name=my-vasp #SBATCH --mail-user=my.mail #SBATCH --mail-type=ALL #SBATCH --export=NONE unset SLURM_EXPORT_ENV #enter submit directory cd $SLURM_SUBMIT_DIR #load modules module load intel64 #set PPN and pinning export PPN=20 export I_MPI_PIN=enable #define executable: VASP=/path-to-your-vasp-installation/vasp #create STOPCAR with LSTOP 1800s before reaching walltimelimit lstop=1800 #create STOPCAR with LABORT 600s before reaching walltimelimit labort=600 #automatically detect how much time this batch job requested and adjust the # sleep accordingly TIMELEFT=$(squeue -j $SLURM_JOBID -o %L -h) HHMMSS=${TIMELEFT#*-} [ $HHMMSS != $TIMELEFT ] && DAYS=${TIMELEFT%-*} IFS=: read -r HH MM SS <<< $TIMELEFT [ -z $SS ] && { SS=$MM; MM=$HH; HH=0 ; } [ -z $SS ] && { SS=$MM; MM=0; } #timer for STOP = .TRUE. SLEEPTIME1=$(( ( ( ${DAYS:-0} * 24 + 10#${HH} ) * 60 + 10#${MM} ) * 60 + 10#$SS - $lstop )) echo "Avilable runtime: ${DAYS:-0}-${HH:-0}:${MM:-0}:${SS}, sleeping for up to $SLEEPTIME1, thus reserving $lstop for clean stopping/saving results" #timer for LABORT = .TRUE. SLEEPTIME2=$(( ( ( ${DAYS:-0} * 24 + 10#${HH} ) * 60 + 10#${MM} ) * 60 + 10#$SS - $labort )) echo "Avilable runtime: ${DAYS:-0}-${HH:-0}:${MM:-0}:${SS}, sleeping for up to $SLEEPTIME2, thus reserving $labort for clean stopping/saving results" (sleep ${SLEEPTIME1} ; echo "LSTOP = .TRUE." > STOPCAR) & lstoppid=!$ (sleep ${SLEEPTIME2} ; echo "LABORT = .TRUE." > STOPCAR) & labortpid=!$ mpirun -ppn $PPN $VASP pkill -p $lstoppid pkill -p $labortpid
Further information
Mentors
- T. Klöffel, RRZE, support-hpc@fau.de
- AG A. Görling (Chair of Theoretical Chemistry)