VASP

Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modeling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.

Availability / Target HPC systems

VASP requires an individual license.

Notes

  • Parallelization and optimal performance:
    • (try to) always use full nodes (ppn=20 for Emmy/Meggie)
    • NCORE=5/10 & PPN=20 results in optimal performance in almost all cases, in general NCORE should be a divisor of PPN
    • OpenMP parallelization is supposed to supersede NCORE
    • use kpar if possible
    • do not use hyperthreads on Emmy!
  • Compilation:
    • use -Davoidalloc
    • use Intel toolchain and MKL
    • in case of very large jobs with high memory requirements add ‘ -heap-arrays 64’ to Fortran flags before compilation (only possible for Intel ifort)
  • Filesystems:
    • Occasionally VASP user reported failing I/O on Meggie’s $FASTTMP (/lxfs), this might be a problem with Lustre and Fortran I/O. Please try to use the fix described here: https://github.com/RRZE-HPC/getcwd-autoretry-preload
    • Since VASP does not do parallel MPI I/O, $WOODYHOME, /home/vault or purchaseable NFS storage (/home/titan /home/saturn) is more appropriate than /lxfs
    • For medium sized jobs, even node local /dev/shm/ might be an option
  • Walltime limit:

Sample job scripts

#!/bin/bash -l
#PBS -lnodes=1:ppn=40,walltime=10:00:00
#PBS -N my-VASP
#PBS -j eo

#enter submit directory
cd $PBS_O_WORKDIR

#define executeable:
VASP=/path-to-your-vasp-installation/vasp

#load modules
module load intel64

#set PPN and pinning
export PPN=20
export I_MPI_PIN=enable

#set stacksize to unlimited
ulimit -s unlimited

#create STOPCAR with LSTOP 1800s before reaching walltimelimit
lstop=1800

#create STOPCAR with LABORT 600s before reaching walltimelimit
labort=600

#timer for STOP = .TRUE.
let SLEEPTIME1=$PBS_WALLTIME-$lstop

#timer for LABORT = .TRUE.
let SLEEPTIME2=$PBS_WALLTIME-$labort

echo "lstop in $SLEEPTIME1 seconds"
echo "labort in $SLEEPTIME2 seconds"

(sleep ${SLEEPTIME1} ; echo "LSTOP = .TRUE." > STOPCAR) &
lstoppid=!$
(sleep ${SLEEPTIME2} ; echo "LABORT = .TRUE." > STOPCAR) &
labortpid=!$

mpirun -ppn $PPN $VASP

pkill -p $lstoppid
pkill -p $labortpid



#! /bin/bash -l
#
#SBATCH --nodes=4
#SBATCH --tasks-per-node=20
#SBATCH --time=24:00:00
#SBATCH --job-name=my-vasp
#SBATCH --mail-user=my.mail
#SBATCH --mail-type=ALL
#SBATCH --export=NONE
unset SLURM_EXPORT_ENV

#enter submit directory
cd $SLURM_SUBMIT_DIR

#load modules
module load intel64

#set PPN and pinning
export PPN=20
export I_MPI_PIN=enable

#define executable:
VASP=/path-to-your-vasp-installation/vasp

#create STOPCAR with LSTOP 1800s before reaching walltimelimit
lstop=1800

#create STOPCAR with LABORT 600s before reaching walltimelimit
labort=600

#automatically detect how much time this batch job requested and adjust the 
# sleep accordingly 
TIMELEFT=$(squeue -j $SLURM_JOBID -o %L -h)
HHMMSS=${TIMELEFT#*-}
[ $HHMMSS != $TIMELEFT ] && DAYS=${TIMELEFT%-*} 
IFS=: read -r HH MM SS <<< $TIMELEFT
[ -z $SS ] && { SS=$MM; MM=$HH; HH=0 ; }
[ -z $SS ] && { SS=$MM; MM=0; }
#timer for STOP = .TRUE.
SLEEPTIME1=$(( ( ( ${DAYS:-0} * 24 + 10#${HH} ) * 60 + 10#${MM} ) * 60 + 10#$SS - $lstop ))
echo "Avilable runtime: ${DAYS:-0}-${HH:-0}:${MM:-0}:${SS}, sleeping for up to $SLEEPTIME1, thus reserving $lstop for clean stopping/saving results"

#timer for LABORT = .TRUE.
SLEEPTIME2=$(( ( ( ${DAYS:-0} * 24 + 10#${HH} ) * 60 + 10#${MM} ) * 60 + 10#$SS - $labort ))
echo "Avilable runtime: ${DAYS:-0}-${HH:-0}:${MM:-0}:${SS}, sleeping for up to $SLEEPTIME2, thus reserving $labort for clean stopping/saving results"

(sleep ${SLEEPTIME1} ; echo "LSTOP = .TRUE." > STOPCAR) &
lstoppid=!$
(sleep ${SLEEPTIME2} ; echo "LABORT = .TRUE." > STOPCAR) &
labortpid=!$

mpirun -ppn $PPN $VASP 

pkill -p $lstoppid
pkill -p $labortpid

Further information

Mentors