HPC User Report from J. Hornich (Professur für Höchstleistungsrechnen)

We developed a multicore wavefront diamond blocking scheme with multi-dimensional cache block sharing for our hybrid-parallel (MPI+OpenMP) optical solar cell simulation code utilizing the Time Harmonic Inverse Iteration Method (THIIM) to discretize Maxwell’s equations.

Motivation and problem definition

Within this project the goal is to study and develop novel approaches to boost the performance of thin film solar cells. For this, 3D optical simulation of the photovoltaic devices is performed by discretizing Maxwell’s equations. Efficient light management in solar cells is based on suitable nano structures of the different layers and materials with optimized optical properties. The design, development and test of new solar cell prototypes with respect to an optimal light management are a time consuming processes. For this reason, suitable models and simulation techniques are required for the analysis of optical properties within thin-film solar cells. But with the introduction of nano structures simulations become computationally very intensive, since finer grids and smaller timesteps are required. To obtain simulation results in a feasible amount of time, high performance and efficient implementations are necessary.

Methods and codes

In our self developed simulation code we use the Time Harmonic Inverse Iteration Method (THIIM) to discretize Maxwell’s equations. The result is a hybrid-parallel (MPI+OpenMP) finite-difference stencil code. The complex nature of the update algorithm (12 vector fields have to be updated per timestep) in combination with high memory requirements (640 Byte per grid cell) result in very high pressure on the main memory bandwidth. To tackle this problem, we developed a multicore wavefront diamond blocking scheme with multi-dimensional cache block sharing scheme (MWD) for our THIIM code [1,2]. By reusing data already loaded from memory the pressure on the main memory bandwidth is reduced and the performance of the simulation is enhanced.

Results

With the implementation of the MWD approach we were able to achieve a single node speed up of our simulation code of up to 4x on Broadwell CPUs, as they are used in the RRZE Meggie cluster, as well as a speed up of up to 2.5x on Skylake CPUs [1]. Furthermore, an efficient parallelization of the MWD approach was implemented which allows to hide and overlap almost all of the boundary data communication behind the computation of the different MWD diamonds. This results in a very good parallel performance with a 2x parallel speedup on 64 nodes compared to the previous implementation.

Outreach

1. Julian Hornich, Georg Hager, Christoph Pflaum, Efficient optical simulation of nano structures in thin-film solar cells, Proc. SPIE 10694, Computational Optics II, 106940R (28 May 2018); doi:10.1117/12.2312545
2. Malas, T. M., Hornich, J., Hager, G., Ltaief, H., Pflaum, C., and Keyes, D. E., Optimization of an electro-magnetics code with multicore wavefront diamond blocking and multi-dimensional intra-tile parallelization, in [2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) ], 142-151 (May 2016).

The authors gratefully acknowledge funding of the German Federal Ministry for Education and Research (BMBF) in the frame of the SeASiTe project as well as the Erlangen Graduate School in Advanced Optical Technologies (SAOT) by the German Research Foundation (DFG) in the framework of the German excellence initiative. We are thankful for the support by the Bavarian Competence Network for Scientific High Performance Computing (KONWIHR). Computer resources for this project have been provided by the Regional Computing Centre Erlangen (RRZE).

Researcher’s Bio and Affiliation

Julian Hornich graduated in Scientific Computing and Computational Engineering from KTH Stockholm and FAU Erlangen-Nürnberg. After working in the team of Prof. Pflaum at the chair for system simulation at FAU, he is now part of the HPC group of Prof. Wellein.