PKDGRAV3: Beyond Trillion Particle Cosmological Simulations for the Next Era of Galaxy Surveys (1609.08621v1)

Abstract: We report on the successful completion of a 2 trillion particle cosmological simulation to z=0 run on the Piz Daint supercomputer (CSCS, Switzerland), using 4000+ GPU nodes for a little less than 80h of wall-clock time or 350,000 node hours. Using multiple benchmarks and performance measurements on the US Oak Ridge National Laboratory Titan supercomputer, we demonstrate that our code PKDGRAV3, delivers, to our knowledge, the fastest time-to-solution for large-scale cosmological N-body simulations. This was made possible by using the Fast Multipole Method in conjunction with individual and adaptive particle time steps, both deployed efficiently (and for the first time) on supercomputers with GPU-accelerated nodes. The very low memory footprint of PKDGRAV3 allowed us to run the first ever benchmark with 8 trillion particles on Titan, and to achieve perfect scaling up to 18000 nodes and a peak performance of 10 Pflops.

Overview of Cosmological Simulations with PKDGRAV3

The paper "PKDGRAV3: Beyond Trillion Particle Cosmological Simulations for the Next Era of Galaxy Surveys" by Potter, Stadel, and Teyssier presents a detailed analysis and results from an impressive cosmological simulation executed using the PKDGRAV3 code. The paper marks a significant advancement in cosmological modeling, particularly with the successful execution of a 2 trillion particle simulation to redshift z=0. This was accomplished on the Piz Daint supercomputer, employing over 4000 GPU nodes for approximately 80 hours of wall-clock time, equivalent to 350,000 node hours. This level of computational capability is crucial for upcoming galaxy surveys aiming at enhanced precision in cosmological parameter estimation.

Key Results and Methodological Innovations

The paper emphasizes PKDGRAV3's position as one of the fastest codes for cosmological N-body simulations, attributed largely to the integration of the Fast Multipole Method (FMM) and adaptive time-stepping on GPU-accelerated nodes. A significant highlight is the minimal memory footprint of the code, which enabled benchmarks involving 8 trillion particles on the Titan supercomputer, achieving nearly perfect scaling up to 18,000 nodes and recording a maximum performance of 10 Pflops.

The authors meticulously detail four computational challenges addressed by PKDGRAV3 to achieve greater than 1% accuracy in the matter density power spectrum, spanning linear to non-linear scales. These challenges included precision in gravity calculations, accuracy in time steps, reducing statistical errors, and ensuring high mass resolution, which translated to a particle count exceeding 2 trillion.

Implications and Future Directions

The implications of PKDGRAV3 stretch beyond its computational achievements. The code serves as an essential tool for preparing and analyzing data from large galaxy surveys such as LSST, Euclid, and WFIRST. These surveys aim to constrain cosmological models with unprecedented accuracy, potentially illuminating the dark matter and dark energy components.

Moreover, the paper outlines future potentials in cosmology simulations owing to PKDGRAV3's capabilities. As cosmological experiments become more sophisticated, the need for rapid, high-resolution simulations grows. The authors argue that memory constraints, rather than processing speed, remain the predominant limitation. Hence, they foresee continuous improvements in time-to-solution as computational hardware evolves. This trajectory suggests that cosmological simulations might soon become integral to real-time data analysis pipelines.

Conclusion

Potter et al.'s work with PKDGRAV3 sets a benchmark in the field of cosmological simulations, embodying a confluence of computational prowess and astrophysical relevance. The successful implementation of extreme N-body simulations not only establishes PKDGRAV3's utility but also underscores the broader potential for shaping cosmological understanding through high-performance computing. It remains crucial for the astrophysics community to keep pace with such technological advancements to unravel the mysteries of the cosmos with greater clarity and precision.