HACC: Simulating Sky Surveys on State-of-the-Art Supercomputing Architectures (1410.2805v1)

Abstract: Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the 'Dark Universe', dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers that enable these simulations. Here we report on HACC (Hardware/Hybrid Accelerated Cosmology Code), a recently developed and evolving cosmology N-body code framework, designed to run efficiently on diverse computing architectures and to scale to millions of cores and beyond. HACC can run on all current supercomputer architectures and supports a variety of programming models and algorithms. It has been demonstrated at scale on Cell- and GPU-accelerated systems, standard multi-core node clusters, and Blue Gene systems. HACC's design allows for ease of portability, and at the same time, high levels of sustained performance on the fastest supercomputers available. We present a description of the design philosophy of HACC, the underlying algorithms and code structure, and outline implementation details for several specific architectures. We show selected accuracy and performance results from some of the largest high resolution cosmological simulations so far performed, including benchmarks evolving more than 3.6 trillion particles.

HACC: Simulating Sky Surveys on State-of-the-Art Supercomputing Architectures

The paper "HACC: Simulating Sky Surveys on State-of-the-Art Supercomputing Architectures" presents an advanced computational framework designed to conduct large-scale cosmological simulations. The framework, known as HACC (Hardware/Hybrid Accelerated Cosmology Code), is aimed at handling the complex datasets derived from current and future sky surveys while addressing pivotal questions regarding the nature of dark energy and dark matter. HACC is engineered to operate efficiently across diverse supercomputing architectures and scale to millions of cores, allowing it to run extensive cosmological N-body simulations necessary for extracting fundamental astrophysical insights.

Key Features

HACC is distinguished by its ability to utilize a hybrid algorithm for gravitational force computations, combining a grid-based long-range particle-mesh (PM) solver with a tunable short-range force solver that adapts to various systems based on architecture-specific optimizations. This design allows HACC to achieve high performance and portability, emphasizing scalability across different platforms, including conventional clusters, GPU-accelerated systems, and many-core architectures like IBM Blue Gene systems.

The framework supports flexible, in situ data analysis, essential for managing the enormous datasets generated by large-scale simulations. This capability is particularly vital given the constraints on storage and data transfer bandwidth in modern computational environments. The suite of analysis tools includes halo finders, tessellation-based density estimators, and mechanisms for generating clustering statistics, all integrated to facilitate real-time analysis during simulations.

Numerical Results and Performance

The paper presents results from some of the largest high-resolution cosmological simulations ever performed, benchmarking HACC's performance across various architectures. A notable achievement highlighted in the paper includes simulating the universe with a 1.1 trillion-particle run on the Mira supercomputer, demonstrating the capability to handle expansive volumes and resolutions needed for modeling cosmic structure formation. The results showcase the system's ability to maintain weak scaling efficiently, even at the level of hardware platforms reaching millions of cores.

HACC's parallel FFT implementation is noteworthy for its scalable performance across supercomputers, ensuring that communication costs do not surpass computational savings as problem sizes increase. Additionally, memory management strategies are crucial in HACC to prevent memory fragmentation, a significant issue when running simulations close to the hardware memory limit.

Implications and Future Directions

HACC's contributions extend beyond computational performance; it provides a crucial tool for interpreting data from major sky surveys and studying phenomena such as the accelerated expansion of the universe and the properties of dark matter. Its rigorous approach to architecture adaptability makes it a valuable framework for ongoing and future explorations in computational cosmology.

The implications of HACC's design are both practical and theoretical. Practically, the framework supports the optimization of sky survey strategies and data pipelines, offering synthetic catalogs to test observational methods. Theoretically, HACC is positioned to explore novel physics ideas that may elucidate outstanding cosmological questions, including the nature of cosmic acceleration.

Moving forward, HACC’s development will focus on expanding its capacity to include additional physics models, such as hydrodynamics and feedback processes, and enhancing its analysis components to handle more sophisticated galaxy modeling. As computational architectures continue to evolve, HACC will adapt to incorporate emerging technologies and programming models, ensuring sustained performance and relevance in the field of cosmology.