HPX with Spack and Singularity Containers: Evaluating Overheads for HPX/Kokkos using an astrophysics application (2405.00016v2)

Abstract: Cloud computing for high performance computing resources is an emerging topic. This service is of interest to researchers who care about reproducible computing, for software packages with complex installations, and for companies or researchers who need the compute resources only occasionally or do not want to run and maintain a supercomputer on their own. The connection between HPC and containers is exemplified by the fact that Microsoft Azure's Eagle cloud service machine is number three on the November 23 Top 500 list. For cloud services, the HPC application and dependencies are installed in containers, e.g. Docker, Singularity, or something else, and these containers are executed on the physical hardware. Although containerization leverages the existing Linux kernel and should not impose overheads on the computation, there is the possibility that machine-specific optimizations might be lost, particularly machine-specific installs of commonly used packages. In this paper, we will use an astrophysics application using HPX-Kokkos and measure overheads on homogeneous resources, e.g. Supercomputer Fugaku, using CPUs only and on heterogenous resources, e.g. LSU's hybrid CPU and GPU system. We will report on challenges in compiling, running, and using the containers as well as performance performance differences.

• The paper quantifies the performance overhead of containerizing HPX/Kokkos with Spack and Singularity using an astrophysics simulation.
• It details a methodology leveraging dependency management and container conversion to deploy Octo-Tiger on diverse HPC platforms.
• The study highlights reproducibility benefits alongside performance trade-offs on heterogeneous architectures, calling for further optimization.

Overview of HPX with Spack and Singularity Containers: Evaluating Overheads for HPX/Kokkos using an Astrophysics Application

The paper presents a comprehensive evaluation of overheads introduced by containerization, specifically using Spack and Singularity, when deploying an astrophysics application built with the HPX and Kokkos frameworks. The investigation is situated within the context of high-performance computing (HPC), an area that increasingly leans toward container technology, offering modularity and reproducibility, albeit with potential trade-offs in performance.

The paper underscores the intersection of HPC and cloud computing facilitated by containerization, acknowledging the benefits in simplified deployments and reproducibility. However, the core emphasis lies in understanding the performance implications when employing containers on heterogeneous and homogeneous computing resources, utilizing Octo-Tiger, an astrophysics simulation application.

Methodological Approach

The methodological approach involves leveraging Spack for dependency management and building, coupled with Singularity containers for execution. The paper discusses generating Docker images, which serve as the basis for Singularity containers—this circumvents the root access requirements associated with Docker, making it feasible for supercomputing environments.

Key challenges in the workflow include managing the compilation environment peculiarities on distinct architectures like A64FX, as illustrated in tests on Supercomputer Fugaku and LSU's DeepBayou cluster. The paper details configuring for Spack and image conversion from Docker to Singularity, providing insights into the intricacies of handling different hardware and software platform requirements.

Performance Evaluation

The performance evaluation involves running Octo-Tiger in various configurations: standalone and within Singularity containers, across singular and multiple nodes, with both CPU and GPU resources. Results on Supercomputer Fugaku revealed non-negligible overhead when using containers, marking approximately 50 seconds additional computation time in singularity runs compared to regular executions.

Contrastingly, performance differences on DeepBayou were minimal in CPU-only scenarios, while combined CPU and GPU executions within containers presented anomalous behavior, particularly in distributed environments, pointing to potential debugging points concerning CUDA resource handling.

Implications and Future Work

The paper concludes that while containers provide substantial benefits in terms of reproducibility and ease of deployment across varied platforms, the associated overheads and challenges need to be carefully weighed, especially in environments requiring optimized HPC performance. Compiling within containers can become complex due to the reliance on vendor-specific tooling and cross-compilation requirements, which can hinder straightforward deployment across architectural varieties.

Future directions suggest expanding evaluations to larger GPU-rich environments like Perlmutter, addressing MPI integration within containers for distributed runs, and further investigating the observed discrepancies in GPU-accelerated computations. Such expansions would deepen the understanding of containers' performance implications in state-of-the-art HPC applications and guide the development of optimization strategies for containerized workflows.

In summary, this paper contributes to the ongoing discourse surrounding the practical integration of containerization in HPC, specifically underlining the need for empirical assessments of performance impacts relative to deployment and reproducibility benefits.