Towards the Democratization and Standardization of Dynamic Resources with MPI Spawning

Abstract: This paper presents an efficient tool for managing dynamic resources in production high-performance computing (HPC) settings, focusing on flexibility, adaptability, and user-friendliness. We introduce a unified dynamic resource management application programming interface (API) that supports a wide range of HPC applications, allowing seamless integration without direct interaction with Dynamic Management of Resources (DMR). The DMR framework, evolved from the DMRlib structure, now supports various dynamic resource managers and includes the Proteo reconfiguration engine to enhance malleability strategies. This integration addresses previous limitations by allowing diverse reconfiguration methods without respawning all processes or lacking RMS support. The paper also showcases the solution's performance and coding productivity with the MPDATA (Multidimensional Positive Definite Advection Transport Algorithm) application. Key contributions include an enhanced modular DMR framework supporting different reconfiguration managers, upgraded DMRlib with the Proteo reconfiguration engine, offering extensive reconfiguration strategies, and a malleable version of the MPDATA solver.

• The paper introduces the DMR framework that standardizes dynamic resource management in HPC using MPI spawning.
• It leverages a modular DMRlib and Proteo reconfiguration engine to enable adaptive resource allocation and job malleability, reducing waiting times by approximately 25%.
• Experimental evaluations on a cluster demonstrate improved node utilization and faster workload completion compared to static resource management.

Democratizing Dynamic Resource Management in HPC with MPI Spawning

Motivation and Context

Dynamic resource management is increasingly imperative in high-performance computing (HPC), especially as systems approach exascale capabilities and workloads demand adaptive resource allocation to maximize throughput and efficiency. Conventional resource manager systems (RMS) such as Slurm provide static allocations, constraining job malleability and inhibiting optimal utilization. The Message Passing Interface (MPI), while a de facto standard for distributed memory programming, has introduced scalable tools for dynamic process management, yet lacks a universally adopted framework for job reconfiguration compatible across RMS and MPI implementations.

This paper introduces the Dynamic Management of Resources (DMR) framework, a unified, modular architecture for dynamic resource management that leverages MPI spawning, an enhanced DMRlib, and the Proteo reconfiguration engine. The aim is democratization and standardization of malleable, dynamically managed jobs through accessible APIs and integration with production RMS like Slurm.

Technological Advancements

DMR Framework and DMRlib Evolution

DMRlib originally served as a high-level API to facilitate transparent adaptation of HPC codes via a communication layer bridging parallel distributed runtimes (PDR) and RMS. The present work transforms DMRlib into a standalone, modular dynamic resource communication layer, decoupling user-facing APIs from underlying reconfiguration engines. This design enables seamless interoperability with multiple dynamic resource managers and provides user-agnostic APIs for dynamic job specification and management.

Figure 1: DMR software stack, illustrating separation of user API, DMR middleware, and integration of dynamic resource managers and reconfiguration engines.

The revised DMRlib supports minimalist MPI-like syntax and abstracts process management, resource querying, and data redistribution, now devoid of reliance on OmpSs/Nanos++ runtimes. Integration with RMS like Slurm is achieved via extensible plugins, enabling in-situ resource reallocation.

Proteo Reconfiguration Engine Integration

Proteo is incorporated into DMRlib to expand malleability strategies, offering extensive spawning options and sophisticated data redistribution. The Proteo framework consists of the Synthetic Application Module (SAM) and the Malleability Module (MaM), which collectively support emulation and integration of malleable parallel applications. Process management strategies in Proteo facilitate both baseline (full respawn) and merge (partial respawn) methods, with support for threaded and asynchronous redistribution patterns.

Proteo's API simplifies the development of malleable codes, automatically handling 1D data redistribution with minimal user input and synchronizing with RMS directives for process expansion/shrinkage. The result is an effortless conversion of scientific codes into malleable jobs, exemplified by the implementation in the MPDATA solver.

Experimental Evaluation

Workload Configuration and Malleability Policy

The framework was evaluated on a cluster of eight dual-socket Intel Xeon nodes with MPICH 4.2.1 and a customized Slurm for process malleability support. The experimental workload consisted of 1,000 MPDATA jobs (domain $1024 \times 128 \times 32$), each configured for periodic malleability within a range of resources (1–4 nodes), with jobs submitted at one-second intervals to simulate heavy load.

A dynamic reconfiguration policy was implemented in Slurm to maximize global productivity, expanding jobs dynamically when resources are available and shrinking jobs to accommodate pending workloads. This ensures optimal utilization and minimizes job waiting times.

Numerical Results

Dynamic resource management reduced workload completion times compared to static allocation. Both full respawn (Baseline) and merge techniques in Proteo consistently achieved high utilization rates and accelerated job throughput.

Figure 2: Mean workload completion times for static and dynamic resource management, highlighting the speedup achieved via dynamic allocations.

Mean execution, waiting, and completion times per job are as follows:

Management Type	Execution Mean Time (s)	Waiting Mean Time (s)	Completion Mean Time (s)
Static	14.67	3356.32	3370.99
DynRes Baseline	23.52	2516.51	2540.03
DynRes Merge	23.51	2541.13	2564.64

The increased execution time in dynamic jobs is offset by substantially reduced waiting times, yielding an overall ~25% improvement in workload completion.

Node utilization rates in dynamic experiments exceed those in the static configuration, even in ideal scenarios for static workloads (e.g., aligned job sizes with cluster capacity), demonstrating superior fragmentation handling and adaptability.

Implications and Future Directions

The presented DMR framework with Proteo integration addresses a lacuna in the HPC landscape: the absence of generic, standard-compliant, and RMS-compatible dynamic resource management. The modularity and universal API abstraction facilitate adoption in scientific applications without significant coding overhead.

Theoretical implications include the possibility of integrating performance-aware and energy-aware reconfiguration policies (via TALP and EAR). Practically, the system enables efficient processing of complex and iterative workloads, reducing queue delays and enhancing cluster utilization.

Future research directions encompass asynchronous Proteo reconfiguration, expanded workloads, diverse job types, large-scale infrastructure (such as Marenostrum 5), and in-depth analyses of resource utilization, energy consumption, and workload times. The framework's extensibility positions it as a candidate for cross-RMS and MPI-session-based malleability solutions, potentially influencing standards development.

Conclusion

The integration of DMR, DMRlib, and Proteo with Slurm and MPI establishes a flexible, extensible architecture for dynamic, malleable job management in HPC clusters. The abstraction of programming interfaces and support for various reconfiguration engines democratize access to adaptive workloads, allowing scientific applications to execute with optimal resource utilization and reduced completion times. Continued empirical evaluation and expansion will yield richer insights and foster further standardization efforts in dynamic resource management for distributed, parallel computation (2604.27430).