A Test Taxonomy and Continuous Integration Ecosystem for Dynamic Resource Management in HPC

Abstract: High-performance computing (HPC) systems are increasingly exploring dynamic resource management and malleable MPI applications to better adapt to heterogeneous architectures, fluctuating workloads, and energy constraints. However, the correctness of the libraries that support these techniques is often evaluated through ad hoc experiments that can be difficult to reproduce and maintain. This article introduces methodology for testing dynamic resource management frameworks that combines a taxonomy of tests for MPI malleable libraries with an HPC-oriented continuous integration (CI) ecosystem. The taxonomy structures functional and non-functional tests at both component-integration and system levels. The CI ecosystem instantiates this taxonomy in a containerized virtual cluster enabling automated validation. The approach is instantiated and evaluated using the Dynamic Management of Resources (DMR) framework as a representative case study. Results show that the proposed methodology improves early fault detection, simplifies maintenance under evolving dependencies, and transfers to other malleability solutions that expose analogous primitives for initialization, readiness checking, and reconfiguration.

• The paper introduces a comprehensive test taxonomy and CI ecosystem for validating dynamic resource management protocols in HPC, using DMR as a case study.
• It details both component integration and system-level tests that verify initialization, readiness, and reconfiguration operations in malleable MPI applications.
• The CI pipeline addresses compatibility issues with evolving HPC software stacks, ensuring reproducible and automated validation workflows.

Test Taxonomy and Continuous Integration for Dynamic Resource Management in HPC

Introduction

Dynamic Resource Management (DynRM) is central to modern High-Performance Computing (HPC) as system complexity and resource heterogeneity continue to increase. Malleable MPI applications—those capable of runtime process topology changes—require robust, reproducible, and sustainable validation methodologies for correct operation under diverse conditions. The paper "A Test Taxonomy and Continuous Integration Ecosystem for Dynamic Resource Management in HPC" (2604.26824) addresses crucial gaps in the systematic testing and CI of DynRM frameworks by proposing a taxonomy of test categories and a CI ecosystem, instantiated and evaluated with the Dynamic Management of Resources (DMR) middleware.

DynRM and Malleability Testing: Context and Motivation

Traditional testing in DynRM and malleability libraries primarily relies on bespoke, manual experiments with poor reproducibility and limited automation. The absence of standardized integration and system tests complicates validation, especially as MPI runtimes, malleability middleware, and resource managers (notably Slurm) evolve independently. The complexity of modern HPC stacks—with intricate initialization, monitoring, and reconfiguration primitives—mandates structured, framework-agnostic test strategies and CI workflows.

The paper identifies persistent architectural patterns across leading malleability frameworks—including PCM, AMPI, Flex-MPI, Elastic MPI, ULFM, DPP, and Proteo—characterized by distinct primitives for initialization, reconfiguration checks, and dynamic transition management. Despite API and backend differences, their life cycles are structurally congruent, as shown in the malleability workflow:

Figure 1: Basic malleability workflow, illustrating the foundational cycle of initialization, periodic reconfiguration checks, and resource or process adaptation within a running HPC application.

These observations underpin the core hypothesis: an abstract, semantically defined test taxonomy mapped to these phases can be instantiated for heterogeneous DynRM implementations, promoting sustainable validation and continuous delivery.

Taxonomy of Tests for MPI Malleable Libraries

The proposed taxonomy delineates two orthogonal test classes—component integration and system-level tests—explicitly mapped to the malleability workflow (initialization, check, reconfiguration). The objective is to maximize coverage of dynamic behavior through tests that exercise real dependencies (MPI runtimes, resource managers) instead of synthetic mocks.

Component Integration Testing

• Initialization: Tests ensure all preconditions and dependencies are met, checking for environment correctness, valid state transitions, and robust option parsing.
• Check: Tests verify that readiness and policy invariants are enforced at handoff points, including state validation, guard inhibition logic, constraint satisfaction, and request lifecycle transitions.
• Reconfiguration: These focus on the integrity and correctness of scale-up/scale-down operations, both in resource allocation and in process layout/data redistribution.

System-Level Testing

• Functional: End-to-end tests capture realistic reconfiguration patterns, from linear growth/shrink cycles to arbitrary, nonlinear evolution scenarios, with focus on robust state handling and absence of latent errors across transition cycles.
• Non-functional: Tests assess core attributes (robustness, latency, scalability), enforcing time constraints for resize operations and exploring maximum tested scales.

This taxonomy directly supports automation and version-aware validation, crucial for continuous integration.

Continuous Integration Ecosystem for Dynamic DynRM Libraries

The CI ecosystem is instantiated in a containerized, virtual cluster emulating the MPI and resource manager stack, enabling portable, reproducible pipelines. It integrates Jenkins (or any similar orchestrator) with Docker Compose-based clusters running Slurm and all required MPI runtimes.

Figure 2: Technology stack illustrating the integration between application, DynRM (e.g., DMR), MPI runtime, and resource manager.

Figure 3: Jenkins deployment architecture, utilizing Docker Compose for isolated, reproducible CI workflows with clear segregation of pipeline orchestration and workload execution.

The Groovy-based pipeline enables dynamic test selection, conditional resource bootstrapping, and isolation of Slurm/MPI instances per test, facilitating granular validation and regression detection upon each code or dependency update.

DMR as a Case Study: Instantiation and Pipeline Realization

DMR (Dynamic Management of Resources) operates as a representative middleware solution, integrating closely with scientific applications, MPI runtimes, resource monitors, and Slurm RMS to enable malleability at production scale. DMR’s core logic comprises well-defined API routines—dmr_init, dmr_check, dmr_reconfigure, and dmr_finalize—which together implement a state-driven malleability protocol.

Figure 4: DMR architecture, highlighting the mediation between application, MPI, performance monitor, and RMS.

Figure 5: DMR Core API state diagram, modeling legal state transitions as enforced by systematic component integration tests.

DMR-specific tests are constructed following the taxonomy:

• State, environment, and precondition validation for initialization.
• Correct check and guard logic enforcing policy and resource constraints.
• Reconfiguration correctness and integrity checks for add/remove operations and data redistributions.

CI Pipeline Operation and Versioned Slurm Compatibility

The instantiated CI pipeline for DMR, summarized below, demonstrates systematic validation across Slurm versions, detecting both behavioral and compilation incompatibilities.

Figure 6: DMR CI pipeline, integrating compilation, component, functional, and non-functional tests across native and malleability-enabled Slurm versions.

Tests are conducted in both production (e.g., MareNostrum 5) and containerized environments, confirming correctness under realistic contention and resource allocation conditions. The approach exposes Slurm API incompatibilities across releases, supporting both immediate fault detection post-update and sustainable adaptation as system software evolves.

Implications and Future Directions

The demonstrated methodology enables high-confidence, reproducible validation for DynRM software in complex, evolving HPC environments. Early detection of API and behavioral incompatibilities is particularly impactful in layered middleware, minimizing downtime and user disruption during dependency upgrades. The framework-agnostic design is extensible to new releases, alternative resource managers (OAR, PBS, Flux), and non-MPI programming models.

Future work must consider:

• Generalization to non-Slurm resources and alternative task models (e.g., AMT).
• Extreme-scale testing and non-trivial integration with facility-operated CI deployments.
• Integration of application-specific performance and energy metrics.

Conclusion

The structured test taxonomy and integrated CI ecosystem detailed in this work advance the state-of-the-art in sustainable DynRM middleware validation. Deploying these best practices in frameworks such as DMR yields superior maintainability, safer upgrades, and easier integration into production science workflows as both infrastructure and application codebases evolve. The approach is generalizable, with high transferability to current and future malleability solutions in HPC.