MPI Malleability Validation under Replayed Real-World HPC Conditions

Abstract: Dynamic Resource Management (DRM) techniques can be leveraged to maximize throughput and resource utilization in computational clusters. Although DRM has been extensively studied through analytical workloads and simulations, skepticism persists among end administrators and users regarding their feasibility under real-world conditions. To address this problem, we propose a novel methodology for validating DRM techniques, such as malleability, in realistic scenarios that reproduce actual cluster conditions of jobs and users by replaying workload logs on a High-performance Computing (HPC) infrastructure. Our methodology is capable of adapting the workload to the target cluster. We evaluate our methodology in a malleability-enabled 125-node partition of the Marenostrum 5 supercomputer. Our results validate the proposed method and assess the benefits of MPI malleability on a novel use case of a pioneer user of malleability (our "PhD Student"): parallel efficiency-aware malleability reduced a malleable workload time by 27% without delaying the baseline workload, although introducing queueing delays for individual jobs, but maintaining the resource utilization rate.

• The paper introduces a novel method using real supercomputer logs to validate MPI malleability under authentic HPC conditions.
• It employs the UBS platform to replay user workloads, preserving statistical job distributions and circadian patterns for realistic testing.
• Experimental results demonstrate that adaptive malleability policies can cut malleable job completion times by up to 27% without degrading overall utilization.

Validation of MPI Malleability with Real-World Workload Replay in HPC Systems

Introduction

The paper "MPI Malleability Validation under Replayed Real-World HPC Conditions" (2604.26576) addresses critical barriers to practical adoption and validation of dynamic resource management (DRM) techniques―specifically MPI malleability―in current HPC infrastructures. While theoretical and simulation-based studies have long touted the benefits of DRM for cluster throughput and resource efficiency, skepticism persists regarding their translation to production clusters. The primary cause has been a deficiency of experiments that faithfully replicate real-world queuing dynamics, user submission patterns, and system contention. This work proposes and implements a methodology to replay authentic user workloads, sampled from supercomputer logs, onto malleability-enabled clusters to close this validation gap.

Methodological Framework

The paper's methodological innovation lies in a hybrid experimental platform—the User-Based Submitter (UBS)—capable of replaying realistic workloads sampled from supercomputer logs. Critical aspects of true cluster usage are preserved by sampling at the user level, not by artificial generation of synthetic jobs. Job arrival and think times maintain the original statistical properties and circadian patterns Figure 1, thus embedding authentic system saturation and fragmentation phenomena.

Figure 1: Temporal distribution of job submissions per day, illustrating bursty and clustered arrivals consistent with real supercomputer usage.

The baseline workload is extracted from the KIT-FH2-2016 trace and rescaled via a calibrated factor to match the physical resources and experimental constraints of the 125-node Marenostrum 5 testbed. User sampling converges to a target daily utilization ($\sim$84%) without distortion of inter-user dependencies or job size distribution. Job runtime and size statistics are displayed in Figures 4 and 5.

Figure 2: Cumulative and probability distribution of job execution times in the baseline workload, showing substantial prevalence of short jobs with a minority consuming long intervals.

Figure 3: Distribution of job sizes and their proportion in overall system node-hours, highlighting the fat-tailed resource demand typical in production clusters.

The experimental campaign introduces a single generative, malleability-enabled user—modeled after a compute-intensive PhD student—who injects a sequence of ten malleable jobs (MPDATA solver) interleaved into the rigid baseline. Dynamic resource management is provided via the DMRlib middleware interfacing with a malleability-enhanced Slurm. Job runtime malleability is governed by either aggressive growth, or parallel-efficiency-aware policies informed by online performance monitors.

Figure 4: Schematic of DMRlib’s integration between malleable applications, MPI interfaces, and Slurm resource managers, enabling dynamic job resizing and data redistribution.

Experimental Results

Makespan and Resource Allocation

A critical evaluation metric is makespan reduction for the generative/malleable user, while maintaining baseline workload performance and overall system utilization. Five experimental conditions are contrasted: baseline-only, two rigid configurations (StaticN32, StaticN16), and two malleable (AlwaysGrow, ParEfficiency).

Parallel-efficiency-aware malleability (ParEfficiency) achieves a 27% reduction in the malleable workload time over the rigid Static32 scenario without prolonging the baseline workload's completion. This reduction is not due to greater system saturation—resource utilization remains constant across experiments (Table 1)—but arises from efficient backfilling: malleable jobs exploit fragmented resources sooner, then expand or shrink so as to maintain high parallel efficiency and avoid wasteful over-provisioning.

Experiments demonstrate that rigid large jobs (StaticN32) maximize resource consumption but actually increase the user’s completion time due to severe queuing delays, while more granular malleable jobs finish earlier by adapting to available slots Figure 5.

Figure 5: Average job completion (waiting plus execution time) per configuration, highlighting decreased times for malleable policies.

Figure 6: Node-hour consumption per job under each policy, showing both the efficiency gains and variability of malleable approaches.

Queuing and Waiting Time Dynamics

While aggregate system utilization is unaffected, increased user-level queueing is observed for jobs competing alongside the malleable workflow. The average waiting time for baseline jobs escalates up to 2.5$\times$ when malleable jobs are introduced, a direct consequence of increased resource contention. However, malleable policies that prioritize shrinkage of low-efficiency jobs can prioritize waiting rigid jobs with negligible effect on the overall experiment makespan.

Overhead and Scalability

Reconfiguration overheads are quantified for different policy configurations. Overly aggressive expansion (AlwaysGrow) leads to unnecessary reconfigurations and increased execution overhead. In contrast, dynamic policies that avoid expensive resizing steps (guided by current parallel efficiency and historical reconfiguration cost, see Figure 7) achieve better overall throughput by minimizing reconfiguration penalties.

Figure 8: MPDATA scalability, delineating the non-linear speedup window—the performance regime where malleability offers maximal node-hour savings.

Figure 7: Empirical profiling of MPDATA reconfiguration times across node-scale transitions, showing quadratic growth for large resizing steps.

Implications and Theoretical Advances

This work provides the first live, non-simulated validation of MPI malleability under authentic queuing, submission, and resource contention patterns using real workload logs. It sharply contrasts with prior methods reliant on synthetic workloads, closed-benchmarking, or single-application pilot studies, and robustly demonstrates that user-level makespan gains are attainable without sacrificing global utilization or system efficiency.

Crucially, the methodology is portable: user log replay and feedback-based job generation can be instantiated on arbitrary clusters with different RMS and malleability backends. This positions the approach as a reference for site-specific validation of DRM strategies, allowing institutions to empirically justify malleability-driven investments.

At a theoretical level, the work substantiates that fragmented resource availability—a systemic property of HPC environments—is best exploited by fine-grained, performance-aware malleability, rather than statically tuned job sizes. The malleability-induced ability to fill resource gaps and adapt allocation in response to system-level events (such as job completions or high queue priority) is both empirically validated and superior to naive policies that prioritize maximum job parallelism.

Incentive mechanisms for users are an open avenue, as malleable jobs can extract greater scientific output per consumed node-hour without penalizing other users, suggesting that quota-discount systems for such job types are justifiable.

Limitations and Future Directions

Despite its experimental robustness, the work acknowledges several limitations: use of a single application and log, user isolation in malleable job injection, and a compressed replay timescale. The implications for jobs with true resource sharing (as opposed to exclusive node allocations) or multi-tenant malleable workloads require further validation. Extending to a suite of representative applications and alternate logs, with statistical quantification over repeated experiments, is an essential future milestone.

From a systems perspective, future work should integrate more sophisticated RMS policies (fairness, backfill, energy-awareness), exploit standardized interfaces (OpenMPI/UCX), and consider interoperability with workflow managers for elastic scientific pipelines.

Conclusion

This study delivers a rigorously validated, field-tested methodology for evaluating the operational impact of MPI malleability under real-world HPC conditions (2604.26576). The confirmation that malleable jobs enable substantial makespan reductions for users without degrading cluster utilization or prolonging baseline workloads paves the way for mainstream adoption and deeper integration of DRM in production supercomputing. The results further establish that malleability awareness—not maximal scale—optimally leverages the intrinsic fragmentation of cluster environments, and that adaptive, parallel-efficiency-aware policies offer the best practical trade-offs among resource consumption, performance, and scheduler-induced delays.