A Treasure Trove of Performance: Analyzing the IO500 Submission Data

Abstract: The IO500 benchmark has become the community standard for evaluating HPC storage system performance, yet the detailed data contained in its submission packages remains largely unexplored beyond aggregate leaderboard rankings. We present a statistical characterization of 61 IO500 submissions from four competition lists (ISC21 through SC22), examining score distributions, inter-phase correlations, and insights derived from detailed log files that accompany each submission. Our analysis reveals that IO500 scores span four orders of magnitude. Spearman correlation analysis shows strong within-domain clustering for both bandwidth (rs = 0.78 to 0.96) and metadata (rs = 0.89 to 0.98) phases, with the composite sub-scores exhibiting rs = 0.92 at per-node level (Pearson r = 0.53). Log-level analysis uncovers file-system-specific patterns in IOR close-time overhead, straggler behavior during the stonewall wear-down phase, and parallel-find load imbalance that are invisible in aggregate scores. These findings demonstrate that IO500 submission packages constitute a valuable research resource for understanding storage system behavior. The full submission dataset is publicly available at https://github.com/IO500/submission-data, and analysis scripts at https://gitlab-ce.gwdg.de/hpc-team/io500-analysis.

• The paper provides a detailed statistical characterization of IO500 benchmark data, highlighting performance variability across diverse system architectures.
• It applies robust statistical methods, including Spearman correlations and Kruskal-Wallis H-tests, to reveal nuanced performance trends per benchmark phase.
• Log-derived analyses expose file-system-specific behaviors and operational bottlenecks, offering actionable insights for enhancing HPC storage design.

Statistical Characterization of IO500 Submission Data: Performance Distributions, Correlations, and Log-Derived Insights

Introduction

The study "A Treasure Trove of Performance: Analyzing the IO500 Submission Data" (2605.02194) provides a comprehensive statistical characterization of the IO500 benchmark dataset, focusing on detailed distributions, cross-phase correlations, and log-derived insights from 61 submissions spanning four competition lists (ISC21 to SC22). The IO500 benchmark suite, which combines bandwidth (IOR) and metadata (MDTest) measurements, is widely adopted by the HPC community for storage system performance evaluation. This work extends beyond aggregate leaderboard rankings, revealing nuanced architectural and operational behaviors by systematically analyzing submission packages, including per-process timing and metadata.

Dataset Composition and Methodology

The dataset is notable for its scale heterogeneity, encompassing systems equipped with Lustre, GPFS/Spectrum Scale, DAOS, WekaFS, BeeGFS, and an array of interconnects, predominantly InfiniBand HDR and EDR. The analysis pipeline encompasses acquisition from the VI4IO archive, quality normalization, and validation, followed by statistical analyses utilizing Spearman rank correlations, Kruskal-Wallis H-tests, and robust visualization techniques.

The scope is limited to 61 submissions due to log file availability at the time. Data cleaning mitigated inconsistencies in file system naming, interconnect speed, and deployment metadata, but some limitations persist due to self-reporting and ambiguous scale metrics.

Score Distributions and Variability

The IO500 scores exhibit extreme variability, spanning approximately four orders of magnitude, indicative of substantial heterogeneity in system scale and architecture. High coefficients of variation (CV $>$ 2.4) were observed across all metrics, with "hard" configurations demonstrating larger dispersions compared to "easy" configurations, underscoring the sensitivity of small-I/O and metadata operations to architectural subtleties.

Figure 1: IO500 scores across 61 submissions reveal extensive variability by file system type, with scores spanning four orders of magnitude (log scale).

Aggregate scores skew right, driven by a subset of high-performance systems. Per-node normalization enables direct architectural efficiency comparisons, removing node count effects which otherwise dominate leaderboard rankings.

Correlation Patterns Across Benchmark Phases

Spearman rank correlation matrices demonstrate strong within-domain clustering for both bandwidth (IOR) and metadata (MDTest) phases, with $r_s = 0.78$–$0.96$ (bandwidth) and $r_s = 0.89$–$0.98$ (metadata). Composite sub-scores (Score$_{BW}$ vs. Score$_{MD}$) also display strong correlation at the per-node level ($r_s = 0.92$, Pearson $r = 0.53$), a result of geometric mean aggregation and residual scale effects. However, individual cross-domain correlations are weaker, affirming the IO500's multidimensional scoring philosophy—no single phase is universally redundant.

Figure 2: Per-node normalized Spearman correlations matrix demonstrates distinct phase clustering and strong within-domain associations.

Interconnect Speed and Node-Level Performance

Analysis by interconnect speed reveals that higher reported network bandwidth is generally associated with higher per-node IO500 performance, particularly for bandwidth-sensitive phases (IOR-easy). However, Kruskal-Wallis results indicate only moderate effect sizes ($\eta^2 = 0.28$ for IOR-easy per-node bandwidth) and fail to reach statistical significance for overall score ($r_s = 0.78$0), reflecting the influence of confounding factors including storage media type, file system architecture, and scale.

Figure 3: Overall per-node IO500 score, grouped by interconnect speed, highlights variability and indirect association with interconnect bandwidth.

Log-Derived Insights: Runtime and Overheads

Detailed log analysis uncovers file-system-specific behavioral patterns invisible in aggregate scores. Analysis of phase runtimes affirms the influence of the stonewall mechanism, with write phases consistently bounded below by 300 seconds, and variations emerging during wear-down and close operations.

Figure 4: Runtime distributions across IO500 phases show stonewall-enforced minimums and rapid completion for some read/stat phases, suggesting possible residual caching.

Close-time overhead analysis reveals pronounced disparities: Lustre exhibits up to tens of seconds for file closing (attributable to cache flushing and metadata finalization), while DAOS demonstrates negligible overhead, aligning with its persistent-memory architecture.

Figure 5: IOR close-time overhead varies dramatically across file systems, with Lustre incurring significant costs and DAOS remaining minimal.

Process-Level Straggler Analysis

Stonewall-relative Q-Q analysis of per-process IOR write runtimes reveals sensitivity to access patterns: IOR-hard configurations are vulnerable to process-level stragglers with deviations up to $r_s = 0.78$1 stonewall time, particularly on HDD-based Lustre deployments. Architectural patterns emerge, such as clustered stragglers (WekaFS) indicative of storage bucket contention, and contiguous stragglers (Lustre) arising from OST-level hotspots.

Figure 6: WekaFS exhibits clustered stragglers in IOR-hard write, revealing architectural bottlenecks at the bucket level.

These findings highlight the value of analyzing log-derived metrics (close time, process completion uniformity, straggler patterns) for illuminating file-system-specific bottlenecks and contention mechanisms.

Parallel Find Load Imbalance

Log analysis of the pfind phase reveals extreme load imbalance, with singular processes traversing over 5 million files while others handle orders-of-magnitude fewer. This reflects inherent challenges in parallel directory traversal, compounded by MDTest-hard’s per-process directory mapping. Partial mitigation via job-stealing is achieved, but most processes remain under-utilized, indicating further opportunities for improved parallel scheduling.

Practical and Theoretical Implications

The research provides actionable guidance for HPC storage system procurement, benchmarking, and design:

• Procurement Strategy: Aggregate scores mask architectural efficiency; per-node and phase-specific analysis should guide workflow-appropriate system selection.
• Benchmarking Practice: Log-level breakdowns (close-time, process-level completion, straggler analysis) are essential for identifying bottlenecks beyond throughput.
• IO500 Community: Submission packages constitute a valuable resource for longitudinal studies and cross-site modeling; improved metadata reporting is necessary for robust analytics.
• Storage Design: Architecture-specific contention patterns inform optimization for storage mapping and load distribution.

From a theoretical perspective, the analysis demonstrates that multidimensional benchmarking and log-level instrumentation are vital for capturing the breadth of storage behaviors in HPC environments. Patterns observed in this well-tuned dataset may evolve as emerging architectures and workload paradigms are integrated, and more expansive datasets become publicly available.

Conclusion

This study offers a foundational statistical characterization of IO500 submission data, documenting substantial performance variability, strong within-phase correlations, and log-derived architectural insights. While constrained by dataset size and metadata quality, the methodology and findings underscore the value of deep submission package analysis for informing procurement, benchmarking practice, and HPC storage system design. Future work will extend these analyses to the full IO500 dataset for robust, longitudinal insights, and develop automated anomaly detection pipelines for enhanced research utility.