IO500 Benchmark Overview

• IO500 is a comprehensive benchmark for HPC storage systems that assesses both bandwidth and metadata operations using varied I/O patterns.
• It integrates large sequential and small-file I/O tests with metadata operations like file creation and namespace traversal to capture detailed performance nuances.
• Log analyses reveal per-node behavior, straggler effects, and filesystem-specific patterns, informing practical tuning and scheduler adjustments.

IO500 is the community’s de facto benchmark for characterizing HPC storage systems across two complementary performance domains: bandwidth and metadata. Rather than a single throughput number, it combines large sequential and small-file I/O via IOR with metadata-intensive operations via MDTest and a namespace traversal rate via parallel find, “pfind.” The benchmark has become the community standard for evaluating HPC storage system performance, while detailed analyses of submission packages have shown that the most informative evidence often lies beyond aggregate leaderboard rankings, in per-node normalization, cross-phase correlation structure, and the per-process logs that accompany official results (Kunkel et al., 4 May 2026).

1. Scope and analytical setting

The paper “Statistical Characterization of IO500 Submission Data: Performance Distributions, Correlations, and Log-Derived Insights” analyzes 61 submission packages spanning four competition lists—ISC21, SC21, ISC22, SC22—drawing on both their aggregate scores and the detailed per-process logs that accompany each official result. This sample covers a diverse set of file systems, including Lustre, GPFS/Spectrum Scale, DAOS, WekaFS, and BeeGFS, and interconnects including IB HDR/EDR and Omni-Path. The data was sourced from the VI4IO archive at the time of data collection and represents well-tuned public submissions in the 2021–2022 window (Kunkel et al., 4 May 2026).

This framing is important because IO500 is not presented merely as a ranking mechanism. The submission packages include CSV timing traces, stonewall statistics, and operation-level breakdowns, which allow storage-system behavior to be studied at a granularity that aggregate scores cannot expose. A plausible implication is that IO500 functions simultaneously as a benchmark suite and as an observational corpus for empirical storage-systems research.

2. Benchmark composition and workload model

IO500 organizes evaluation around a bandwidth phase, a metadata phase, and a namespace traversal phase. In the bandwidth phase, IOR-easy uses a single shared file and large sequential I/O to emulate optimized scientific workflows. IOR-hard uses a file-per-process pattern with interleaved small records, for example 47,008-byte blocks, stressing small-write behavior and metadata interactions typical of checkpoint/restart. Both report throughput in GiB/s (Kunkel et al., 4 May 2026).

In the metadata phase, MDTest-easy targets a shared-directory layout, while MDTest-hard uses a directory-per-process structure with small payloads. MDTest reports metadata operation rates for create, stat, read, and delete in kIOPS. Parallel find, or pfind, measures rate of namespace traversal across the file trees created by MDTest, also in kIOPS. Within IO500’s design, pfind is part of the metadata sub-score rather than an independent third domain, which reinforces that namespace traversal is treated as a metadata-intensive behavior rather than as bulk data movement.

A further organizing mechanism is the pair stonewall and wear-down. IO500 imposes a 300-second minimum runtime, “stonewall,” for write phases. After the stonewall is reached, processes complete in-flight work during a “wear-down” period. This exposes stragglers and bulk-synchronous effects that are invisible in a steady-state throughput alone. The benchmark therefore encodes not only rate-based performance but also completion regularity under sustained operation.

3. Scoring, aggregation, and normalization

IO500 aggregates its components into sub-scores and an overall score using geometric means, intentionally blending different units, GiB/s and kIOPS, so neither domain dominates purely by scale. In the paper, the scoring is described as follows (Kunkel et al., 4 May 2026):

$$\text{Score}_{\text{BW}} = \left( S_{\text{ior-easy-w}} \cdot S_{\text{ior-easy-r}} \cdot S_{\text{ior-hard-w}} \cdot S_{\text{ior-hard-r}} \right)^{1/4}$$

$$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$

$$\text{Score}_{\text{overall}} = \sqrt{ \text{Score}_{\text{BW}} \times \text{Score}_{\text{MD}} }$$

These formulas implement the geometric mean as a central aggregation concept in IO500:

$$\operatorname{GM}(x_1,\dots,x_n) = \left(\prod_{i=1}^{n} x_i\right)^{1/n}$$

To compare heterogeneous system scales, the paper normalizes phase scores per node and per process and employs correlation analyses at those normalized levels. Because benchmark data are strongly skewed, rank-based statistics are emphasized. Spearman’s rank correlation coefficient is defined as

$$\rho = 1 - \frac{6\sum_{i=1}^{n} d_i^2}{n(n^2 - 1)}$$

where $d_i$ is the difference in ranks of paired observations and $n$ is the sample size. Pearson correlation is also reported for comparison:

$$r = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})} {\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2}\,\sqrt{\sum_{i=1}^{n}(y_i - \bar{y})^2}}$$

The statistical treatment includes Spearman with Benjamini–Hochberg FDR control at $\alpha = 0.05$, Pearson for comparison, Kruskal–Wallis group tests, cleaning and normalization of file-system naming and interconnect speed groupings, validation, and visualization. This suggests that IO500 analysis, at least in this study, is explicitly distribution-aware rather than dependent on Gaussian assumptions.

4. Distributional properties and correlation structure

Across the 61 submissions, overall IO500 scores span roughly 10,000×, illustrating extreme heterogeneity in system scale and architecture. Summary statistics reinforce strong right-skew, with means far above medians and coefficients of variation above 2 across all metrics. The overall score ranges from 3.2 to 36,850 with $CV \approx 4.17$; the bandwidth sub-score ranges from 0.6 to 3,422 with $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$0; and the metadata sub-score ranges from 10.5 to 396,873 with $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$1 (Kunkel et al., 4 May 2026).

Correlation analysis reveals domain-wise clustering rather than a single axis of performance. Spearman correlations among bandwidth-phase components are strong, with $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$2 to $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$3, and among metadata-phase components even stronger, with $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$4 to $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$5. This indicates that, in this dataset, the IOR variants co-move as a group and MDTest variants co-move as a group, even after per-node normalization. Cross-domain correlations are positive but weaker, supporting IO500’s design to measure multiple dimensions of storage behavior rather than a single scalar notion of storage quality.

At the per-node level, the composite sub-scores also show a strong rank correlation between $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$6 and $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$7, with $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$8, but only a moderate Pearson correlation, with $$\text{Score}_{\text{MD}} = \left( S_{\text{md-easy-w}} \cdot S_{\text{md-easy-s}} \cdot S_{\text{md-hard-w}} \cdot S_{\text{md-hard-s}} \cdot S_{\text{find}} \right)^{1/5}$$9. The reported interpretation is that systems with more resources tend to do well across both domains, but linear co-variation is modest once outliers are considered. Additional grouping by interconnect suggests that higher reported network speeds associate with higher per-node IOR-easy bandwidth, with Kruskal–Wallis $$\text{Score}_{\text{overall}} = \sqrt{ \text{Score}_{\text{BW}} \times \text{Score}_{\text{MD}} }$$0, $$\text{Score}_{\text{overall}} = \sqrt{ \text{Score}_{\text{BW}} \times \text{Score}_{\text{MD}} }$$1, and $$\text{Score}_{\text{overall}} = \sqrt{ \text{Score}_{\text{BW}} \times \text{Score}_{\text{MD}} }$$2, whereas the overall per-node score association is weaker, with $$\text{Score}_{\text{overall}} = \sqrt{ \text{Score}_{\text{BW}} \times \text{Score}_{\text{MD}} }$$3, $$\text{Score}_{\text{overall}} = \sqrt{ \text{Score}_{\text{BW}} \times \text{Score}_{\text{MD}} }$$4, and $$\text{Score}_{\text{overall}} = \sqrt{ \text{Score}_{\text{BW}} \times \text{Score}_{\text{MD}} }$$5. These are associational findings, limited by self-reported metadata quality and confounding.

5. Log-derived behavior and file-system-specific patterns

A central contribution of the paper is mining the per-process logs bundled with IO500 submissions to uncover behavior patterns that aggregate scores hide. One example is IOR close-time overhead. IOR logs record the time spent closing files, and this time is included in measured runtime. On Lustre, close times can reach tens of seconds, reflecting cache flushes and metadata finalization costs; this is application-visible latency. On DAOS, close overhead is negligible, consistent with its object-store and persistent-memory design. The practical consequence stated in the paper is that two systems with similar write throughput may present very different end-of-I/O latencies, which matter for iterative HPC workflows and checkpointing (Kunkel et al., 4 May 2026).

A second pattern concerns stragglers in the stonewall wear-down phase. The analysis examines the ratio of each process’s total runtime to the stonewall time; values near 1.0 indicate uniform completion, while larger values indicate stragglers. IOR-easy generally shows ratios near 1.0 with modest tails, suggesting more balanced completion. IOR-hard exhibits much larger deviations, up to 2× to 5×, revealing significant straggler sensitivity in small-file patterns. File-system fingerprints also emerge: WekaFS shows clustered groups of slow processes, consistent with contention within storage buckets, while Lustre shows contiguous ranges of slow ranks, pointing to hotspots on particular OSTs due to striping or mapping. These effects are invisible in a single throughput number but are directly relevant to parallel efficiency and jitter.

A third pattern is parallel find load imbalance. In pfind, a single process may check over five million files while others handle about 100,000, even with job-stealing. MDTest-hard’s per-process directory trees are difficult to divide evenly, and most processes spend more time waiting than actively traversing. The paper states that this macroscopic imbalance depresses metadata and find scores and indicates opportunities for scheduler improvements in find implementations and directory layout strategies in benchmarks. This suggests that pfind results can encode scheduler behavior and namespace geometry, not merely raw namespace lookup capability.

6. Interpretation, reproducibility, and limitations

The paper advises against over-interpreting the leaderboard. The overall geometric mean blends different units and domains and is heavily influenced by system scale. Phase scores, bandwidth versus metadata, and per-node normalization are recommended for understanding architectural efficiency and workload relevance. A system that ranks lower overall might still be the best choice for metadata-heavy applications. The same caution applies to correlations: strong within-domain Spearman correlations, up to $$\text{Score}_{\text{overall}} = \sqrt{ \text{Score}_{\text{BW}} \times \text{Score}_{\text{MD}} }$$6, suggest that components in the same domain often rise and fall together in well-tuned systems from 2021–2022, but this does not make any phase redundant because the benchmark intentionally spans a broad performance space (Kunkel et al., 4 May 2026).

The practical tuning implications reported in the paper are correspondingly phase-aware and log-aware. On Lustre-like systems, long close times point to cache and metadata bottlenecks; practical steps include tuning client writeback parameters, RPC credits, MDS/MDT performance, stripe counts or widths, and commit policies to reduce end-of-I/O latency. To reduce stragglers, the paper recommends using stonewall-relative ratios or per-process Q–Q views to identify lagging rank groups and then rebalancing stripe targets, avoiding oversubscription of specific OSTs or storage buckets, aligning process-to-file mappings, and verifying that per-process file sizes and record sizes do not disproportionately stress particular back-ends. For parallel find, suggested responses include adjusting pfind’s job-stealing chunk sizes, increasing worker count relative to directory breadth, or refactoring MDTest parameters and directory layouts to improve partitionability. The paper also notes that read and stat phases sometimes complete in about 10 seconds, raising concerns about residual caching despite read-after-write rules; practitioners are advised to confirm cache-clearing between phases or incorporate cache-awareness in result interpretation.

Reproducibility is an explicit part of the study. The submission data repository is available at https://github.com/IO500/submission-data, which now includes 131+ sites, while the paper used the 61-submission subset available at analysis time. The analysis scripts are available at https://gitlab-ce.gwdg.de/hpc-team/io500-analysis. Readers can rerun the pipeline or expand it to the full repository for longitudinal or cross-architecture studies.

The scope of the reported findings is bounded. The analyzed submissions are from ISC21 to SC22 and are likely skewed toward well-tuned, public results. Since the analysis, the community released a larger corpus, 131+ sites, which could alter correlation magnitudes and distributions. Self-reported details, including interconnect speed, are sometimes ambiguous, and NIC counts per node are not consistently reported; interconnect speed is therefore treated as a categorical grouping variable rather than a precise bandwidth measure. Multiple submissions can originate from the same site, potentially violating independence assumptions; p-values are approximate; and the findings are descriptive of the analyzed sample, not necessarily causal or generalizable.

Within these limits, IO500 emerges as a benchmark that measures multiple, partially overlapping aspects of storage performance. The paper’s strongest conclusion is that the submission packages themselves constitute a valuable research resource for understanding storage system behavior, because per-node normalization, domain-wise correlations, and especially log-derived evidence on close-time overheads, stragglers, and find imbalance provide a more discriminating view than any single composite score.