The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains

Abstract: Benchmark evaluation across AI and safety-critical domains overwhelmingly relies on simple averaging. We demonstrate that this practice produces substantially misleading rankings when two conditions co-occur: (1) the evaluation matrix is sparse and (2) items vary substantially in difficulty. Through controlled simulation experiments across four domains -- NLP (GLUE), clinical drug trials, autonomous vehicle safety, and cybersecurity -- we show that Spearman rank correlation $ρ$ between simple-average rankings and ground-truth rankings degrades from $ρ= 1.000$ at 100% coverage to $ρ= 0.809$ at 67% coverage with high difficulty heterogeneity (mean over 20 seeds). A standard two-parameter logistic (2PL) Item Response Theory (IRT) model maintains $ρ\geq 0.996$ across all conditions. A 150-condition grid sweep over sparsity $S \in [0, 0.70]$ and difficulty gap $D \in [0.5, 5.0]$ confirms that ranking error forms a failure surface with a strong $S \times D$ interaction ($γ_3 = +0.20$, $t = 13.05$), while IRT maintains $ρ\geq 0.993$ throughout. We discuss implications for Physical AI benchmarking, where evaluation matrices are often incomplete and difficulty gaps are extreme.

• The paper demonstrates that simple averaging loses ranking accuracy as sparsity, item-difficulty gaps, and biased missingness increase, reaching ρ = 0.24 in extreme simulations while 2PL IRT remains above ρ = 0.993.
• The paper validates the failure pattern across NLP, clinical trials, autonomous-vehicle safety, and cybersecurity, where IRT improves ranking recovery from averages as low as ρ = 0.809 to near-perfect accuracy.
• The paper recommends measuring coverage and difficulty heterogeneity, using IRT when evaluation data are sparse or non-random, and validating the approach on real episode-level Physical AI benchmarks.

Overview and motivation

This paper examines a methodological weakness at the core of benchmark evaluation: the use of simple arithmetic averaging to aggregate per-item performance into a single system score. The author, an independent researcher, argues that this estimator is unbiased only under full coverage or missing-completely-at-random (MCAR) observation patterns—conditions that rarely hold in practice. When evaluation matrices are sparse and items differ substantially in difficulty, systems tested on easy subsets are systematically inflated while systems tested on hard subsets are penalized. The paper's central contribution is the identification and controlled validation of what the author terms the Evaluation Failure Scaling Law: the rank-order accuracy of simple averaging degrades monotonically as a function of the product of sparsity $S$ and item difficulty gap $D$, whereas two-parameter logistic (2PL) Item Response Theory (IRT) estimation remains robust across the entire $(S, D)$ surface.

The work builds on a psychometrics tradition applied recently to NLP evaluation (2605.11205), including IRT-based leaderboards [rodriguez2021], tinyBenchmarks [polo2024], PSN-IRT analysis across 11 LLM benchmarks [zhou2025a], and benchmark harmony diagnostics [uzunoglu2025]. The stated gap is that prior IRT work has largely been confined to single benchmark ecosystems, leaving open whether simple-averaging failure can be characterized as a function of measurable evaluation-matrix properties across domains.

Experimental design

The study is a controlled simulation. For each domain, ground-truth parameters $(\theta^*_j, a^*_i, b^*_i)$ are calibrated to published data; a realistic observation mask $M$ is generated; binary responses are sampled from the 2PL model $P(R_{jik}=1) = \sigma(a_i(\theta_j - b_i))$; and both estimators are scored against the known ground-truth ranking via Spearman $\rho$. Simulation is justified as the only design permitting definitive comparison against ground truth, following standard practice for evaluating statistical estimators.

Four domain-calibrated conditions span the density–heterogeneity spectrum:

Domain	Coverage	Difficulty gap $D$	Trials per pair
NLP (GLUE)	100%	1.61	500
Clinical drug trials	65%	2.50	200
AV safety	60%	2.50	1000
Cybersecurity	67%	3.50	500

Each sparse domain embeds a deliberately adversarial pair: a "Fake" system of mediocre ability tested only on easy items, and a "True" system of top ability ($\theta^* = +2.0$) tested only on hard items. Missingness constraints require each system to appear on at least 2 items and each item on at least 3 systems. IRT parameters are estimated by marginal maximum likelihood via L-BFGS-B with Gaussian priors, and standard errors come from the inverse observed Fisher information.

Main results

The cross-domain pattern is consistent: simple averaging degrades with sparsity and difficulty heterogeneity while IRT maintains near-perfect rank recovery (means over 20 seeds):

Domain	$\rho_{\text{avg}}$	$D$0
NLP (GLUE)	$D$1	$D$2
Clinical trials	$D$3	$D$4
AV safety	$D$5	$D$6
Cybersecurity	$D$7	$D$8

The critical rank displacements illustrate the mechanism concretely. In clinical trials, the Fake Miracle Drug is inflated three positions by simple averaging; in cybersecurity, DeepScan AI drops from rank #2 to #5 while Enterprise Shield rises from #3 to #1. In every case IRT recovers the correct ordering. Item parameter recovery is exact: Spearman correlation between true and estimated difficulties is 1.000 in all four domains, and recovered discrimination parameters correctly identify the most diagnostic items (e.g., CoLA in GLUE with $D$9; dense urban night driving with $(S, D)$0). These recovered parameters carry actionable content—regulators can identify which test conditions to mandate, and security buyers which threat types differentiate products.

The failure surface

A 150-condition grid sweep over $(S, D)$1 and $(S, D)$2 (15 seeds per cell, $(S, D)$3, $(S, D)$4, $(S, D)$5) extends the four-domain result into a two-dimensional failure surface. Under difficulty-biased missingness, simple averaging degrades from perfect ranking to $(S, D)$6 at the most extreme corner, while IRT maintains minimum mean $(S, D)$7. An MCAR control degrades only mildly (minimum $(S, D)$8), establishing that severe failure requires the interaction of sparsity, difficulty heterogeneity, and non-random missingness rather than any single factor.

A centered interaction regression on ranking error $(S, D)$9 yields a strongly significant positive interaction coefficient $(\theta^*_j, a^*_i, b^*_i)$0 ($(\theta^*_j, a^*_i, b^*_i)$1), with $(\theta^*_j, a^*_i, b^*_i)$2; the MCAR analogue gives $(\theta^*_j, a^*_i, b^*_i)$3 and less total degradation. At $(\theta^*_j, a^*_i, b^*_i)$4, biased missingness produces mean ranking error of 0.118 versus 0.057 under MCAR—an additional 0.061 attributable purely to the missingness mechanism. A one-dimensional power-law fit explains only $(\theta^*_j, a^*_i, b^*_i)$5, so the author explicitly reframes the "scaling law" as an empirical failure surface rather than a closed-form law—a candid concession that tempers the paper's titular claim.

Sensitivity analysis sharpens the necessary conditions: at full coverage, simple averaging achieves $(\theta^*_j, a^*_i, b^*_i)$6 regardless of difficulty gap; under MCAR it degrades only mildly even at 50% coverage; IRT holds $(\theta^*_j, a^*_i, b^*_i)$7 down to 30% coverage under extreme biased missingness, subject to the overlap constraints noted above.

Mechanism

The analytical decomposition makes the bias direction explicit. The expected simple average for system $(\theta^*_j, a^*_i, b^*_i)$8 is a mean of $(\theta^*_j, a^*_i, b^*_i)$9 over its observed item set $M$0; when $M$1 is difficulty-biased, the expectation is biased regardless of true ability. A worked example shows two systems separated by 2.5 ability units producing identical expected averages (~0.73) because one was tested on hard items and the other on easy ones. IRT succeeds because the likelihood couples all systems and items jointly: an item's difficulty is calibrated from other systems' responses, so a high-ability system's failures on hard items are attributed to item difficulty rather than low ability. The separability of $M$2 in the 2PL form places all abilities on a common scale irrespective of which items were observed.

Implications for Physical AI benchmarking

The paper argues that Physical AI (robotics/VLA) evaluation exhibits the paper's failure conditions in their most extreme form. Published evaluation tables are highly incomplete—the author estimates coverage well below 50%—and LIBERO-PRO reported that models exceeding 90% accuracy on standard LIBERO tasks collapse to 0% under perturbation, indicating extreme effective difficulty gaps. Because cross-paper comparisons aggregate sparse, non-overlapping results by simple average, the scaling law predicts substantially degraded rankings in exactly this regime. The extrapolation is explicitly flagged as such: it rests on the simulation-calibrated law plus estimates of real-world coverage, not on episode-level empirical validation.

Limitations and open questions

The paper is transparent about scope. All non-NLP experiments use synthetic data generated from the 2PL model itself; the author correctly characterizes this as a well-specified simulation study comparing estimators on identical observations, not proof that real benchmarks satisfy 2PL assumptions. Real-world data may violate unidimensionality and local independence, though moderate violations are known to be tolerable and 3PL or multidimensional extensions exist. The functional form of the failure surface remains unresolved—the power-law approximation fits poorly, and the precise closed-form relationship between $M$3 and ranking error is left open. Sample sizes are modest (8–12 systems, 4–8 items), consistent with current Physical AI ecosystems but untested at scales of hundreds of systems. Finally, stable IRT estimation requires sufficient overlap (each system on ≥2 items, each item on ≥3 systems), and missing-not-at-random patterns violating the response model may still bias estimates. The most consequential open question is whether episode-level Physical AI data—binary success/failure outcomes per trial per scenario—exhibit rankings consistent with IRT correction in practice; the paper identifies this as essential future work rather than claiming it as a result.

Conclusion

This paper provides a controlled, reproducible demonstration that simple averaging fails predictably—and sometimes catastrophically—as a joint function of data sparsity, item difficulty heterogeneity, and non-random missingness, while 2PL IRT recovers ground-truth rankings with $M$4 across all tested conditions. The practical prescription is modest and actionable: compute coverage and difficulty gap, use IRT when either is unfavorable, and report both rankings side by side. The mathematical machinery dates to Lord and Novick (1968) and implementations such as py-irt are freely available; what remains, per the author's own account, is episode-level data infrastructure and empirical validation in Physical AI and other sparse evaluation ecosystems.