Construct validity of capability benchmarks (e.g., MMLU)

Evaluate and establish the construct validity of widely used capability benchmarks, such as determining whether performance on question answering (e.g., MMLU) genuinely indicates model understanding, and develop methodologies for validating benchmark-to-construct mappings.

Background

Benchmarks are used as proxies for complex constructs like understanding, yet the link between a score and the targeted construct may be unproven.

Demonstrating construct validity would improve confidence in benchmark-based evaluations and their use in governance decisions.

References

For example, while MMLU claims to assess a model's understanding and memorization of knowledge from pretraining through the proxy of performance on question answering, it is unclear how well the ability to accurately answer questions serves as an indicator for understanding.

— Open Problems in Technical AI Governance  (2407.14981 - Reuel et al., 2024) in Section 3.4.1 Downstream Impact Evaluations

They do not establish semantic correctness beyond the configured checks or equivalence to real-world defects. Mutation adequacy was measured using three AST-level mutation operators, and difficulty denotes a structural proxy whose relationship to APR-system success has not yet been validated.

— Specification-Driven Benchmarking for Automated Program Repair From Static Corpora to Executable Specifications  (2609.28896 - Ebrahim, 24 Sep 2026) in Section 6.6, page 8

Whether greCAPTCHA offers valid indicators of actual human authorship remains an empirical question.

— greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AI  (2609.20481 - Payan et al., 17 Sep 2026) in Section 1, Introduction, subsection “Assessment for Research Authorship”

This fragmentation limits reliability and scalability: papers and model releases typically report only a small subset of popular benchmarks, so it is unclear whether progress generalizes across domains or concentrates on a few well-known tasks, and comparisons are confounded by differences in benchmark implementations, environment setups, and evaluation protocols.

— Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation  (2609.04298 - Shi et al., 3 Sep 2026) in Section 1, Introduction

Task-capability classification also remains unvalidated in this rebuild. The deterministic null extractor used in CI assigns no capability levels to its 5,863 discovery-derived tracks. Completing and evaluating those labels is a separate task from maintaining the benchmark catalog.

— Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation  (2609.11115 - Wu et al., 10 Sep 2026) in Section 6, “Limitations and Future Work,” paragraph 5

We do not test whether any real benchmark has that property; it is a separate measurement question, which \citet{hardy2026cartography} approach through factor analysis of leaderboard data.

— The AI Evaluation Ecosystem  (2610.09296 - Dave et al., 7 Oct 2026) in Section 4, subsection Holdout design: private holdouts shrink the gap unevenly across benchmarks

Hence, it is unclear whether reported gains reflect genuine improvements in capability or differences in evaluation setup.

— Meta-Video: Efficient Standardized Evaluation for Video Understanding  (2610.18535 - Jahagirdar et al., 13 Oct 2026) in Abstract, page 1

Priors rooted in specialized domains or in rapidly evolving knowledge fall outside this taxonomy, and whether the patterns of prior adherence we observe extend to such cases remains an open question for future benchmarks to address.

— Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs  (2610.21902 - Ma et al., 15 Oct 2026) in Limitations section