Construct validity of QA benchmarks as measures of understanding

Establish whether performance on question‑answering benchmarks such as MMLU validly measures a model’s underlying understanding rather than superficial behavior, and characterize the conditions under which QA performance serves as a reliable indicator of understanding.

Background

Benchmarks like MMLU are widely used to evaluate models’ knowledge and understanding via QA tasks, but their construct validity—whether they truly capture understanding—remains uncertain.

Clarifying this validity is essential for using benchmark scores to inform governance decisions, deployment risk assessments, and claims about model capabilities.

References

For example, while MMLU claims to assess a model's understanding and memorization of knowledge from pretraining through the proxy of performance on question answering, it is unclear how well the ability to accurately answer questions serves as an indicator for understanding.

— Open Problems in Technical AI Governance  (2407.14981 - Reuel et al., 2024) in Section 3.4.1 “Downstream Impact Evaluations”

Nor does the paper establish A1 where it matters most for the observational half. The QA gains rest on the question--answer overlap block (Section~\ref{sec:samelabel}), and a sentence that reuses the question's words is a plausible carrier of relevance as well as of surface form. The rule-based intervention of Appendix~\ref{app:c2} shows the raw score following manipulated overlap; it does not show that overlap is construct-external, so the QA positives remain conditional on A1 for that block.

— When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits  (2609.24194 - Weon et al., 21 Sep 2026) in Section 9, subsection “What we have not established”

Our benchmarks share a common synthetic construction procedure, and validation on independently collected biomedical QA remains future work.

— AdaPath: Query-Adaptive Path-Finding via Path-Bank for Multi-Hop Implicit Biomedical KGQA  (2608.30556 - Kim et al., 31 Aug 2026) in Limitations section