Identifiable fully explanatory IRT model with domain-varying structural effects

Develop and identify a fully explanatory Item Response Theory model that propagates item-difficulty uncertainty into the domain contrast while natively absorbing subject clustering.

Background

The paper notes that the ideal specification would jointly model item difficulty as a function of structural indicators, account for uncertainty in estimated difficulty, allow domain-varying effects, and model the dependence induced by the 57 MMLU subject clusters. The authors attempted such an explanatory IRT model, but its free per-item residual scale absorbed much of the variation attributed to the indicators, its uncertainty estimates were unreliable, and interaction signs changed with increased optimization budgets. Consequently, the model was weakly identified and the desired integrated specification remains unresolved.

References

The ideal specification would be a fully explanatory IRT model propagating difficulty uncertainty into the domain contrast while natively absorbing subject clustering. We were unable to make that model identify, and Appendix~\ref{sec:mirt} documents the attempt as a formal negative result.

— What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores  (2609.09372 - Paquin et al., 8 Sep 2026) in Section Limitations, subsection “Statistical and methodological bounds”; Appendix A, Section “Confirmatory Two-Dimensional IRT”