On the Measure of a Model: From Intelligence to Generality

Abstract: Benchmarks such as ARC, Raven-inspired tests, and the Blackbird Task are widely used to evaluate the intelligence of LLMs. Yet, the concept of intelligence remains elusive- lacking a stable definition and failing to predict performance on practical tasks such as question answering, summarization, or coding. Optimizing for such benchmarks risks misaligning evaluation with real-world utility. Our perspective is that evaluation should be grounded in generality rather than abstract notions of intelligence. We identify three assumptions that often underpin intelligence-focused evaluation: generality, stability, and realism. Through conceptual and formal analysis, we show that only generality withstands conceptual and empirical scrutiny. Intelligence is not what enables generality; generality is best understood as a multitask learning problem that directly links evaluation to measurable performance breadth and reliability. This perspective reframes how progress in AI should be assessed and proposes generality as a more stable foundation for evaluating capability across diverse and evolving tasks.

• The paper challenges conventional intelligence benchmarks and establishes generality—defined as multitask performance breadth—as a more reliable measure.
• It derives formal generalization bounds from multitask learning theory, demonstrating a reduction in estimation error with increased task diversity.
• The proposed framework highlights actionable implications for designing dynamic, diverse, and evolving evaluation benchmarks for AI models.

Reconsidering Model Evaluation: From Intelligence to Generality

Introduction

"On the Measure of a Model: From Intelligence to Generality" (2511.11773) offers a theoretically rigorous and empirically informed critique of conventional model evaluation in AI, particularly as it pertains to LLMs. The authors scrutinize the prevailing use of "intelligence" as an evaluative construct for AI systems, challenge the foundation of current intelligence benchmarks, and defend a paradigm shift towards grounding evaluation in generality—defined formally as multitask performance breadth and reliability. Through conceptual analysis and multitask learning theory, the paper argues that only generality is a stable and sufficient foundation for model evaluation, while assumptions about stability and realism are either unjustified or actively misleading.

Critique of Intelligence-Based Evaluation

The dominant trend in LLM benchmarking relies on abstract, human-inspired notions of intelligence, operationalized via tests such as ARC, Raven-style challenges, and Blackbird. These benchmarks implicitly assume that high performance indicates a real, unitary cognitive capacity. The authors highlight that the definition of intelligence is highly contested and vague across philosophy, neuroscience, and cognitive science. Attempts to correlate intelligence with neural substrates or to encapsulate it in a stable, measurable property have failed to reach consensus or predictive utility.

Empirically, the correlation between performance on 'intelligence' benchmarks and real-world task efficacy is weak. The paper provides evidence that models excelling in such tests do not consistently outperform others on benchmarks more directly aligned with human utility or preference. This performance dissociation is illustrated in the divergence between scores on intelligence-oriented AGI benchmarks and those elicited on task-specific or human preference-aligned benchmarks.

Figure 1: The performance of LLMs on task-specific benchmarks OpenBookQA, Entity Extraction, and StackUnseen.

Figure 1 demonstrates that performance on intelligence-focused evaluations is not a reliable proxy for robust, universal competence in practical contexts. This calls into question the foundational role assigned to intelligence benchmarks in current evaluation protocols.

Analysis of Underlying Evaluation Assumptions

Through formalization, the paper identifies and decouples three core assumptions underlying intelligence-based evaluation:

• Generality: Models should be measured on their ability to perform well across diverse tasks.
• Stability: A fixed set of tasks can adequately and persistently indicate intelligence or capability.
• Realism: Intelligence is a latent, real property, and observable performance is explained by possession of this trait.

The authors show that only generality is logically necessary and empirically supported for coherent evaluation. Stability presupposes an unchanging task set, which is incompatible with deployment in dynamic environments and is vulnerable to saturation and obsolescence. Realism introduces untestable metaphysical commitments and does not enhance predictive or diagnostic power for model capabilities.

Through a formal framework modeling models as task-performance mappings $f_M : \mathcal{T} \to [0,1]$, the paper demonstrates that generality corresponds to expected performance under a stochastic task environment, requiring neither a fixed task set nor a latent intelligence variable. The associated agent-characteristic curve $\psi_M(h)$ and generality metric $\Gamma_M$ (inverse ACC spread) robustly quantify multitask capability in a way that is invariant to benchmark composition and meaningful under distributional shift.

Theoretical Justification via Multitask Learning

The generality principle aligns with and is substantiated by multitask learning (MTL) theory. The authors derive and prove generalization error bounds demonstrating that empirical evaluation over $n$ independent tasks reduces estimation variance by a factor of $\sqrt{n}$ compared to single-task settings. Formally, the generalization gap decays as $O(1/\sqrt{nm})$, reinforcing that evaluation across diverse tasks yields more stable and predictive assessments of real-world competence. This is a central justification for operationalizing evaluation in terms of generality rather than intelligence.

Moreover, multitask learning frameworks in both theory [baxter2000model, maurer2006bounds] and practice provide strong evidence that breadth (not depth on a singular task) is the superior predictor of generalization and transfer.

Implications: Evaluation Design, Practice, and Future Directions

Grounding evaluation in generality has immediate ramifications for benchmark design and interpretation. Static benchmarks rapidly become less informative as models saturate their task sets, and reliance on high-level cognitive constructs introduces subjectivity and hinders progress. Evaluation frameworks must instead select representative, diverse, and evolving task distributions, and aggregate model performance using metrics that emphasize both mean proficiency and low variance across tasks.

This theoretical foundation directly addresses the perceived "evaluation crisis" in NLP and AI benchmarking [bowman2021will, hofmann2025fluid]. Conflation of narrow indicator tasks with general competence can no longer be justified. Future research will need to focus on robust sampling of tasks, quantification of task similarity, detection of distribution drift, and design of evaluation protocols that remain relevant as both tasks and models evolve.

Additionally, the paper's framework underpins arguments against the prioritization of AGI or intelligence as research goals, in favor of practical generalization and adaptability. As models are increasingly deployed in non-stationary, multi-modal, and unconstrained environments, generality is the only stable predictor of utility and safety.

Conclusion

This work provides a systematic and formally supported critique of intelligence-centric evaluation and offers a compelling, actionable alternative: the adoption of generality as the primary, principled measure of model capability. The implications are substantial for benchmarking practice, interpretability of progress, and the strategic direction of AI research. Future advances in AI will be best measured not by ambiguous proxies of intelligence, but by quantifiable and transferable generality across diverse and shifting task distributions.

Reference: “On the Measure of a Model: From Intelligence to Generality” (2511.11773)