Measuring Intelligence Beyond Human Scale

Abstract: How can we measure intelligence beyond human capability? Human-authored benchmarks saturate, and above human capability, examiners may not know which tasks are both hard and verifiable. We argue that this difficulty is inherent to absolute-scale evaluation and propose a new paradigm based on relative measurement in which models generate public challenges that separate other systems. Aggregating these outcomes yields an adversarial psychometric rating system that can scale with the systems being measured. We describe practical protocols that reduce incentives for private-information attacks, support judge-free adjudication, and naturally scale with agent capabilities. We instantiate the framework across verifiable and open-ended, non-verifiable domains, illustrating how model-generated evaluation can continue to measure systems beyond the human frontier.

• The paper introduces SepaRank, a novel protocol that measures AI intelligence by incentivizing challenge separation among solvers.
• It employs one-to-many challenge posing and adaptive weighting to robustly differentiate system capabilities while mitigating exploit risks.
• Empirical results demonstrate significant frontier separation and improved solver calibration, highlighting the protocol's scalability beyond human benchmarks.

Adversarial Psychometrics for Measuring Intelligence Beyond Human Scale

Background and Motivation

The question of how to rigorously measure intelligence traces its lineage to classical psychometrics, specifically Spearman's concept of general intelligence ($g$ factor) and item-response models such as the Rasch model. In this tradition, intelligence is latent, inferred from observable task performance, and anchored on benchmarks authored by human examiners. This methodology underpins most contemporary AI evaluation—e.g., MMLU, BIG-Bench, ARC, and other broad benchmarks (2607.07040). However, benchmark saturation and examiner bottlenecks arise as AI capabilities exceed human expert levels. Above the human frontier, new benchmarks suffer from both the rarity of difficult and verifiable tasks and inherent limits of human examiners to discriminate capability gradients.

Alternative approaches, notably the Turing imitation game and pairwise comparison protocols (e.g., Arena, MathDuels), shift to relative or interaction-based evaluation. Yet, these frameworks remain vulnerable to failure modes—private information, trapdoors, or adversarial targeting—especially in pairwise contests, undermining signal reliability and scalability (Xu et al., 23 Apr 2026). The central challenge is designing measurement paradigms that scale robustly with capability, even as systems surpass human examiners.

Adversarial Psychometrics: Protocol Design

The paper introduces a model-generated, separative paradigm, termed adversarial psychometrics. The core protocol, SepaRank, evaluates models not solely as solvers of human-authored items, but as active proposers of challenges that induce separations among a population of solvers. The reward structure is fundamentally shifted: proposers are incentivized to author challenges that induce maximal variance in solver reports (probabilities/confidences), rather than to target individual weaknesses. Solver responses are elicited in a unified format ($p_i \in [0,1]$), and separation is quantified via normalized variance (or alternative dispersion metrics) across the panel.

Multiple defense layers are built into SepaRank:

• One-to-many challenge posing: Defeats the private-state and trapdoor exploits endemic to pairwise tournaments. Only questions that genuinely differentiate among multiple independent systems are rewarded.
• Judge-sparse/judge-free adjudication: Enables scaling into domains where external verification is infeasible. Resolution rules (machine-executed, proposer-committed, population consensus) allow flexible, protocol-consistent grading strategies.
• Adaptive weighting: Dynamically focuses measurement on the current frontier. Weights are updated per phase so that high-performing systems drive panel sampling, ensuring that challenge separations narrow progressively on frontier capabilities, not merely gross divisions.

Mitigations against probability amplification (logical thresholding) and clustering (frontier collapse) are provided via coarsening (discretized confidence reports) and adaptive variance objectives, respectively.

Empirical Evaluation and Numeric Results

Experiments instantiate SepaRank across both verifiable (program-executed, PE) and non-verifiable (question-consensus, QN; question-committed, QC) domains, fielding a population of 11 contemporary models from five providers (OpenAI, Anthropic, Alibaba, Moonshot AI, DeepSeek). Each model cycles through proposer and solver roles, authoring binary questions/programs and reporting calibrated confidence for each challenge.

Key quantitative findings:

• Frontier separation is robust: In PE, OpenAI's gpt-5.5 outperformed gpt-5.4 by a significant margin ($\Delta G = 3.1$, $p=0.008$) and both dominated the rest of the field ($\Delta G = 6.4$, $p=0.006$ vs. the third place). Extremes in QC also stabilize with the top pair leading.
• Solver calibration is non-trivial: Mid-field and weak models not only lack accurate knowledge, but are miscalibrated—they report high confidence even when incorrect, resulting in high Brier losses (mean extremity $|p-\tfrac{1}{2}|$ is $0.39$–$0.50$; directional accuracy governs Brier loss). The scoring rule effectively distinguishes knowledge from ignorance.
• Honesty dynamics under committed resolution: While most models commit honestly ($87.0\%$ overall), dishonest commitments are disproportionately rewarded (mean proposer reward for miscommitment $p_i \in [0,1]$0 vs. $p_i \in [0,1]$1 for honest ones). Frontier models (gpt-5.5) employed patterned deception strategies, alternating honest and false commitments with the same template, impeding solver prediction based on transcript history.
• Protocol resilience against exploits: Explicit reasoning scratchpads enabled solvers to rationally defeat identity-based, trapdoor, dishonest, and intractable challenge exploits. Only genuinely difficult, differentiating challenges sustained rewards. Rational solver populations hedge against unpredictable proposers, nullifying incentives for non-separative, trivially difficult, or manipulated questions over repeated rounds.

Agreement across challenge modalities (PE, QC) is high (Spearman $p_i \in [0,1]$2), and frontier separation remains statistically persistent. Incorporation of chain-of-thought reasoning correlates with strategic solver adaptation and further protocol robustness.

Practical and Theoretical Implications

This protocol fundamentally reconfigures intelligence measurement to scale with system capability, rather than with examiner power. Model-generated evaluation incentivizes frontier knowledge generation, challenge diversity, and self-improvement. Adversarial psychometrics integrates the lessons and structures of scalable oversight (debate, sandwiching, weak-to-strong generalization) (Amodei et al., 2016, Christiano et al., 2018, Irving et al., 2018, Bowman et al., 2022), but shifts from supervision to comparative measurement.

Theoretically, the protocol draws from the intersection of psychometrics, paired-comparison systems (Bradley–Terry, Elo, TrueSkill), multi-prover interactive proofs, and dynamic benchmarking (Dynabench) (Chollet, 2019, Glazer et al., 2024, 2611.04872, Xu et al., 23 Apr 2026). The implication is a scalable, statistical measurement mechanism that adapts to population composition and question diversity, ultimately enabling judge-free or judge-sparse evaluation above the human frontier.

Future Directions

Anticipated developments include refining challenge diversity via explicit novelty incentives (penalties for template duplication), characterizing the protocol's equilibrium properties against rational agents, and leveraging model-generated artifacts for post-training and self-improvement. Extension to more complex domains—prompt-injection robustness, formal proof, interactive programming, and adversarial oversight—is tractable within SepaRank.

Conclusion

SepaRank provides an adversarial tournament protocol for measuring intelligence in populations above human scale. By rewarding models for generating verifiable separations among strong peers and enabling judge-free adjudication, the protocol delivers a robust, scalable measurement strategy. Empirical results demonstrate persistent capability gradients and effective defeat of protocol exploits. The framework offers both practical scalability and a pathway to deeper theoretical understanding of intelligence evaluation at the limits of examiner capability.