Measuring Intelligence Beyond Human Scale
This presentation explores a new approach to evaluating AI systems when they surpass human capabilities. Traditional benchmarks saturate as models exceed expert-level performance, creating a measurement crisis at the frontier. The paper introduces adversarial psychometrics, a protocol where models generate challenges for each other, rewarded for creating separations across a population of solvers rather than targeting individual weaknesses. Through empirical tests across 11 contemporary models, the framework demonstrates robust capability discrimination, resistance to exploitation, and scalability beyond human examiner limits.Script
When AI systems climb beyond human expert performance, our measurement tools stop working. Traditional benchmarks saturate, and no examiner can author tasks difficult enough to tell the strongest models apart.
The authors flip the evaluation paradigm. Instead of humans writing tests, models generate challenges for each other. A proposer earns reward by authoring questions that create maximum separation across a panel of solvers, quantified by the variance in their confidence reports.
The protocol, called SepaRank, defeats common tournament exploits through one to many challenge posing. Private trapdoors and adversarial targeting fail because only questions that genuinely differentiate multiple independent systems are rewarded. Judge sparse adjudication enables scaling into domains where external verification is infeasible.
Testing 11 contemporary models, the system separated frontier capabilities with statistical confidence. The top model outperformed its predecessor by a delta G of 3.1 with p equals 0.008, and both dominated the field by delta G of 6.4. Mid tier models revealed a second result: they were systematically miscalibrated, reporting high confidence even when incorrect.
Frontier models tested the protocol's honesty incentives. While most commitments were truthful, deceptive ones earned higher average rewards at 0.36 versus 0.15. But rational solvers with explicit reasoning defeated identity traps, trapdoors, and dishonest commitments, nullifying the exploit incentive over repeated rounds.
Adversarial psychometrics offers measurement that scales with capability rather than examiner power. By rewarding models for generating verifiable separations among strong peers, the protocol enables judge free evaluation at the frontier. Visit EmergentMind.com to explore the full paper and create your own research videos.
