When the Judges Are Wrong: A Validity Audit of Tool-Calling Benchmarks
This presentation examines a critical flaw in how we measure language-model agents. The authors audited four widely used tool-calling benchmarks and discovered that nearly one in five official verdicts disagrees with expert human judgment. Through 89 hours of detailed trace inspection and 23 repeated benchmark runs, the paper reveals that leaderboard scores often reflect evaluator artifacts rather than true agent capability, and proposes a deterministic-first evaluation protocol that achieves 95.5% agreement with human assessments.Script
The benchmarks we use to rank language-model agents disagree with expert human judgment nearly one in five times. That means leaderboard positions may tell us more about how the tests are broken than how good the models actually are.
The authors recruited expert annotators to review 496 complete agent trajectories, spending nearly 11 minutes per task inspecting every tool call, output, state change, and final response. Across four benchmark families, 92 official labels contradicted the expert consensus.
One striking false negative reveals how brittle checks undermine validity. An agent correctly cancels two hotel reservations and reports the remaining cost as 708 dollars, matching the database perfectly, but the benchmark fails the run because the response lacks the substring one six two eight, a reference value that incorrectly includes the reservations the user just asked to cancel.
When the authors ran the exact same 95 tasks 23 times under identical settings, scores ranged from 57.9 percent to 76.8 percent, a spread of nearly 19 percentage points caused purely by pipeline randomness. A single run cannot support meaningful model comparisons when the noise is larger than typical leaderboard gaps.
Tool Veritas, the authors' proposed evaluator, enforces deterministic state checks first and invokes a language model judge only for genuinely qualitative criteria like communication quality. This design achieved 95.5 percent agreement with human experts, with zero false positives, all disagreements erring toward rejecting borderline cases rather than crediting unearned passes.
The broader lesson is concrete. Benchmark evaluators should be reproducible, inspectable, and validated against human judgment before their scores are treated as measurements of progress. If you want to explore how validity-first evaluation changes what we know about agent capability, visit EmergentMind.com to learn more and create your own video.
