DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

Abstract: Current LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with objective correctness labels and arena-style preference data that better reflect open-ended user interactions. We introduce DualEval, a latent model-item calibration framework that represents models and evaluation items in a shared space, jointly estimating model ability together with item difficulty and sharpness. We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries. Our evaluation uses 18 frontier LLMs, static benchmark labels, and reward-model scores validated against held-out human preferences for open-ended model responses. Empirically, our framework produces reliable and balanced model rankings, and its learned item-level profiles support downstream applications such as benchmark compression for sample-efficient evaluation and anomaly detection for contamination or outlier analysis. Overall, DualEval unifies static and arena-style evaluation through joint model-item calibration, producing model rankings and item-level diagnostics that support more sample-efficient, interpretable, and auditable evaluation pipelines.

• The paper introduces DualEval, a joint item response theory framework that calibrates model abilities with item difficulty and sharpness across static correctness and arena preference data.
• DualEval preserves strong static ranking agreement with Spearman correlations of 0.959–0.990, improves robustness to noisy reward models, and reduces ranking variance by 30–60% compared with static-only calibration.
• Fisher-information selection recovers full-data rankings with the top 10% of items at ρ ≥ 0.95, while residual analysis detects synthetic contamination with AUROC up to 0.995 under clean-reference evaluation.

DualEval is a latent-variable evaluation framework that unifies two otherwise disconnected modes of LLM assessment—static benchmark correctness and arena-style preference data—through joint model-item calibration. Rather than aggregating item-level outcomes into a single model score, the framework treats each evaluation item as a measurement instrument with its own difficulty and sharpness, placing model abilities and item properties on a shared latent scale. The result is both a model ranking mechanism and an item-level diagnostic toolkit for benchmark maintenance, contamination auditing, and sample-efficient evaluation.

Motivation and positioning

The paper's starting point is a structural weakness in current evaluation practice: static benchmarks (e.g., MMLU-style suites, coding harnesses) provide clean ground-truth labels but saturate and suffer contamination as models improve, while arena-style Bradley–Terry rankings over human preferences capture open-ended quality but involve annotator subjectivity and evaluator bias. Existing integrations of the two remain weak, and both are largely model-centric—treating all items as equally informative. DualEval's premise, borrowed from Item Response Theory (IRT), is that ranking signal depends on model-item interaction: items that are uniformly solved or uniformly failed carry no separation, while items where small ability differences produce detectable outcome differences carry the most. Prior IRT applications to LLM evaluation operate within a single evaluation source, typically correctness labels; DualEval's distinguishing move is coupling static correctness with reward-model–distilled open-ended preferences in one shared latent space.

Method

Each model $i$ receives an ability $\theta_i \in \mathbb{R}$; each item $q$ receives a difficulty $b_q$ and a sharpness $a_q = \exp(k_q) > 0$. For static benchmarks, the framework uses a two-parameter logistic (2PL) IRT model, $p_{i,q} = \sigma(a_q(\theta_i - b_q))$, fit by binary cross-entropy on correctness labels.

For open-ended arena data, a scalar reward model scores each response; rewards are globally standardized to $z_{i,q}$ and converted into soft pairwise preference targets $p^{*}_{ijq} = \sigma(z_{i,q} - z_{j,q})$. The model's predicted pairwise preference is $$\hat P(i \succ j \mid q) = \sigma(\gamma(p_{i,q} - p_{j,q}))$$ with a learned, L2-regularized arena temperature $\gamma$. Two pair-treatment mechanisms filter the arena loss: tie filtering removes pairs whose standardized reward gap falls below a configured percentile, and both-bad anchoring identifies pairs where both models receive low rewards, treating them as absolute failure evidence via a term pushing both models' success probabilities down. An ablation shows both-bad anchoring is the dominant component: removing the loss drops both-bad AUC from 0.903 to 0.622 while leaving held-out pair accuracy near 0.78. The full objective combines static and arena losses with L2 regularization, and translation invariance is fixed post-step by zero-centering abilities and absorbing the shift into difficulties.

Experimental setup

The evaluation spans four domains—coding, math, miscellaneous domain knowledge, and generic everyday queries—over 18 frontier LLMs from seven providers. The first three domains combine static benchmarks (LiveCodeBench v6, MBPP-Plus, SWE-Bench-lite, TerminalBench-2.0; AIME 2025/2026, HLE-Math, Olympiad-Math; HLE subfields and SimpleQA) with reward-scored open-ended arena prompts filtered by LLM judges; the generic domain is arena-only. Arena supervision primarily uses a proprietary scalar reward model (Qwen-3-32B backbone trained on over five million human-preference pairs with a Bradley–Terry objective), with the public Skywork-Reward-V2-Qwen3-8B used for robustness replication. Each (model, question) pair uses a single generated response under a consistent inference setup, with fixed agent harnesses (Terminus2, mini-SWE-agent v2) for agentic coding tasks.

Fit quality and ranking integration

DualEval reconstructs both signal types: 88–92% binary accuracy on static labels in the three static-anchored domains and 68–81% decisive-pair agreement on arena comparisons, with held-out expected calibration error of 0.04–0.07 on static cells and 0.01–0.03 on soft arena targets. The arena agreement is comparable to the reward model's own agreement with held-out human preferences, indicating the latent structure captures genuine preference signal rather than noise.

The integration results expose a source-specialization tradeoff. Static-only 2PL aligns perfectly with the static reference but transfers weakly to arena rankings; arena-only Bradley–Terry shows the reverse pattern. DualEval preserves nearly the static-only model's static-reference agreement (Spearman $\theta_i \in \mathbb{R}$0 of 0.959–0.990 across the three domains) while improving arena-reference agreement over static-only training. The most consequential result concerns reward-model robustness: substituting the public Skywork RM collapses arena-only rankings' agreement with public sub-leaderboards to $\theta_i \in \mathbb{R}$1 for coding/math/misc, whereas DualEval with the same noisy RM retains Static $\theta_i \in \mathbb{R}$2 and cross-RM ranking agreement of 0.963/0.992/0.835. The arena-only generic domain provides the natural counterfactual—without static anchors, proprietary- and public-RM rankings diverge sharply ($\theta_i \in \mathbb{R}$3). This isolates static anchoring as the mechanism stabilizing the latent scale under noisy preference supervision. Bootstrap analysis (question-level cluster bootstrap, $\theta_i \in \mathbb{R}$4) shows the joint fit reduces ranking variance by 30–60% relative to Static 2PL, and on the arena-only generic domain it matches or slightly improves Arena BT's uncertainty, indicating the IRT structure adds discriminative information without inflating variance.

Item informativeness and benchmark compression

DualEval scores each item by its Fisher information about the model ability vector—$\theta_i \in \mathbb{R}$5 for static items, and an analogous trace over ability parameters for arena items, averaged over decisive pairs to prevent an artificial $\theta_i \in \mathbb{R}$6 advantage. Subset-recovery experiments refit abilities on the top-selected fraction of items and measure Spearman correlation with the full-data ranking. The headline result: the top 10% of items by Fisher information recover the full-data ranking at $\theta_i \in \mathbb{R}$7 in all three static-anchored domains, and the top 30% at $\theta_i \in \mathbb{R}$8. Ablations show sharpness explains much of the signal, with the non-saturation weight adding consistency at small fractions; difficulty matching alone is unreliable; and a single-source 2PL+BT baseline trails throughout, suggesting jointly fit item parameters carry information absent from disjoint-source rankings. Learned difficulty–sharpness structure differs by modality—static items show near-zero difficulty–sharpness correlation, while arena items exhibit $\theta_i \in \mathbb{R}$9 positive coupling—confirming the parameters reflect underlying outcome structure. The practical implication is direct: leaderboard refreshes can target the high-Fisher subset at a fraction of the cost, and low-information items are principled retirement candidates. The authors note the caveat that informativeness is pool-relative, so high-signal subsets must be recalibrated as model abilities shift.

Residual-based anomaly detection

The joint fit's per-pair expectations enable standardized residuals—unexpected successes by weak models on hard items are natural contamination candidates. In a synthetic injection protocol (label flips on low-probability static cells; reward inflation on bottom-30% arena cells, at rates of 0.5–5%), a clean-reference detector recovers static-label contamination with AUROC $q$0 and AUPRC 0.80–0.99, and arena-reward contamination with AUROC 0.93–0.98. A contaminated-fit detector that refits on corrupted data remains above chance but is less sensitive, because flipped labels are absorbed into the refitted latent parameters. This gap argues operationally for preserving vetted historical benchmark snapshots against which new submissions are screened. Qualitative anomaly candidates on unperturbed data—weak models passing widely archived multiple-choice sequence puzzles or named MBPP function signatures—are flagged as consistent with prior exposure, though the authors explicitly caution these are candidates, not verified contamination; large residuals can also reflect MC guessing, judge artifacts, or narrow genuine capability, and perturbation-based audits remain future work.

Limitations

The paper is candid about three constraints. First, the primary arena supervision depends on a proprietary, non-replicable reward model; validation against held-out human preferences and the Skywork replication mitigate this, but broader validation across multiple public reward models and direct human annotations is outstanding. Second, single-response evaluation per (model, question) pair cannot separate model capability from sampling variability or decoding-parameter sensitivity. Third, the scalar per-domain ability parameter cannot represent multi-skill specialization; multidimensional IRT is identified as the natural extension. An additional implicit assumption is that reward-model scores constitute valid preference proxies—the framework's robustness to RM noise is demonstrated only when static anchors are present.

Conclusion

DualEval reframes LLM evaluation as joint model-item calibration, demonstrating that a shared latent IRT scale can integrate static correctness and reward-distilled preferences into rankings that are more balanced and more robust to reward-model noise than either source alone, while the learned item profiles support Fisher-based benchmark compression (full ranking recovery from 10% of items) and residual-based contamination screening with near-perfect AUROC under clean-reference conditions. The open questions it leaves are concrete: validation across multiple public reward models with direct human annotation, extension to repeated responses to disentangle capability from stochasticity, and multidimensional ability structures that would let item selection adapt to specific target capabilities rather than overall model strength.