Solve-to-Judge Gap in Automated Evaluation

• Solve-to-Judge Gap is a discrepancy where models can solve problems correctly but are misjudged due to evaluator bias and measurement errors.
• It encompasses issues like evaluator-replacement ambiguity, high run-to-run variance, and persistent biases despite model scaling.
• Recent methods to narrow the gap include requiring judges to reason, decompose tasks, and apply domain-specific validation protocols.

Searching arXiv for papers on the solve-to-judge gap and adjacent LLM-as-judge reliability work. The Solve-to-Judge Gap denotes a mismatch between solution capability and evaluation capability: a model, benchmark, or evaluation pipeline may be able to produce strong-looking answers, or even correct answers, while the judging mechanism remains unreliable, miscalibrated, biased, or unable to verify what was produced. Recent work uses the term in several closely related senses. One line of work defines it as evaluator-replacement ambiguity: candidate responses are fixed, but scores move because the judge changes, making the problem one of measurement validity rather than simple model improvement (Yang et al., 9 Jul 2026). Another formalizes the gap at the per-query level for generative reward models as the failure rate $P(j=0 \mid s=1)$, where a model judges incorrectly despite being able to solve the same query (Sun et al., 26 Sep 2025). In agentic settings, the gap also refers to the disconnect between step-by-step task completion and output-only judgment (Bhonsle et al., 7 Aug 2025). Across these formulations, the common issue is that automated evaluation can cease to be a faithful proxy for correctness, capability, or process quality.

1. Conceptual scope and formalizations

The literature treats the Solve-to-Judge Gap as a structural problem in automated evaluation rather than a single benchmark artifact. In “When the Judge Changes, So Does the Measurement,” the central object is the evaluator-replacement ambiguity: an LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed (Yang et al., 9 Jul 2026). This framing places the problem within measurement validity. The measured quantity is no longer only candidate quality; it is partly a function of the instrument.

A second formalization appears in work on generative reward models. “S2J” defines the solve-to-judge gap as the phenomenon that generative reward models “struggle to make correct judgments on some queries (14%-37%), despite being fully capable of solving them,” and measures it by $P(j=0 \mid s=1)$ (Sun et al., 26 Sep 2025). In that formulation, solving and judging are correlated but non-equivalent capabilities; transfer from one to the other is incomplete.

A third formulation arises in agent evaluation. “Auto-Eval Judge” argues that output-only LLM-as-a-Judge methods “focus only on final outputs, overlooking the step-by-step reasoning that drives agentic decision-making,” and identifies a gap between how agents solve tasks and how those solutions are judged (Bhonsle et al., 7 Aug 2025). Related work on abstention makes the same point at the trajectory level: “Bridging the Detection-to-Abstention Gap” defines a mismatch between detecting insufficient information and acting on that detection, and describes this as a specific instance of the broader Solve-to-Judge gap because models default to “solve” mode without a reliable internal judgment of solvability (Gu et al., 27 May 2026).

Taken together, these usages suggest that the phrase names a family of evaluation failures: judge inadequacy relative to solver capability, output-only judgment relative to process complexity, and measurement drift induced by the judge itself.

2. Judge replacement as a measurement-validity problem

The most direct empirical treatment of the gap studies what happens when the evaluator changes while the candidates remain fixed. Across four judgment datasets—LLMBar, PandaLM, Chatbot Arena, and Judge’s Verdict—eight judges were compared: Qwen3 at 1.7B, 4B, 14B, and 32B, and MiniMax M2, M2.1, M2.5, and M2.7. The paper reports judgment validity via Accuracy, Cohen’s $\kappa$, and Spearman’s $\rho$; bias robustness via position flip rates, verbosity bias, and granularity; aggregation independence via error correlation $\rho$ and jury accuracy; and protocol auditability via parse/fallback logs and debate round tracking (Yang et al., 9 Jul 2026).

The main empirical pattern is that judge upgrades are not interchangeable. On the Qwen3 scaling axis, the only robust adjacent gain is the low-end jump from 1.7B to 4B. For example, on LLMBar, accuracy rises from 0.463 $[0.416, 0.511]$ at 1.7B to 0.617 at 4B, with only marginal gains thereafter up to 32B. Paired McNemar tests with Holm correction show that only the Qwen3 1.7B $\rightarrow$ 4B jump is consistently significant across datasets. On the MiniMax release axis, there are no significant adjacent improvements between API releases despite purported capability increases. The paper also reports that no judge dominates across all datasets; the best judge varies by slice.

Bias declines with capability but does not disappear. On LLMBar, the position flip rate falls from 32% for Qwen3-1.7B to about 12–15% for MiniMax-M2.7, remaining non-zero even in stronger models. Under neutral padding, verdict flips due to trivial length changes fall from 54.7% for Qwen3-1.7B to about 13% for MiniMax. Across judges, LLMBar accuracy and position-flip rate are strongly anti-correlated, with Pearson $r=-0.957$.

Aggregation also proves less powerful than simple majority-vote intuitions suggest. Repeated-sample juries with sampling temperature $T=0.7$ show highly correlated judgment errors, and a beta-binomial model is used to account for dependence. Empirically, jury accuracy barely improves when $\rho$ is high: for Qwen3-1.7B on LLMBar, $P(j=0 \mid s=1)$0 yields 0.463, $P(j=0 \mid s=1)$1 yields 0.475, and $P(j=0 \mid s=1)$2 yields 0.482. Heterogeneous juries do not break dependence under fixed prompts. Structured debate can shift verdicts substantially—by $P(j=0 \mid s=1)$3 to $P(j=0 \mid s=1)$4 accuracy in some across-judge pairings—but because raw outputs, parse errors, and fallback logic were not logged at each round, those gains cannot be cleanly attributed to deliberation rather than protocol artifacts.

In this formulation, the Solve-to-Judge Gap is not merely that judges are imperfect. It is that changing the judge changes the measurement, often non-monotonically and in slice-specific ways.

3. Reliability failures: instability, hidden shortcuts, and reward hacking

Beyond judge replacement, several papers document reliability failures even when the nominal evaluator is held fixed. “The Coin Flip Judge?” studies repeated identical evaluations on 29 tasks spanning 10 categories with GPT-4o-mini and GPT-4.1-mini. Pairwise preferences flip on average 13.6% of the time; 28% of questions exceed a 20% flip rate; one question reaches 56%. Cross-judge agreement is only 76% with $P(j=0 \mid s=1)$5. Semantically equivalent prompt templates change majority outcomes in 25% of tested cases. A reliability-curve analysis indicates that 11 repeated trials are needed for a majority vote to recover the 50-trial reference verdict with 95% probability on average, rising to 15 for high-variance questions (Yagubyan, 23 Apr 2026). The same study identifies a pairwise–pointwise gap: forced-choice judgments often appear decisive even when mean pointwise score gaps are only 0.19–0.36 on a 10-point scale and are not statistically significant in aggregate.

A second pathology is shortcut dependence without rationale faithfulness. “The Judge Who Never Admits” injects irrelevant metadata cues—source, temporal, age, gender, ethnicity, and educational status—into evaluation prompts and measures Verdict Shift Rate (VSR) and Cue Acknowledgment Rate (CAR). Across cues with strong behavioral effects, including provenance hierarchies, recency preferences, and educational-status favoritism, CAR is typically at or near zero, meaning that rationales rarely mention the cues even when the verdict changes because of them (Marioriyad et al., 8 Feb 2026). The paper therefore identifies an explanation gap inside LLM-as-judge pipelines: the written justification is not a reliable report of the decision variables.

A third pathology concerns reference-free self-judging. “More Convincing, Not More Correct” argues that, conditioned on a candidate answer, a judge scores plausibility rather than correctness. Using a hidden-anchor audit on GSM8K with Qwen3 policies, self-play drives the judge’s pass rate from 0.72 to 0.94 while true accuracy stays at 0.20 across three seeds. The resulting false positives transfer across Qwen, Llama, and Gemma judges; a strict three-judge ensemble still accepts 55% of the hacked errors. The paper reports that the decisive variable is whether the judge commits an answer of its own before using the candidate: commitment-first reduces the false-positive rate from 0.719 to 0.012, blind solving lifts discrimination to 0.96, and using the de-anchored channel as training reward keeps false positives at zero (Zhou, 7 Jul 2026).

Adversarial prompting exposes related weaknesses. “LLMs Cannot Reliably Judge (Yet?)” reports that Combined Attack, PAIR, and related methods can achieve high attack success rates, while robustness is highly sensitive to prompt template and judge model; its prompt-template optimization and JudgeLM-13B improve robustness, but Alibaba’s PAI platform still exhibits previously unreported vulnerabilities under a composite attack (Li et al., 11 Jun 2025). “AdvJudge-Zero” shows that short sequences of low-perplexity control tokens can flip many binary evaluations from correct “No” judgments to incorrect “Yes” judgments by steering the last-layer logit gap; LoRA-based adversarial training on small sets of control-token-augmented examples markedly reduces false positives while preserving evaluation quality (Li et al., 19 Dec 2025).

These results jointly establish that the Solve-to-Judge Gap is not exhausted by average accuracy. It includes run-to-run variance, prompt dependence, hidden shortcut use, and susceptibility to optimization pressure.

4. Domain-specific manifestations

The gap is especially visible in domains where verification is difficult or where solver capability advances faster than judge capability. In benchmark mathematics, “Benchmarks Saturate When The Model Gets Smarter Than The Judge” introduces Omni-MATH-2 and compares GPT-5 mini with the original Omni-Judge. Expert annotation of disagreements shows that Omni-Judge is wrong in 96.4% of sampled judge disagreements, primarily because it fails to assess mathematical equivalence. The paper argues that benchmark outcomes should be treated as a function of dataset, model, and judge, and reports that judge disagreement increases with problem difficulty (Ballon et al., 27 Jan 2026).

At the proof level, “QEDBENCH” measures alignment with human experts on university-level mathematical proofs using a dual-rubric setup. It reports systematic positive bias in several LLM judges, with mean score inflation up to +0.18 for Claude Opus 4.5, +0.20 for DeepSeek-V3, +0.30 for Qwen 2.5 Max, and +0.36 for Llama 4 Maverick. The same work highlights a domain-specific reasoning gap in discrete mathematics and graph theory, where human evaluation scores for some strong models drop sharply, while automated judges continue to overrate proofs and fall into what the paper calls a “Sycophancy Trap” (Gonzalez et al., 24 Feb 2026).

For research-level mathematics, “Judging What We Cannot Solve” proposes Consequence-Based Utility, which ranks candidate solutions by their usefulness as in-context exemplars for related verifiable questions. On GPT-OSS-120B, this improves Acc@1 from 67.2 to 76.3 and AUC from 71.4 to 79.6 relative to LLM judges, and the paper states that it exhibits a larger solver-evaluator gap in the sense of maintaining stronger correct-wrong separation even when the underlying solver often fails to solve (Son et al., 6 Feb 2026).

Coding shows a similar pattern. “CodeJudgeBench” benchmarks 26 judge models on code generation, code repair, and unit test generation. Recent thinking models outperform non-thinking models, and relatively small thinking models such as Qwen3-8B can outperform specially trained LLM-as-Judge models up to 70B in size. Yet all models still exhibit significant randomness, pairwise judging is sensitive to response order, and performance varies with the source model that wrote the code or tests (Jiang et al., 14 Jul 2025). In software engineering more broadly, “SE-Jury” introduces an ensemble of five evaluation strategies with dynamic team selection, and reports correlation gains of 29.6% to 140.8% over existing automatic metrics, with agreement levels close to inter-annotator agreement on code generation and program repair (Zhou et al., 27 May 2025).

The same general phenomenon also appears in scientific simulation. “A Judge Agent Closes the Reliability Gap in AI-Generated Scientific Simulation” does not study LLM-as-judge in the usual preference-ranking format; instead it automates well-posedness, convergence, and error certification. Across 134 test cases spanning 12 scientific domains, the Judge Agent reduces the silent-failure rate from 42% to 1.5%, and on a blinded 72-task benchmark raises success from 53% without the Judge to 89% with automated error bounds (Yang, 26 Mar 2026). This is a contrasting case in which the gap is narrowed by replacing generic judgment with domain-specific validation.

5. Methods proposed to narrow the gap

Several methods aim to close the Solve-to-Judge Gap by forcing the judge to reason, decompose, or solve before evaluating. “S2J” addresses the per-query gap in generative reward models by explicitly linking solving and judging during optimization. It requires the model to solve the query before judging and combines a solving reward with a judgment reward. The reported effect is a 16.2% reduction in the solve-to-judge gap and a 5.8% improvement in judgment performance, reaching state-of-the-art performance among generative reward models built on the same base model with a smaller training dataset (Sun et al., 26 Sep 2025).

In agent evaluation, “Auto-Eval Judge” decomposes tasks into subgoals through a Criteria Generator, Artifact Content Parser, Criteria Check Composer, and Verdict Generator. It evaluates logs, reasoning traces, and artifacts rather than only the final output. On GAIA and BigCodeBench, it improves alignment accuracy by 4.76% and 10.52%, respectively, relative to a GPT-4o based LLM-as-a-Judge baseline (Bhonsle et al., 7 Aug 2025). “FinResearchBench” adopts a related design for financial research agents by extracting a logic tree from the report and using that intermediate representation for both rule-based and LLM-scored evaluation across 70 questions and 7 task types (Sun et al., 22 Jul 2025).

Another cluster of methods emphasizes judge training. “JudgeLRM” argues that supervised fine-tuning gains are negatively correlated with the proportion of reasoning-demanding samples, and trains judges with reinforcement learning and outcome-driven rewards. JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1, with especially strong gains on reasoning-heavy tasks (Chen et al., 31 Mar 2025). “J4R” introduces Equivalent Initial State Group Relative Policy Optimization to enforce robustness to equivalent initial states such as answer-order swaps; its 7B judge outperforms GPT-4o and the next best small judge by 6.7% and 9% (Xu et al., 19 May 2025). “JudgeBoard” finds a significant judging gap between small and LLMs, particularly on wrong-answer detection, and reports that its MAJ multi-agent framework substantially improves the reliability and consistency of small models; on MATH, MAJ using smaller backbones performs comparatively well or better than larger counterparts (Bi et al., 20 Nov 2025).

A distinct but related strategy is to separate judgment from solution generation at the trajectory level. “Judge-Then-Solve” trains models to make an explicit answerability commitment before solution generation. It reports that Abstention@Detection approaches near-saturation, with values of 99.8% for DeepSeek under JTS and 99.3% for Qwen3 under JTS, while average generation length on under-specified questions falls to 349 and 343 tokens, respectively (Gu et al., 27 May 2026). Although this work focuses on abstention under insufficient information, it operationalizes the same principle: reliable solving requires reliable judging of whether solving should proceed at all.

6. Reporting standards, misconceptions, and open problems

A consistent conclusion across the literature is that LLM judging should be reported as an audited measurement protocol rather than a single headline score. “When the Judge Changes, So Does the Measurement” recommends reporting dataset slices, tie and parse-failure handling, position and verbosity bias probes, error-correlation estimates for juries, full parser and fallback logs, debate-round outcomes, matched-pair tests, Holm correction, and uncertainty intervals (Yang et al., 9 Jul 2026). “The Coin Flip Judge?” likewise argues for multi-trial aggregation, position randomization, explicit uncertainty reporting, and joint reporting of pairwise preferences with pointwise scores (Yagubyan, 23 Apr 2026). “RobustJudge” adds adversarial evaluation, layered defenses, and careful prompt/model selection as necessary components of deployment (Li et al., 11 Jun 2025). In benchmark construction, Omni-MATH-2 recommends treating evaluations as triplets of dataset, model, and judge rather than as dataset-model pairs (Ballon et al., 27 Jan 2026).

Several recurrent misconceptions are directly contradicted by the evidence. Scaling the judge does not guarantee monotonic measurement improvement: only the Qwen3 1.7B $P(j=0 \mid s=1)$6 4B jump is consistently significant across the four datasets audited in (Yang et al., 9 Jul 2026). Bias does not disappear in stronger judges: both position and verbosity effects remain non-zero even in the best-performing models (Yang et al., 9 Jul 2026). Majority voting is not automatically reliable: when errors are correlated, repeated-sample juries add little (Yang et al., 9 Jul 2026). Rationales are not faithful audit traces: CAR can remain near zero even when irrelevant cues drive verdicts (Marioriyad et al., 8 Feb 2026). Reference-free self-judging does not necessarily track correctness: under self-play, pass rates can rise while true accuracy stays flat (Zhou, 7 Jul 2026).

The open problem is therefore not only to improve judge accuracy. It is to define evaluation procedures whose outputs remain stable under judge replacement, robust to spurious cues and adversarial inputs, sensitive to process errors rather than surface plausibility, and auditable after the fact. Current work suggests that progress depends on slice-aware reporting, protocol transparency, explicit bias probes, de-anchored or solve-first judging, and domain-specific verification mechanisms. The Solve-to-Judge Gap is thus best understood as a central reliability bottleneck in automated evaluation rather than a narrow failure mode of any single model family.