Solve-to-Judge (S2J) Evaluation Methods

• Solve-to-Judge (S2J) is a set of methods where judgment is performed through a reasoning process that combines solving abilities with evaluative decision-making.
• Researchers employ techniques like dual reward signals, reinforcement learning, and consequence-based evaluation to bridge the gap between solving and judging.
• Benchmarks indicate that S2J approaches can reduce judgment errors by up to 16.2% and improve evaluation accuracy across diverse domains.

Solve-to-Judge (S2J) denotes a family of research ideas in which judging is treated as a nontrivial reasoning problem rather than a simple post hoc classifier. In contemporary large-language-model research, the term most often refers to the relationship between a model’s ability to solve a task and its ability to evaluate candidate answers, including methods that explicitly couple self-solving and judgment during training or inference (Sun et al., 26 Sep 2025, Chen et al., 31 Mar 2025). A related usage in research-level mathematics evaluates candidate proofs by their downstream utility on nearby, verifiable problems, thereby providing an oracle-free judging signal when direct verification is unavailable or too costly (Son et al., 6 Feb 2026). The phrase has also appeared in earlier, terminologically distinct literatures on secure majority protocols and automated code-evaluation platforms (0910.4044, Wasik et al., 2017).

1. Definitions and conceptual scope

The general LLM-as-a-judge formulation can be written as

$R = J(C_1, \ldots, C_n),$

where $J$ is the judge model, $\{C_i\}$ are candidate outputs, and $R$ is a score, ranking, or selection (Li et al., 2024). Within that broader paradigm, S2J emphasizes that evaluation itself often requires solving-like cognition: evidence verification, error identification, and decision justification are not reducible to surface scoring (Chen et al., 31 Mar 2025).

Recent papers use the term in closely related but not identical ways. In generative reward modeling, the solve-to-judge gap is defined as the phenomenon where a model fails to judge correctly on some queries even though it can solve them; the paper quantifies this as the conditional failure probability $P(j=0 \mid s=1)$ (Sun et al., 26 Sep 2025). In consequence-based evaluation for research mathematics, the central observation is different: an evaluator may still separate correct from incorrect candidate solutions even when the underlying solver rarely solves the target problem directly, producing what the paper calls a stronger solver-evaluator gap (Son et al., 6 Feb 2026). Both usages share the claim that judgment and direct problem solving are coupled, but not identical.

The literature also distinguishes judging formats and supervision regimes. Inputs may be pointwise, pairwise, or listwise, and outputs may be score-based, ranking-based, or selection-based (Li et al., 2024). Judge-specialized systems then differ in whether they rely on supervised fine-tuning, reinforcement learning, direct preference optimization, rubric prompting, multi-agent deliberation, or downstream consequence tests (Chen et al., 31 Mar 2025, Yu et al., 17 Feb 2025, Yang et al., 6 Feb 2026).

2. The solve-to-judge gap and its empirical characterization

The most direct empirical statement of the S2J gap appears in generative reward modeling. The paper “S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models” reports that GRMs struggle to make correct judgments on some queries, in the range of 14%–37%, despite being fully capable of solving them (Sun et al., 26 Sep 2025). That result challenges the common assumption that better problem-solving ability automatically implies proportionally better judgment.

A related diagnosis appears in judgment-oriented LLM training. “JudgeLRM: Large Reasoning Models as a Judge” reports a negative correlation between supervised fine-tuning performance gains and the proportion of reasoning-demanding samples, arguing that judgment tasks become increasingly resistant to shallow adaptation as they require more evidence verification and multi-step reasoning (Chen et al., 31 Mar 2025). This directly situates S2J as a reasoning bottleneck rather than a mere formatting problem.

In research-level mathematics, the gap manifests differently. “Judging What We Cannot Solve” shows that LLM-Judge separation between correct and incorrect solutions degrades rapidly as difficulty increases, becoming almost random when the model cannot solve the underlying problem, whereas Consequence-Based Utility maintains robust separability even on problems the model almost never solves directly (Son et al., 6 Feb 2026). The paper explicitly states that CBU maintains a stronger correct-wrong separation on instances where the underlying solver often fails to solve.

Benchmarking work further shows that judge limitations can dominate observed progress. “Benchmarks Saturate When The Model Gets Smarter Than The Judge” compares GPT-5 mini with Omni-Judge on Omni-MATH-2 and reports that expert annotations found Omni-Judge wrong in 96.4% of the judge disagreements, indicating that judge errors can mask genuine differences between strong models well before benchmark saturation (Ballon et al., 27 Jan 2026). The paper argues that increasingly competent judges become essential as problems become more challenging.

At smaller model scales, the same pattern appears as a capacity gap. “JudgeBoard” reports a significant performance gap between SLMs and LLMs in isolated judging tasks, especially on reasoning-heavy evaluation, even though its Multi-Agent Judging framework narrows that gap through collaborative deliberation (Bi et al., 20 Nov 2025).

3. Methods for linking solving and judging

A central methodological response to S2J is to require the judge to solve before it judges. In the GRM setting, the S2J method samples a trajectory

$$\tau = (\hat{y}, c, \hat{l}) \sim \pi_\theta(\cdot \mid x, y_a, y_b),$$

where $\hat{y}$ is the model’s self-generated solution, $c$ is an explanation, and $\hat{l}$ is the preferred candidate. Training uses a dual reward

$$\mathcal{R}(\tau) = \mathcal{R}_{\text{solve}}(\tau) + \mathcal{R}_{\text{judge}}(\tau),$$

so that solving accuracy and judgment correctness are optimized jointly (Sun et al., 26 Sep 2025). On the reported benchmarks, this reduces the solve-to-judge gap by 16.2% and improves judgment performance by 5.8%, reaching 72.7% average judging accuracy while using 20K training samples and no more powerful external model for distillation (Sun et al., 26 Sep 2025).

A second line of work treats judgment as a specialized reasoning policy. JudgeLRM is trained with reinforcement learning using judge-wise, outcome-driven rewards and a structured output format with > and <answer> tags (Chen et al., 31 Mar 2025). Its reward combines structural, relation, absolute, and confidence components, and optimization uses Grouped Relative Policy Optimization (GRPO). The paper reports that JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score; on PandaLM test, JudgeLRM-7B reaches 75.05 F1 versus 67.98 for Qwen2.5-7B-Instruct-SFT and 72.48 for DeepSeek-R1 (Chen et al., 31 Mar 2025).

A third line emphasizes judge training as a general capability rather than a task-local evaluator. “Improve LLM-as-a-Judge Ability as a General Ability” uses a two-stage SFT warm-up plus DPO enhancement pipeline, together with synthetic judgmental data generation and answer-order/length bias mitigation (Yu et al., 17 Feb 2025). The paper reports that RISE-Judge-Qwen2.5-32B achieves 92.7 on RewardBench using 40k samples, which it describes as about 2% to 40% of the data required by comparable methods, and that the same training improves downstream DPO policy optimization beyond GPT-4o-annotated training (Yu et al., 17 Feb 2025).

A fourth line makes judging itself a regularized policy over task, rubric, and mode. FairJudge defines

$J$0

where $J$1 is input, $J$2 is context such as rubric or reference, $J$3 is evaluation mode, and $J$4 is the judgment (Yang et al., 6 Feb 2026). Its data-centric pipeline constructs FairJudge-16K with canonicalization, debiasing perturbations, and consistency supervision, and training proceeds via SFT, DPO, and GRPO to align rubric adherence, bias mitigation, and cross-mode consistency (Yang et al., 6 Feb 2026).

These methods differ in where they attach the solving signal. S2J for GRMs inserts self-solving into each judgment trajectory; JudgeLRM strengthens reasoning during judging; RISE-Judge treats judge ability as a transferable model capability; FairJudge formalizes judgment as a policy over rubric, bias, and mode. A plausible implication is that S2J is not one algorithm but a design principle: evaluation quality improves when the model is forced to ground judgment in task-level reasoning rather than surface preference heuristics.

4. Consequence-based S2J in research-level mathematics

The most explicit oracle-free mathematical instantiation of S2J is Consequence-Based Utility (CBU). The method begins from the hypothesis that a meaningful solution to a research-level mathematics problem should contain enough method-level information that, when applied to a neighborhood of related questions, it yields better downstream performance than incorrect solutions (Son et al., 6 Feb 2026). For a target problem $J$5, candidate solution $J$6, neighborhood $J$7, solver $J$8, and automated verifier $J$9, the utility is defined as

$\{C_i\}$0

with empirical estimate

$\{C_i\}$1

The candidate is therefore judged not by direct proof checking, but by whether it functions as a useful in-context exemplar for solving nearby verifiable problems (Son et al., 6 Feb 2026).

The evaluation uses ExpertMath*, consisting of 192 expert-written, research-level mathematics problems across areas including representation theory, algebraic geometry, combinatorics, and homotopy theory; each problem is paired with up to 2 neighborhood variants, one expert solution, and 9 LLM-generated candidate solutions that were curated and manually judged for correctness (Son et al., 6 Feb 2026). Metrics include Acc@1, Recall@5, AUC, and HumanWin / MeanWin, and the paper uses 64 independent rollouts per candidate for CBU and LLM-Judge (Son et al., 6 Feb 2026).

CBU is reported to outperform reward models, generative reward models, and LLM judges on ranking quality. The paper highlights the following excerpted results:

Method	Acc@1 (%)	AUC (%)
GPT-OSS-120B LLM-Judge	67.2	71.4
GPT-OSS-120B CBU	76.3	79.6
GPT-OSS-20B LLM-Judge	72.1	69.0
GPT-OSS-20B CBU	74.6	79.2
Best GenRM baseline	65.4	67.9

The error analysis further states that CBU more reliably downranks wrong solutions exhibiting incorrect reasoning, unjustified compression, unjustified interpretation, and external references, while LLM-Judges are more easily fooled by style, authority cues, and verbosity (Son et al., 6 Feb 2026). The paper also reports that only a small number of neighborhood questions and rollouts, for example about 8, are needed for stable signaling, and that automatically generated neighborhoods can still be informative (Son et al., 6 Feb 2026).

This research changes the meaning of “solve to judge” in a specific way. Instead of asking a judge to solve the original target, it asks whether a candidate solution transfers to nearby solvable tasks. The judgment signal is therefore a consequence test rather than an oracle or line-by-line proof check.

5. Benchmarks, capability decomposition, and evaluation protocols

The survey literature organizes LLM-as-a-judge along three dimensions: what to judge, how to judge, and how to benchmark (Li et al., 2024). This framing is important for S2J because it separates the judged attribute—helpfulness, safety, reliability, relevance, logic, or overall quality—from the judgment protocol itself, including pointwise versus pairwise inputs, score versus ranking outputs, and prompting or multi-agent aggregation choices (Li et al., 2024).

Capability-oriented benchmarking pushes this decomposition further. “Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation” introduces M-JudgeBench, a ten-dimensional benchmark with 3,712 curated pairwise instances spanning pairwise CoT comparison, length bias avoidance, and process error detection (Chen et al., 28 Feb 2026). The benchmark is explicitly designed to diagnose systematic weaknesses in MLLM-as-a-judge systems, including confusion on similar-style pairs, persistent length bias, and weak detection of subtle process errors (Chen et al., 28 Feb 2026). The same work introduces Judge-MCTS, which generates Short-Correct, Short-Error, Long-Correct, and Long-Error reasoning trajectories to train M-Judger models via SFT and RL (Chen et al., 28 Feb 2026).

JudgeBoard offers a different evaluation protocol: direct querying of models to assess whether a candidate answer is correct, rather than comparing answers against other candidates through an external comparison metric (Bi et al., 20 Nov 2025). It constructs leaderboards for MATH, ARC-Challenge, GSM8K, OMNIMATH, and GPQA using both accuracy-based ranking and Elo-based ratings, with explicit Student Wrong and Student Right splits to expose sycophancy and rejection failures (Bi et al., 20 Nov 2025). Its Multi-Agent Judging framework uses agents with distinct reasoning profiles, an independent-judgment phase, a debate phase, and majority voting; on MATH, MAJ with Qwen3-14B reportedly outperforms a larger Qwen3-30B-A3B judge by +2% in accuracy (Bi et al., 20 Nov 2025).

Benchmark construction itself can also expose S2J failure. Omni-MATH-2 revises the original Omni-MATH into a clean exact-answer subset of $\{C_i\}$2 and a tagged non-standard subset of $\{C_i\}$3, after auditing all 4,428 problems for LaTeX compilability, solvability, and verifiability, including 647 edited items and tags such as image, proof, estimation, and should delete (Ballon et al., 27 Jan 2026). The study reports substantial discrepancies between GPT-5 mini and Omni-Judge on both clean and tagged subsets, and states that neither judge identifies the present failure modes for the tagged problems (Ballon et al., 27 Jan 2026). In this setting, benchmark quality and judge quality are jointly limiting.

6. Reliability, attacks, and mechanistic accounts

S2J systems inherit the standard vulnerabilities of LLM-as-a-judge, but their coupling to reasoning makes those vulnerabilities more consequential. “Optimization-based Prompt Injection Attack to LLM-as-a-Judge” introduces JudgeDeceiver, an attack that appends an optimized adversarial sequence to an attacker-controlled candidate response so that the judge selects it for an attacker-chosen question (Shi et al., 2024). The attack formulates adversarial suffix search as a gradient-based optimization over target-aligned generation loss, target-enhancement loss, and adversarial perplexity loss. On the reported evaluation, JudgeDeceiver reaches attack success rates of about 89–93%, positional attack consistency of 79–94%, and outperforms handcrafted prompt injections and GCG-style baselines; known-answer detection, perplexity detection, and windowed perplexity detection are all reported as insufficient (Shi et al., 2024).

Another line of work locates reliability failures inside the model rather than at the prompt boundary. “Judge Circuits” studies Gemma-3, Qwen2.5, and Llama-3 with Position-aware Edge Attribution Patching (PEAP) and reports a sparse, generalized Latent Evaluator sub-graph in the mid-to-late MLPs, together with fragile, format-specific terminal branches (Feldhus et al., 15 May 2026). The paper argues that a continuous judgment signal is computed in a shared trunk and then mapped through output-format-specific branches, so that apparent inconsistency between a 1–5 rating and a True/False label is partly a formatter problem rather than a failure of abstract evaluation (Feldhus et al., 15 May 2026). Zero-ablation of the Latent Evaluator collapses judgment while preserving world knowledge in architecturally modular models, and the paper concludes that benchmark-level reliability comparisons across formats are partially measuring formatter geometry rather than evaluation quality (Feldhus et al., 15 May 2026).

These findings interact directly with the debiasing literature. FairJudge targets position, length, format, and provenance biases, as well as pointwise-versus-pairwise inconsistency, by training judging behavior as a learnable policy with explicit regularization (Yang et al., 6 Feb 2026). M-JudgeBench likewise isolates length bias and process error detection as distinct capability dimensions rather than treating judge failure as a single scalar weakness (Chen et al., 28 Feb 2026). Taken together, the literature treats S2J reliability as a composite problem involving adversarial robustness, non-semantic bias, cross-format consistency, and mechanistic separation between abstract evaluation and output formatting.

7. Earlier and distinct uses of the term

Before its recent LLM-centered usage, “solve-to-judge” appeared in unrelated settings. In secure multi-party computation, “How to Work with Honest but Curious Judges?” studies majority computation among $\{C_i\}$4 judges who wish to reach a verdict without revealing individual decisions (0910.4044). The paper generalizes a three-judges protocol in two ways, introduces conditional anonymity, and formalizes privacy with epistemic temporal logic such as

$\{C_i\}$5

Here S2J refers to a privacy-preserving decision protocol, not to LLM evaluation (0910.4044).

In automated programming assessment, “A Survey on Online Judge Systems and Their Applications” uses S2J to describe the workflow in which a user submits a program, the system executes it in a homogeneous environment, and the result is judged for correctness and efficiency (Wasik et al., 2017). The paper formalizes a test instance as $\{C_i\}$6, the executable as $\{C_i\}$7, and the evaluation engine as

$\{C_i\}$8

followed by score aggregation over instances (Wasik et al., 2017). This is again a different sense of “solve” and “judge”: the system runs submitted code rather than asking a reasoning model to evaluate another model’s answer.

The coexistence of these usages shows that S2J is not a single canonical term across all of computer science. In current arXiv usage, however, the dominant meaning concerns the relationship between solving ability and evaluation ability in LLMs, reward models, and oracle-free mathematical judging (Sun et al., 26 Sep 2025, Son et al., 6 Feb 2026).