---
title: Solve-to-Judge (S2J) Evaluation Methods
url: https://www.emergentmind.com/topics/solve-to-judge-s2j
type: topic
---

# Solve-to-Judge (S2J) Evaluation Methods

Solve-to-Judge (S2J) denotes a family of research ideas in which judging is treated as a nontrivial reasoning problem rather than a simple post hoc classifier. In contemporary large-language-model research, the term most often refers to the relationship between a model’s ability to solve a task and its ability to evaluate candidate answers, including methods that explicitly couple self-solving and judgment during training or inference [2509.22099][2504.00050]. A related usage in research-level mathematics evaluates candidate proofs by their downstream utility on nearby, verifiable problems, thereby providing an oracle-free judging signal when direct verification is unavailable or too costly [2602.06291]. The phrase has also appeared in earlier, terminologically distinct literatures on secure majority protocols and automated code-evaluation platforms [0910.4044][1710.05913].

## 1. Definitions and conceptual scope

The general LLM-as-a-judge formulation can be written as
\[
R = J(C_1, \ldots, C_n),
\]
where \(J\) is the judge model, \(\{C_i\}\) are candidate outputs, and \(R\) is a score, ranking, or selection [2411.16594]. Within that broader paradigm, S2J emphasizes that evaluation itself often requires solving-like cognition: evidence verification, error identification, and decision justification are not reducible to surface scoring [2504.00050].

Recent papers use the term in closely related but not identical ways. In generative reward modeling, the solve-to-judge gap is defined as the phenomenon where a model fails to judge correctly on some queries even though it can solve them; the paper quantifies this as the conditional failure probability \(P(j=0 \mid s=1)\) [2509.22099]. In consequence-based evaluation for research mathematics, the central observation is different: an evaluator may still separate correct from incorrect candidate solutions even when the underlying solver rarely solves the target problem directly, producing what the paper calls a stronger solver-evaluator gap [2602.06291]. Both usages share the claim that judgment and direct problem solving are coupled, but not identical.

The literature also distinguishes judging formats and supervision regimes. Inputs may be pointwise, pairwise, or listwise, and outputs may be score-based, ranking-based, or selection-based [2411.16594]. Judge-specialized systems then differ in whether they rely on supervised fine-tuning, reinforcement learning, direct preference optimization, rubric prompting, multi-agent deliberation, or downstream consequence tests [2504.00050][2502.11689][2602.06625].

## 2. The solve-to-judge gap and its empirical characterization

The most direct empirical statement of the S2J gap appears in generative reward modeling. The paper “S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models” reports that GRMs struggle to make correct judgments on some queries, in the range of 14%–37%, despite being fully capable of solving them [2509.22099]. That result challenges the common assumption that better problem-solving ability automatically implies proportionally better judgment.

A related diagnosis appears in judgment-oriented LLM training. “JudgeLRM: Large Reasoning Models as a Judge” reports a negative correlation between supervised fine-tuning performance gains and the proportion of reasoning-demanding samples, arguing that judgment tasks become increasingly resistant to shallow adaptation as they require more evidence verification and multi-step reasoning [2504.00050]. This directly situates S2J as a reasoning bottleneck rather than a mere formatting problem.

In research-level mathematics, the gap manifests differently. “Judging What We Cannot Solve” shows that LLM-Judge separation between correct and incorrect solutions degrades rapidly as difficulty increases, becoming almost random when the model cannot solve the underlying problem, whereas Consequence-Based Utility maintains robust separability even on problems the model almost never solves directly [2602.06291]. The paper explicitly states that CBU maintains a stronger correct-wrong separation on instances where the underlying solver often fails to solve.

Benchmarking work further shows that judge limitations can dominate observed progress. “Benchmarks Saturate When The Model Gets Smarter Than The Judge” compares GPT-5 mini with Omni-Judge on Omni-MATH-2 and reports that expert annotations found Omni-Judge wrong in 96.4% of the judge disagreements, indicating that judge errors can mask genuine differences between strong models well before benchmark saturation [2601.19532]. The paper argues that increasingly competent judges become essential as problems become more challenging.

At smaller model scales, the same pattern appears as a capacity gap. “JudgeBoard” reports a significant performance gap between SLMs and LLMs in isolated judging tasks, especially on reasoning-heavy evaluation, even though its Multi-Agent Judging framework narrows that gap through collaborative deliberation [2511.15958].

## 3. Methods for linking solving and judging

A central methodological response to S2J is to require the judge to solve before it judges. In the GRM setting, the S2J method samples a trajectory
\[
\tau = (\hat{y}, c, \hat{l}) \sim \pi_\theta(\cdot \mid x, y_a, y_b),
\]
where \(\hat{y}\) is the model’s self-generated solution, \(c\) is an explanation, and \(\hat{l}\) is the preferred candidate. Training uses a dual reward
\[
\mathcal{R}(\tau) = \mathcal{R}_{\text{solve}}(\tau) + \mathcal{R}_{\text{judge}}(\tau),
\]
so that solving accuracy and judgment correctness are optimized jointly [2509.22099]. On the reported benchmarks, this reduces the solve-to-judge gap by 16.2% and improves judgment performance by 5.8%, reaching 72.7% average judging accuracy while using 20K training samples and no more powerful external model for distillation [2509.22099].

A second line of work treats judgment as a specialized reasoning policy. JudgeLRM is trained with reinforcement learning using judge-wise, outcome-driven rewards and a structured output format with `<think>` and `<answer>` tags [2504.00050]. Its reward combines structural, relation, absolute, and confidence components, and optimization uses Grouped Relative Policy Optimization (GRPO). The paper reports that JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score; on PandaLM test, JudgeLRM-7B reaches 75.05 F1 versus 67.98 for Qwen2.5-7B-Instruct-SFT and 72.48 for DeepSeek-R1 [2504.00050].

A third line emphasizes judge training as a general capability rather than a task-local evaluator. “Improve LLM-as-a-Judge Ability as a General Ability” uses a two-stage SFT warm-up plus DPO enhancement pipeline, together with synthetic judgmental data generation and answer-order/length bias mitigation [2502.11689]. The paper reports that RISE-Judge-Qwen2.5-32B achieves 92.7 on RewardBench using 40k samples, which it describes as about 2% to 40% of the data required by comparable methods, and that the same training improves downstream DPO policy optimization beyond GPT-4o-annotated training [2502.11689].

A fourth line makes judging itself a regularized policy over task, rubric, and mode. FairJudge defines
\[
\pi_{\text{judge}}(y \mid x, c, m),
\]
where \(x\) is input, \(c\) is context such as rubric or reference, \(m\) is evaluation mode, and \(y\) is the judgment [2602.06625]. Its data-centric pipeline constructs FairJudge-16K with canonicalization, debiasing perturbations, and consistency supervision, and training proceeds via SFT, DPO, and GRPO to align rubric adherence, bias mitigation, and cross-mode consistency [2602.06625].

These methods differ in where they attach the solving signal. S2J for GRMs inserts self-solving into each judgment trajectory; JudgeLRM strengthens reasoning during judging; RISE-Judge treats judge ability as a transferable model capability; FairJudge formalizes judgment as a policy over rubric, bias, and mode. A plausible implication is that S2J is not one algorithm but a design principle: evaluation quality improves when the model is forced to ground judgment in task-level reasoning rather than surface preference heuristics.

## 4. Consequence-based S2J in research-level mathematics

The most explicit oracle-free mathematical instantiation of S2J is Consequence-Based Utility (CBU). The method begins from the hypothesis that a meaningful solution to a research-level mathematics problem should contain enough method-level information that, when applied to a neighborhood of related questions, it yields better downstream performance than incorrect solutions [2602.06291]. For a target problem \(Q\), candidate solution \(C\), neighborhood \(\mathcal{N}(Q)\), solver \(M_\theta\), and automated verifier \(v\), the utility is defined as
\[
U(C) = \frac{1}{|\mathcal{N}(Q)|} \sum_{Q^* \in \mathcal{N}(Q)}
\mathbb{E}_{\tilde{C} \sim M_\theta(\cdot \mid Q,\, C,\, Q^*)}\left[v(Q^*, \tilde{C})\right],
\]
with empirical estimate
\[
\widehat{U}(C) = \frac{1}{|\mathcal{N}(Q)|\, T} \sum_{Q^* \in \mathcal{N}(Q)} \sum_{t=1}^{T} v(Q^*, \tilde{C}_t).
\]
The candidate is therefore judged not by direct proof checking, but by whether it functions as a useful in-context exemplar for solving nearby verifiable problems [2602.06291].

The evaluation uses ExpertMath*, consisting of 192 expert-written, research-level mathematics problems across areas including representation theory, algebraic geometry, combinatorics, and homotopy theory; each problem is paired with up to 2 neighborhood variants, one expert solution, and 9 LLM-generated candidate solutions that were curated and manually judged for correctness [2602.06291]. Metrics include Acc@1, Recall@5, AUC, and HumanWin / MeanWin, and the paper uses 64 independent rollouts per candidate for CBU and LLM-Judge [2602.06291].

CBU is reported to outperform reward models, generative reward models, and LLM judges on ranking quality. The paper highlights the following excerpted results:

| Method | Acc@1 (%) | AUC (%) |
|---|---:|---:|
| GPT-OSS-120B LLM-Judge | 67.2 | 71.4 |
| GPT-OSS-120B CBU | 76.3 | 79.6 |
| GPT-OSS-20B LLM-Judge | 72.1 | 69.0 |
| GPT-OSS-20B CBU | 74.6 | 79.2 |
| Best GenRM baseline | 65.4 | 67.9 |

The error analysis further states that CBU more reliably downranks wrong solutions exhibiting incorrect reasoning, unjustified compression, unjustified interpretation, and external references, while LLM-Judges are more easily fooled by style, authority cues, and verbosity [2602.06291]. The paper also reports that only a small number of neighborhood questions and rollouts, for example about 8, are needed for stable signaling, and that automatically generated neighborhoods can still be informative [2602.06291].

This research changes the meaning of “solve to judge” in a specific way. Instead of asking a judge to solve the original target, it asks whether a candidate solution transfers to nearby solvable tasks. The judgment signal is therefore a consequence test rather than an oracle or line-by-line proof check.

## 5. Benchmarks, capability decomposition, and evaluation protocols

The survey literature organizes LLM-as-a-judge along three dimensions: what to judge, how to judge, and how to benchmark [2411.16594]. This framing is important for S2J because it separates the judged attribute—helpfulness, safety, reliability, relevance, logic, or overall quality—from the judgment protocol itself, including pointwise versus pairwise inputs, score versus ranking outputs, and prompting or multi-agent aggregation choices [2411.16594].

Capability-oriented benchmarking pushes this decomposition further. “Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation” introduces M-JudgeBench, a ten-dimensional benchmark with 3,712 curated pairwise instances spanning pairwise CoT comparison, length bias avoidance, and process error detection [2603.00546]. The benchmark is explicitly designed to diagnose systematic weaknesses in MLLM-as-a-judge systems, including confusion on similar-style pairs, persistent length bias, and weak detection of subtle process errors [2603.00546]. The same work introduces Judge-MCTS, which generates Short-Correct, Short-Error, Long-Correct, and Long-Error reasoning trajectories to train M-Judger models via SFT and RL [2603.00546].

JudgeBoard offers a different evaluation protocol: direct querying of models to assess whether a candidate answer is correct, rather than comparing answers against other candidates through an external comparison metric [2511.15958]. It constructs leaderboards for MATH, ARC-Challenge, GSM8K, OMNIMATH, and GPQA using both accuracy-based ranking and Elo-based ratings, with explicit Student Wrong and Student Right splits to expose sycophancy and rejection failures [2511.15958]. Its Multi-Agent Judging framework uses agents with distinct reasoning profiles, an independent-judgment phase, a debate phase, and majority voting; on MATH, MAJ with Qwen3-14B reportedly outperforms a larger Qwen3-30B-A3B judge by +2% in accuracy [2511.15958].

Benchmark construction itself can also expose S2J failure. Omni-MATH-2 revises the original Omni-MATH into a clean exact-answer subset of \(n=4181\) and a tagged non-standard subset of \(n=247\), after auditing all 4,428 problems for LaTeX compilability, solvability, and verifiability, including 647 edited items and tags such as image, proof, estimation, and should delete [2601.19532]. The study reports substantial discrepancies between GPT-5 mini and Omni-Judge on both clean and tagged subsets, and states that neither judge identifies the present failure modes for the tagged problems [2601.19532]. In this setting, benchmark quality and judge quality are jointly limiting.

## 6. Reliability, attacks, and mechanistic accounts

S2J systems inherit the standard vulnerabilities of LLM-as-a-judge, but their coupling to reasoning makes those vulnerabilities more consequential. “Optimization-based Prompt Injection Attack to LLM-as-a-Judge” introduces JudgeDeceiver, an attack that appends an optimized adversarial sequence to an attacker-controlled candidate response so that the judge selects it for an attacker-chosen question [2403.17710]. The attack formulates adversarial suffix search as a gradient-based optimization over target-aligned generation loss, target-enhancement loss, and adversarial perplexity loss. On the reported evaluation, JudgeDeceiver reaches attack success rates of about 89–93%, positional attack consistency of 79–94%, and outperforms handcrafted prompt injections and GCG-style baselines; known-answer detection, perplexity detection, and windowed perplexity detection are all reported as insufficient [2403.17710].

Another line of work locates reliability failures inside the model rather than at the prompt boundary. “Judge Circuits” studies Gemma-3, Qwen2.5, and Llama-3 with Position-aware Edge Attribution Patching (PEAP) and reports a sparse, generalized Latent Evaluator sub-graph in the mid-to-late MLPs, together with fragile, format-specific terminal branches [2605.16023]. The paper argues that a continuous judgment signal is computed in a shared trunk and then mapped through output-format-specific branches, so that apparent inconsistency between a 1–5 rating and a True/False label is partly a formatter problem rather than a failure of abstract evaluation [2605.16023]. Zero-ablation of the Latent Evaluator collapses judgment while preserving world knowledge in architecturally modular models, and the paper concludes that benchmark-level reliability comparisons across formats are partially measuring formatter geometry rather than evaluation quality [2605.16023].

These findings interact directly with the debiasing literature. FairJudge targets position, length, format, and provenance biases, as well as pointwise-versus-pairwise inconsistency, by training judging behavior as a learnable policy with explicit regularization [2602.06625]. M-JudgeBench likewise isolates length bias and process error detection as distinct capability dimensions rather than treating judge failure as a single scalar weakness [2603.00546]. Taken together, the literature treats S2J reliability as a composite problem involving adversarial robustness, non-semantic bias, cross-format consistency, and mechanistic separation between abstract evaluation and output formatting.

## 7. Earlier and distinct uses of the term

Before its recent LLM-centered usage, “solve-to-judge” appeared in unrelated settings. In secure multi-party computation, “How to Work with Honest but Curious Judges?” studies majority computation among \(2n+1\) judges who wish to reach a verdict without revealing individual decisions [0910.4044]. The paper generalizes a three-judges protocol in two ways, introduces conditional anonymity, and formalizes privacy with epistemic temporal logic such as
\[
AG(\phi_{i,j} \Rightarrow (\neg K_i(d_j = 1) \wedge \neg K_i(d_j = 0))).
\]
Here S2J refers to a privacy-preserving decision protocol, not to LLM evaluation [0910.4044].

In automated programming assessment, “A Survey on Online Judge Systems and Their Applications” uses S2J to describe the workflow in which a user submits a program, the system executes it in a homogeneous environment, and the result is judged for correctness and efficiency [1710.05913]. The paper formalizes a test instance as \(t_i=(d_i,o_i,p_i)\), the executable as \(b\), and the evaluation engine as
\[
E(b,t_i) \rightarrow (s_i, v_i, e_i),
\]
followed by score aggregation over instances [1710.05913]. This is again a different sense of “solve” and “judge”: the system runs submitted code rather than asking a reasoning model to evaluate another model’s answer.

The coexistence of these usages shows that S2J is not a single canonical term across all of computer science. In current arXiv usage, however, the dominant meaning concerns the relationship between solving ability and evaluation ability in LLMs, reward models, and oracle-free mathematical judging [2509.22099][2602.06291].

Source: https://www.emergentmind.com/topics/solve-to-judge-s2j