Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization

Abstract: Text-space skill optimization adapts a frozen agent by evolving a natural-language skill document, accepting each candidate through a validation gate. Existing gates rely on verifiable rewards, confining these methods to tasks with an automatic verifier. Replacing the verifier with an LLM-judge gate would lift that restriction, but whether such a gate carries usable signal is untested. We ask a prior question: can we tell, before placing a judge in the loop, whether its scores separate correct from incorrect answers at all? We formalize a reference-free judge as a latent solver -- its verdict rests on agreement with whatever it would itself conclude, so its capacity to evaluate is bounded by its capacity to solve. The model yields a closed-form bound on discriminability (ROC-AUC) in the judge's competence $c$ and answer-space size $k$, a necessary condition $c > 1/k$, and the result that the marginal AUC is confounded by item difficulty while a within-question estimator is not. A non-intervening probe records judge scores on genuine optimization runs without altering any decision. We find discriminability at chance where competence sits near the floor and usable above it; that a judge's benchmark accuracy overstates the competence that matters; and, in a closed-loop study, that the screen predicts which kind of gating error occurs. The result is a cheap pre-deployment diagnostic for judge gates.

• The paper derives an AUC bound, AUC = 1/2 + (ck − 1)/[2(k − 1)], showing that judge competence must exceed 1/k for above-chance discrimination, though this condition is not sufficient under collusion.
• The paper proposes a non-intervening, episode-level probe and within-question AUC to separate answer-level judgment from item difficulty, revealing task-dependent performance from near-random mathematics scores of 0.457 to factual-QA scores of 0.855.
• The paper finds that benchmark accuracy can substantially overstate competence—Sonnet’s GPQA-Diamond score falls from 0.840 answered-only to 0.547–0.602 after correction—and shows that failing judges can import harmful candidates during closed-loop optimization.
• follow_up_questionsconversation: What assumptions are required for the latent-solver AUC bound to hold?
• How does within-question AUC remove difficulty-related confounding in judge evaluation?
• Why can a judge with below-threshold competence actively harm skill optimization?
• How should competence be estimated for open-ended tasks without a single gold answer?
• Find recent papers about reference-free LLM judge evaluation and judge-gated optimization.

Motivation and problem statement

Text-space skill optimization adapts a frozen agent by evolving a natural-language skill document, accepting each candidate through a validation gate. Existing gates rely on verifiable rewards—exact match or executable checks—which confines the paradigm to tasks with an automatic verifier. Replacing the verifier with an LLM-judge gate would lift this restriction, but the authors observe that no prior work establishes whether such a gate carries usable signal before deployment. Judge-based reward signals exist in RL post-training (Alzubi et al., 3 Mar 2026), but always in weight space and always assuming usable signal; in text-space skill optimization the optimized object is a persistent document and the judge must supply its own standard of correctness, since no gold answer exists for the candidate being evaluated.

The paper's central question is deliberately prior to deployment: can one tell, before placing a judge in the acceptance loop, whether its scores separate correct from incorrect answers at all? The answer is operationalized as discriminability (ROC-AUC) between verifier-labeled correct and incorrect episodes, measured on genuine optimization runs via a non-intervening probe that records judge scores without altering any accept/reject decision.

The judge-as-latent-solver model

The theoretical core models a reference-free judge as a latent solver under three assumptions: (A1) implicit re-solving—the judge forms an internal solution $\hat{a}$ conditionally independent of the candidate $a$, with competence $c = \Pr(\hat{a} = a^*)$; (A2) error dispersion—errors are uniform over the remaining $k-1$ answers; (A3) consistency scoring—the score is $\mathbb{1}[a = \hat{a}]$. This is a Dawid–Skene noisy-annotator model whose confusion structure is realized as explicit re-solving, making $c$ separately measurable by closed-book self-solve accuracy.

Proposition 1 gives the closed-form result:

$\mathrm{AUC} = \frac12 + \frac{ck - 1}{2(k-1)}$

with three consequences. The chance-floor threshold: AUC exceeds chance if and only if $c > 1/k$. Under relaxed assumptions allowing correlated errors between judge and candidate ("collusion," rate $\rho$), only the forward implication survives: above-chance competence is necessary but not sufficient for discrimination. The paper is careful throughout to state this as a necessary condition only. The free-text limit: as $k \to \infty$, AUC approaches $a$0, since independent errors almost never coincide verbatim; at small $a$1, the judge's wrong solution may endorse the candidate's wrong answer, so discriminability is ordered jointly by $a$2. An inversion estimator recovers $a$3 from observed AUC and $a$4, constructed purely from grading behavior and therefore orthogonal to benchmark contamination.

Two robustness results shape how the closed form should be read. Label-independent noise shrinks AUC toward chance by a positive factor without changing sign (Lemma 1), and super-uniform collusion makes Proposition 1 an attainable upper bound rather than a fitted curve (Remark 1). Separately, Remark 2 shows the marginal AUC is confounded by item difficulty: when difficulty acts on both judge and candidate, positives are enriched in easy items and negatives in hard ones, inflating marginal AUC—in the extreme, a judge scoring only perceived difficulty attains above-chance AUC with no answer-level discrimination. Proposition 2 establishes that the within-question AUC, pairing positives and negatives only within the same item, is invariant to any item-only score component, restoring identification of the threshold with pair-weighted competence.

Measurement instruments

Three design choices follow from the model. First, a non-intervening probe records $a$5 per episode while leaving the verifier-controlled acceptance rule untouched, so probed and unprobed runs are path-wise identical in distribution; what is measured is discriminability on the distribution induced by the verifier gate—a necessary condition for a usable judge gate, not sufficiency under judge control. Second, episode-level scoring preserves the per-episode consistency structure that skill-level averaging would erase. Third, an answer-content feed is required: feeding a bare multiple-choice letter renders scores independent of answers and forces AUC to exactly one half—an artifact the authors report encountering themselves as a spurious null in an early version of the probe.

Competence is measured by empty-inclusive closed-book self-solve accuracy, counting non-convergent episodes as unsolved to avoid survivorship bias. Because headline benchmark accuracies are frequently reported answered-only, the paper decomposes the gap between reported accuracy and genuine competence using estimators with orthogonal error sources: a proxy benchmark matched on domain and difficulty but less exposed, and the grading-only inversion estimate.

Empirical findings

Discriminability varies sharply across tasks

Using SkillOpt as optimizer, Haiku as target, and Claude Sonnet as judge across three tasks, marginal AUC ranges from near-random on research mathematics (0.457) to strong on factual QA (0.855), with GPQA-Diamond intermediate (0.735). Stratification matters materially: on factual QA a difficulty-confound share of +0.12 drops the honest within-question reading to 0.735, while on mathematics the two estimates agree. GPQA admits no within-question strata, so only the optimistic marginal figure is available there. Even after stratification, mathematics stays near chance—the headline claim being that judge signal cannot be assumed and is task-dependent.

The bound holds where it can be cleanly tested

Against independently measured competence, the predicted upper bound holds on every task admitting within-question stratification. Mathematics is the clearest case for the threshold: Sonnet's self-solve competence is 0.267 against a floor of $a$6, and observed discriminability is 0.489, marginally below chance. The single violation (Haiku on GPQA, observed 0.794 against a predicted bound of 0.694) falls in the only cell lacking both stratification and contamination-free competence measurement; two mechanisms are consistent with it—the difficulty-confounded marginal AUC, and $a$7 lower-bounding grading-context competence. The paper does not resolve which mechanism applies, noting that separation requires a task both contamination-controlled and multiply evaluated.

Benchmark accuracy overstates genuine competence

On GPQA-Diamond, Sonnet's answered-only headline of 0.840 decomposes into three components, each pushing in the same direction:

Component	Estimate	Mechanism
Answered-only	0.840	survivorship convention
Empty-inclusive	0.626	censoring removes unsolvable items
Proxy benchmark	0.547	benchmark exposure
Inversion	0.602	grading-path estimate

Censoring accounts for the largest share (21.4 points): reasoning judges fail to converge precisely on items they cannot solve, so discarding non-convergent episodes selects for solvable items. Exposure accounts for a materially smaller residual (7.9 points), though this inherits proxy-mismatch uncertainty. For Sonnet, all three independent estimators agree in the 0.55–0.63 range, well below the headline. For Haiku they diverge by 0.34, and the paper explicitly restricts the agreement claim to Sonnet, treating Haiku as a case of estimator divergence plausibly reflecting a stronger solve/grade scaffolding effect for the weaker model. The directional conclusion holds regardless: uncorrected benchmark figures are optimistic inputs to the chance-floor test.

Cross-provider generalization

Repeating the diagnostic with four judges from independent families (DeepSeek-8B, Qwen-7B, MiMo-7B, DeepSeek-V4-flash) reproduces the pattern everywhere: on mathematics, where competence sits at or below the floor (8–21%), discriminability is near-random (0.31–0.57); on factual QA, where competence is moderate-to-high (33–80%), it is usable (0.67–0.80). DeepSeek-V4-flash provides within-judge evidence by varying competence across tasks while holding the judge fixed: competent and discriminable on GPQA (73%, 0.76), incompetent and near-random on mathematics (8%, 0.31). One caveat applies: this matrix reports marginal AUC because per-episode scores were not retained, though the confound is approximately zero on mathematics—the column carrying the collapse result.

Closed-loop safety

In a small-scale study where only the gate varies, the screen predicts the composition of gating errors rather than final performance. Judges failing the screen import regressions: false-accept rates of 0.33 on factual QA and 0.50 on mathematics, the latter matching a coin flip. On mathematics, where the point estimate falls below one half, the judge gate is worse than random (final score 0.238 vs. 0.350, with non-overlapping per-seed ranges)—an anti-correlated signal actively drives the system away from improvement. The passing judge never admitted a harmful candidate (false-accept rate 0.00) and errs only by over-rejection (false-reject rate 0.44), forgoing some improvements rather than importing regressions, though its aggregate score matches the random gate rather than the verifier. The authors flag these as preliminary observations at two to three seeds, supporting ordering and error-type claims but not effect-size estimates.

Limitations and open questions

The scope restrictions are stated plainly. Both the model and measurements require a single gold answer and a verifier label, so the diagnostic does not yet extend to genuinely open-ended tasks where correctness is non-binary, the answer space has no natural size, and judging is rubric-driven; rubric-based and evidence-grounded judging, where verification may be easier than generation, fall outside assumption (A1). The chance-floor test is necessary but not sufficient: collusion at small $a$8 can suppress discrimination even given adequate competence, and clearing the screen does not predict competitive final performance. The single bound violation cannot be attributed between difficulty confounding and competence underestimation. Whether a judge that clears the screen survives sustained optimization under drift and Goodhart pressure is explicitly left open, as is sufficiency of the screen under judge-controlled acceptance beyond the pilot scale studied here.

Conclusion

This paper supplies the missing prerequisite analysis for judge-gated skill optimization: a latent-solver model yielding a closed-form bound on discriminability, a necessary chance-floor condition, an identification result separating genuine discrimination from difficulty confounding, and a non-intervening probe plus reusable audit protocol. Its empirical support spans five judges and three tasks, with the collapse-to-chance pattern reproduced across independent model families and the screen shown to predict harmful acceptance in closed-loop use. The practical upshot is a cheap pre-deployment screen—estimate uncontaminated competence on the target task and decline to gate on a judge that does not clear the floor—with the distinction between genuine competence and headline benchmark accuracy mattering most exactly where contamination is most likely.