JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

Abstract: We ask whether Jev, a typed classifier that returns probabilities over permitted answers without generating text, can replace an LLM rubric judge. We compare it with three flash-tier LLM judges on nine panels drawn from seven benchmarks, giving every judge identical criterion texts. Jev's accuracy differs significantly from an LLM judge's in only 8 of 27 paired comparisons, ahead mostly on binary criteria and behind only on graded ones, and most of the other comparisons are inconclusive. Summed over the nine panels, the LLM judges, called once per criterion, cost 29 to 325 times as much as Jev and took 30 to 220 times as long. On graded criteria all four judges agree more with one another than with the labels and mostly assign lower levels than the raters. One of several observational accounts is that raters followed scale conventions our criterion texts omit. Jev's confidence ranks its own errors on most panels, which should make a cheap classifier the ideal first stage of a cascade that defers its uncertain verdicts to an LLM judge. Correlated errors undo that advantage. The LLM judges repeat nearly all of Jev's most confident errors, so a cascade replayed on the recorded verdicts lowers cost but gains at most 1.5 points over the best single judge with cross-fitted thresholds, and at most 2.0 even with oracle thresholds.

• The paper evaluates whether Jev, a typed classifier, can replace LLM-based rubric judges in terms of cost and accuracy.
• Jev is found to be substantially cheaper and faster, with comparable accuracy on binary criteria, but less accurate on graded criteria compared to LLM judges.
• Jev's confidence estimates enable selective prediction and cost-saving cascades, though shared errors between judges limit accuracy gains.

Research question and experimental design

The paper evaluates whether Jev, a typed classifier that returns probabilities over a fixed answer set without generating text, can replace conventional LLM-based rubric judges. The comparison is framed around criterion-level evaluation: each judged submission is paired with an individual rubric criterion, and the judge must return either a binary verdict or an ordinal level. The central claim is not that Jev universally matches LLM judges, but that its substantially lower computational cost may justify its use when accuracy is comparable and when a more expensive judge can handle uncertain cases.

The study compares Jev with three flash-tier LLM judges—GPT-5.6 Luna, Gemini 3.8 Flash, and DeepSeek V4.1 Flash—using identical criterion texts and nine evaluation panels drawn from seven public benchmarks. The panels contain 5,003 judged pairs in total: two binary panels, RiceChem and HealthBench, and seven graded panels covering essays, dialogue turns, complete dialogues, assistant responses, and long-form answers. The graded panels contain 3,778 pairs. The judges are evaluated against benchmark labels, rater references where multiple individual ratings are available, and one another.

Jev is tested through multiple typed interfaces. Jev Noul returns a binary probability, Jev Choice returns a categorical distribution and uses the most probable option, and Jev Score returns an expected ordered level that is rounded to a discrete category. The main matched comparison uses Jev Choice, while Jev Score provides a secondary analysis of how the classifier’s interface and decoding rule affect ordinal judgments. The LLM judges operate through AutoRubric, with one API call per criterion pair. By contrast, Jev receives all criteria for a unit in one request, producing the structural cost advantage that motivates the study.

The statistical protocol appropriately distinguishes separation, equivalence, and inconclusive comparisons. Paired accuracy differences are estimated through unit-level bootstrap resampling, thereby accounting for correlations among criteria belonging to the same submission. The authors apply Holm correction over the 27 comparisons between Jev Choice and an LLM judge. For graded criteria, they report exact accuracy, within-one accuracy, QWK, and signed offset rather than relying on correlation alone. This is important because a judge can preserve ordinal ranking while systematically shifting predictions toward lower or higher levels.

Accuracy relative to LLM judges

The primary accuracy result is conditional on criterion type. Jev performs strongly on the two binary panels. On RiceChem, Jev Choice reaches 81.0% verdict accuracy, compared with 77.8% for Luna, 76.1% for Gemini, and 79.2% for DeepSeek. On HealthBench, it reaches 77.1%, behind Gemini at 79.6% but ahead of Luna at 70.4% and slightly ahead of DeepSeek at 76.4%. Jev Choice is therefore the most accurate matched judge on RiceChem and the second-most accurate on HealthBench.

The corresponding result on graded criteria is less favorable. Jev Choice is never the most accurate matched judge on a graded panel. Gemini is best or tied for best on every graded panel except LFQA, while Jev Choice generally ranks between second and fourth. The largest difference occurs on ELLIPSE, where Gemini reaches 29.8% exact accuracy and Jev Choice 13.4%, a paired advantage of 16.4 percentage points for Gemini with a 95% interval from 14.1 to 18.7 points. Gemini also leads Jev Choice on FED-Dialogue by 7.9 points in the uncorrected paired comparison, and its leads on ELLIPSE and FED-Dialogue survive Holm correction.

Across all 27 comparisons, only eight show statistically separated accuracy differences after the study’s inferential procedure; the remainder are either at parity under the prespecified equivalence criterion or inconclusive. Among the 21 graded comparisons, 16 do not separate Jev Choice from the corresponding LLM judge. This pattern does not establish broad equivalence: many intervals are too wide to demonstrate either separation or parity. The paper is therefore careful to distinguish “not detectably different” from “equally accurate.”

The graded results also expose weaknesses in exact accuracy as a standalone measure. On HelpSteer2, Jev Choice scores 48.6%, while a constant no-read baseline reaches 60.0%. The constant predictor is significantly more accurate than every judge except Gemini in pair-level sign tests. On LFQA, a provenance-only baseline reaches 68.1% exact accuracy, comparable to or better than the judges, although all judges achieve higher QWK than the baseline. These outcomes indicate that some benchmark labels encode source or distributional information that the rubric criteria do not fully specify. They also show that a judge can exhibit useful ranking behavior without improving exact agreement with labels.

Jev Score produces a materially different ordinal behavior from Jev Choice. It is significantly more accurate than Jev Choice on ELLIPSE, FED-Turn, FED-Dialogue, and USR-Topical-Chat, with the largest difference on FED-Turn: 66.7% versus 56.2%, a 10.5-point advantage. Jev Score also has higher within-one accuracy on all seven graded panels. On FED-Turn, it exceeds every matched judge in exact accuracy, within-one accuracy, QWK, and mean absolute error. The difference is primarily attributable to the probability distributions produced by the two Jev primitives rather than to the decoding rules. This result qualifies any model-level characterization of Jev: the choice of typed primitive is itself an important part of the evaluator.

Cost and latency

The operational advantage of Jev is large and consistent. Across all nine panels, Jev Choice costs $0.0633, compared with$1.844 for Luna, $4.183 for DeepSeek, and$20.587 for Gemini. The corresponding cost ratios are approximately 29 times, 66 times, and 325 times. At the panel level, the ratio ranges from 18 times for Luna on HealthBench to 770 times for Gemini on FED-Dialogue.

Wall-clock time shows a similar pattern. Jev Choice completes the nine-panel evaluation in 28.6 seconds, compared with 859 seconds for Luna, 950 seconds for Gemini, and 6,298 seconds for DeepSeek. Thus, Jev is approximately 30 to 220 times faster under the reported concurrency configuration. The advantage grows with the number of criteria per unit because the LLM judges make one call per criterion while Jev evaluates all criteria for a unit in a single request.

These figures are tied to the particular implementation and pricing conditions. The LLM judges generate reasoning tokens, especially Gemini, whose reasoning tokens account for 70.4% of its ELLIPSE bill. An LLM judge that evaluates all criteria in one call, or one configured not to generate reasoning tokens, could reduce the ratios. Conversely, Jev’s cost advantage is especially relevant to per-criterion grading systems in which the unit-by-criterion product dominates evaluation volume. The paper therefore establishes a strong systems result, but not a model-independent price law.

Shared disagreement with human labels

The most consequential result concerns the relationship among the four judges and the human labels. On every graded panel, the judges agree more closely with one another than with the labels. The mean judge–judge QWK exceeds the mean judge–label QWK on all seven panels. On five panels, even the least-agreeing pair of judges agrees more closely than the best judge agrees with the labels.

This pattern is not interpreted as evidence that the judges are collectively correct. Rather, it indicates that the judges share systematic departures from the labeling process. On 31 of 35 graded criteria, the four matched judges have negative offsets, meaning that they assign lower levels than the labels. On six of seven graded panels, every judge’s mean level is below the human label mean. The effect is particularly pronounced on ELLIPSE: Jev Choice has an offset of -1.25 levels, Gemini -0.77, Luna -1.28, and DeepSeek -1.16.

ELLIPSE provides the clearest example of scale compression and downward displacement. The labels assign 36.3% of essay-trait pairs to levels 4 or 5, whereas no judge assigns more than 4.2% of predictions to those levels. Jev Choice is below the rounded label on 86.1% of pairs and above it on only 0.5%. Its exact accuracy is only 13.4%, compared with 51.4% for the rater reference. A post-hoc upward shift of one level increases Jev Choice’s accuracy to 50.4%, close to the rater reference. Across criteria, a leave-one-out constant shift eliminates most of the judge–reference gap: no criterion remains more than 8.4 points below the unshifted reference after shifting.

The implication is that much of the apparent evaluation failure is locational rather than purely discriminative. The judges often preserve ordering information. For example, the aggregate of Jev Choice’s six ELLIPSE trait scores correlates with the holistic essay score at Spearman $\rho = 0.70$, and its correlation with the unrounded rater mean is comparable to the agreement between individual raters on some traits. Nevertheless, a rank-preserving shift is consequential if the scores are used as absolute labels, reward signals, or threshold decisions. Correlation and QWK can therefore conceal systematic scale misalignment.

The paper proposes several observational explanations for this shared offset. ELLIPSE raters may have applied a population norm learned through training, judging essays relative to grade 8–12 English learners rather than relative to an abstract standard of proficiency. On USR, identical criterion texts produce different judge–label offsets across Topical-Chat and PersonaChat. For the “engaging” criterion, the offset is small on Topical-Chat but strongly negative on PersonaChat; shifting predictions upward by one level raises Jev Choice’s PersonaChat exact accuracy from 16.9% to 69.5%. This is consistent with a corpus-relative standard, although the two corpora differ in additional respects.

Other examples suggest annotation conventions not represented in the criterion text. On USR-Topical-Chat, raters mark the criterion “uses knowledge” as met even when the context contains no fact, whereas all judges interpret the absence of a fact as failure. On “understandable,” raters penalize self-contradiction despite a criterion sentence that explicitly excludes general well-formedness, while the judges follow the literal criterion. On LFQA factuality, raters assign the absolute top level to answers containing factual errors that every judge identifies, suggesting that the labels incorporate a more permissive annotation habit than the rubric wording implies.

These accounts remain observational. The study did not provide exemplars, restore the original rater instructions, or experimentally vary the criterion descriptions. Shared training priors, the LLM harness prompt, model-specific conservatism, and the authors’ rubric rewrites remain viable alternative explanations. The evidence supports a diagnosis of shared scale and convention mismatch, not a definitive causal attribution.

Confidence, abstention, and cascades

Jev exposes a confidence estimate that can be used for selective prediction. On six of the seven graded panels, confidence ranks Jev’s own correctness with AUROC values between 0.57 and 0.70. Error rates generally decline in higher-confidence bands. On FED-Turn, for example, Jev’s error rate falls from 52.8% in the lowest confidence band to 23.2% in the highest; on LFQA it falls from 43.8% to 13.5% across the principal confidence bands.

This property is not universal. On ELLIPSE, confidence has AUROC 0.489 and the error rate remains approximately 86–87% across the substantial confidence bands. Confidence also fails on specific criteria, including LFQA factuality with AUROC 0.413 and USR-Chat PersonaChat engaging with AUROC 0.523. The paper consequently recommends criterion-level validation before confidence-based routing. A panel-level confidence statistic is insufficient when a small number of criteria dominate the error distribution.

Jev’s abstention behavior is also informative but depends on the answer framing. Jev Choice abstains on 28 of 3,414 ordinal pairs, including 17 FED-Dialogue pairs for “error recovery.” The probability assigned to the NA option correlates with the number of human raters who selected N/A, with Spearman $\rho = 0.62$. This suggests that the typed classifier can detect criterion applicability or conditional evaluability. By contrast, Jev does not select CANNOT_ASSESS on any of the 8,798 binary pairs, nor do the LLM judges. The graded NA option and binary CANNOT_ASSESS option therefore induce different abstention regimes.

The paper’s cascade analysis tests whether Jev can serve as a cheap first-stage judge and defer low-confidence pairs to an LLM. The result is mixed. Cascades often reduce cost substantially. With the best LLM judge as fallback, cross-fitted cascades cost 16–48% as much as the LLM judge on six panels, though their held-out accuracy is 0.2–1.8 points lower. Oracle thresholds can match the best LLM judge at 10–59% of its cost on several panels. On ELLIPSE and FED-Dialogue, however, nearly every pair must be deferred because Jev’s accuracy is too low.

The accuracy gains are small. With cross-fitted thresholds, the largest average improvement over the best single judge is 1.5 points on HealthBench. With oracle thresholds, the largest improvement is 2.0 points, also on HealthBench. Six of nine panels show a cross-fitted decline relative to the best single judge, with the largest decline reaching 7.3 points on FED-Turn. On RiceChem, where Jev alone is stronger than the LLM alternatives, the cascade costs only 6.6% as much as the best LLM judge and improves accuracy by 1.2 points.

The limiting factor is correlated error. Among 84 high-confidence Jev errors on the graded panels, the LLM judges repeat Jev’s wrong verdict 242 times out of 252, compared with 126.7 repetitions under the study’s independence baseline. Thus, 96.0% of LLM verdicts reproduce Jev’s answer on this sample, even though Jev is wrong. The three-LLM median also reproduces Jev’s wrong answer on 82 of the 84 confident errors. The cheap and expensive judges therefore fail on many of the same pairs, limiting the complementary information available to a cascade or jury.

Limitations and open questions

The evaluation tests one typed classifier, one alias of Jev, and three flash-tier LLM configurations. The LLM judges use default harness prompts, no few-shot exemplars, and provider-dependent decoding behavior; Luna operates at an enforced temperature of 1.0. The study does not measure the best achievable performance of an LLM judge under prompt optimization, exemplar conditioning, criterion-specific calibration, or multi-criterion batching. Jev’s own graded wrapper was not tested, and the influence of jointly presenting multiple criteria in one Jev request was not isolated.

The model runs are also imperfectly reproducible. The main graded evaluation uses one pass per judge. Jev was rerun only on binary panels, where approximately 3% of RiceChem pairs changed verdict across repeated passes. A Luna spot check agreed on 71 of 82 repeated judgments. Gemini and DeepSeek were not rerun, and DeepSeek calls were distributed across multiple upstream providers. Model aliases are not dated snapshots, so the reported results are tied to a particular operational state rather than a permanently specified model.

Several analyses are post hoc. The 5-point equivalence margin was selected after verdict collection, threshold choices for oracle cascades use evaluation data, and the scale shifts are optimized against the labels. Cross-fitting reduces, but does not eliminate, the variance and selection issues because some folds contain as few as 12 units. Per-criterion confidence results are based on small samples, and the most-confident-error analysis selects only 12 errors per graded panel.

The benchmark labels introduce additional uncertainty. Nearly half of ELLIPSE labels depend on half-up rounding of two-rater means, and the rater reference does not represent a fixed upper bound on machine performance. HelpSteer2 correctness labels track overall helpfulness at Pearson $r = 0.94$, while LFQA provenance alone performs competitively with the judges. These properties complicate interpretation of “human agreement” as a single ground-truth target. The candidate conventions are plausible but untested: exemplars, original benchmark instructions, alternative scale descriptions, and intervention on absolute top-level language could distinguish missing annotation context from shared model priors.

Finally, the panels are English-only, four of seven graded panels involve dialogue, and the longest graded unit contains 1,114 words. The reported prices and latencies are deployment-specific and may change with provider pricing, concurrency, caching, and reasoning defaults. The findings therefore establish a strong comparative result for this protocol, not a universal ranking of typed classifiers and LLM judges.

Conclusion

The paper shows that Jev can replace flash-tier LLM judges on some binary rubric evaluations at a dramatic reduction in cost and latency. On RiceChem it is the most accurate matched judge, and on HealthBench it is close to Gemini while costing a small fraction as much. On graded criteria, however, Jev Choice trails the best LLM judge on several panels, with a particularly large deficit on ELLIPSE.

The more general finding is that judge disagreement with human labels is often shared. All judges tend to assign lower graded levels than the raters, and much of the discrepancy can be represented as a criterion-specific location shift. This supports reporting signed offsets and judge–judge agreement alongside ordinary accuracy or correlation. Jev’s confidence enables cost-saving cascades on many panels, but correlated errors sharply limit accuracy gains: the LLM judges frequently reproduce Jev’s most confident mistakes. The paper consequently supports Jev as a low-cost evaluator and first-stage router under validated conditions, while showing that adding nominally independent judges does not by itself resolve rubric ambiguity or annotation conventions.