JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
This lightning talk examines whether Jev, a typed classifier that returns probabilities over fixed answer sets, can replace conventional large language model judges for rubric-based evaluation. The authors compare Jev against three flash-tier LLM judges across binary and graded evaluation tasks, finding dramatic cost and speed advantages but also discovering that all judges share systematic disagreements with human labels in the same ways.Script
Evaluating AI outputs with rubrics is expensive. The authors tested whether a tiny typed classifier called Jev can match the accuracy of conventional language model judges while running 30 to 300 times cheaper.
On binary tasks like chemistry reasoning, Jev outperformed every flash-tier model judge. But the real surprise came when the authors looked at graded tasks. All four judges agreed more closely with each other than with the human labels.
The judges did not just disagree randomly. On every graded panel, they systematically assigned lower scores than human raters. On essay evaluation, Jev placed 86 percent of judgments below the label and almost never above it. A simple upward shift of one level raised its accuracy from 13 percent to over 50.
Jev exposes a confidence score that can route uncertain cases to a more expensive judge. The authors tested cascades where Jev handles high-confidence pairs and defers the rest. Cost dropped to 10 to 50 percent of using the language model judge alone, but accuracy gains were tiny, rarely exceeding 2 points.
The problem is correlated error. When Jev made a high-confidence mistake, the language model judges repeated that same wrong answer 96 percent of the time. Adding a more expensive judge does not fix the problem if both judges fail in the same places.
Jev can replace flash-tier judges on validated binary tasks where it performs well and costs almost nothing. But on graded criteria, shared conventions and scale compression mean no judge is a drop-in substitute for careful rater alignment. If you want to explore this research further or generate your own video summaries, visit EmergentMind.com.
