Assess cross-criterion influence within a single Jev request

Determine whether presenting multiple criterion questions for the same unit in one Jev request causes the questions to influence one another’s answers.

Background

Jev receives all of a unit’s criteria together in a single structured request, whereas the LLM judges are called separately for each pair. This difference could affect the independence and accuracy of Jev’s criterion-level verdicts.

The study did not experimentally isolate this factor. A comparison between joint multi-criterion requests and otherwise identical one-criterion requests would be needed to determine whether criterion co-presentation changes Jev’s answers.

References

Jev receives a unit's criteria together in a single request, and we did not test whether the questions of one request influence one another's answers.

— JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places  (2609.29769 - Rao et al., 24 Sep 2026) in Limitations, paragraph “Judge configuration” (Section \ref{sec:limitations})