Trade-off Between Rubric Granularity and Automated Judge Reliance

Investigate the trade-off between fine-grained rubric task decomposition and reliance on an LLM-based automated judge for grading PaperBench submissions; specifically, determine how judge reliability affects the level of rubric granularity required to maintain accurate and trustworthy grading outcomes.

Background

PaperBench uses hierarchical rubrics with hundreds of leaf nodes per paper and an LLM-based judge (SimpleJudge) to grade complex replication submissions. Creating highly detailed rubrics is labor-intensive, but can improve grading precision when judges are imperfect.

The paper notes that as automated judges become more reliable, less granular rubrics may suffice, potentially reducing rubric-writing effort. However, the relationship between judge reliability and necessary rubric granularity—and its impact on grading fidelity—has not been quantified or characterized.

References

We further note that the more reliable your judge is, the less fine-grained your rubric's task decomposition needs to be, potentially reducing the effort necessary for task decomposition as judges become more capable. We leave it to future work to study the trade-off between careful specification and delegation to the judge.

— PaperBench: Evaluating AI's Ability to Replicate AI Research  (2504.01848 - Starace et al., 2 Apr 2025) in Appendix A.2, Improving automated judges

A harness that grades all of a unit's criteria in one call, or an LLM judge run without reasoning, would narrow the ratios. We costed neither, and their effect on accuracy is unknown.

— JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places  (2609.29769 - Rao et al., 24 Sep 2026) in Section 3.2, “Jev is cheaper and faster than every LLM judge” (Section \ref{sec:price})

However, we must caution that our rubric judge was never validated on imagined tasks, and so future work is warranted to validate this claim.

— Training AI Scientists to Replicate Research  (2608.13331 - Falck et al., 13 Aug 2026) in Discussion, paragraph “Towards innovation”