Robustness of synthetically generated rubric criteria as scoring anchors

Determine whether the synthetically generated, non–expert-verified rubric criteria used in DeepResearch-Bench RACE are robust anchors for scoring large language model responses.

Background

The paper critiques DeepResearch-Bench RACE for relying on synthetically generated criteria that were not verified by expert humans. The authors argue that such criteria may introduce bias—particularly favoring Gemini-2.5-Pro because both the evaluation criteria and the reference high-quality answer were produced by that model—and question the reliability of these criteria as stable anchors for fair scoring.

ProfBench is proposed as a remedy, with rubrics curated by domain experts across Physics, Chemistry, Finance, and Consulting. The authors emphasize that robust, expert-validated criteria are essential for fair and effective evaluation, highlighting uncertainty around the robustness of synthetic criteria as a key unresolved question in current benchmark design.

References

It also falls short due to synthetically generated criteria that were not verified by expert humans---meaning that it's unclear if such criteria are robust anchors for scoring.

— ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge  (2510.18941 - Wang et al., 21 Oct 2025) in Introduction, Section 1

A limitation inherent to using rubrics as rewards on unverifiable tasks is that there is no independent signal against which to check whether the criteria describe it faithfully. Validating the criteria would require human raters, which we leave to future work.

— DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training  (2609.04094 - Gandhi et al., 3 Sep 2026) in Section 'Limitations'

Generating them from the reference solution is the obvious step, but generated rubrics fell short of expert ones in the comparisons made so far, which generated them without a reference solution \citep{wang2026generating, zhou2026rubricbench}, the one legal set generated from expert solutions was only partly reviewed \citep{oh2026korean}, and whether expert review can close the gap at acceptable cost is open; rubrics refined against human-scored answers matched detailed human-written ones in essay scoring \citep{harada2025automated}, a route to which the caveat of S5 applies.

— SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges  (2610.19992 - Nagl, 14 Oct 2026) in Section 6, “Open problems,” subsection “Who writes the rubric, and how is a generated one judged?”