Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks

Abstract: Recently, rubrics have been used to guide LLM judges in capturing subjective, nuanced, multi-dimensional human preferences, and have been extended from evaluation to reward signals for reinforcement fine-tuning (RFT). However, rubric generation remains hard to control: rubrics often lack coverage, conflate dimensions, misalign preference direction, and contain redundant or highly correlated criteria, degrading judge accuracy and producing suboptimal rewards during RFT. We propose RRD, a principled framework for rubric refinement built on a recursive decompose-filter cycle. RRD decomposes coarse rubrics into fine-grained, discriminative criteria, expanding coverage while sharpening separation between responses. A complementary filtering mechanism removes misaligned and redundant rubrics, and a correlation-aware weighting scheme prevents over-representing highly correlated criteria, yielding rubric sets that are informative, comprehensive, and non-redundant. Empirically, RRD delivers large, consistent gains across both evaluation and training: it improves preference-judgment accuracy on JudgeBench and PPE for both GPT-4o and Llama3.1-405B judges, achieving top performance in all settings with up to +17.7 points on JudgeBench. When used as the reward source for RFT on WildChat, it yields substantially stronger and more stable learning signals, boosting reward by up to 160% (Qwen3-4B) and 60% (Llama3.1-8B) versus 10-20% for prior rubric baselines, with gains that transfer to HealthBench-Hard and BiGGen Bench. Overall, RRD establishes recursive rubric refinement as a scalable and interpretable foundation for LLM judging and reward modeling in open-ended domains.

• The paper introduces Recursive Rubric Decomposition (RRD) to improve LLM judge accuracy by generating informative, comprehensive, and non-redundant rubrics.
• It employs a three-stage process—rubric proposal, recursive refinement, and correlation-aware weighting—to systematically address rubric misalignment and redundancy.
• Empirical results demonstrate significant accuracy gains and reward improvements, showcasing RRD's scalability and effectiveness in open-ended tasks.

Principled Rubric Generation for LLM Judging and Reward Modeling in Open-Ended Tasks

Introduction and Motivation

Rubric-based LLM evaluation has gained traction as a means of systematizing the assessment of open-ended generations. Unlike holistic LLM-as-a-judge approaches that provide a monolithic verdict, rubric-based evaluation seeks to enhance accuracy, transparency, and alignment with multidimensional human preferences by scoring responses against a set of distinct, explicit criteria. However, the process of rubric generation remains largely heuristic or single-shot, resulting in rubrics that are incomplete, conflated, misaligned, or redundant. This, in turn, limits both judge accuracy and the reliability of rubric-based rewards when used for downstream reinforcement fine-tuning (RFT).

The paper "Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks" (2602.05125) addresses these challenges via Recursive Rubric Decomposition (RRD), a theoretically grounded, algorithmic approach for generating rubric sets that are informative, comprehensive, and non-redundant. The approach is validated across judge accuracy and RFT learning metrics on open-ended domains, demonstrating substantial empirical gains and providing a scalable foundation for robust model alignment.

The RRD Framework

RRD formulates rubric generation as a recursive, three-stage process designed to systematize the discovery and optimization of evaluation criteria.

Stage I: Initial Rubric Proposal

An LLM is prompted to propose a candidate set of rubrics, conditioned on the task prompt and sampled responses. This enables immediate surface-level coverage grounded by actual generational diversity, addressing the common failure modes of prompt-only rubric design.

Stage II: Recursive Decomposition and Filtering

Rubrics satisfied by an excessive number of responses are recursively decomposed by the LLM into finer subcriteria, each vetted for discriminative power and uniqueness. Redundant (i.e., highly correlated or overlapping) and misaligned (i.e., reverse-polarity) rubrics are filtered using counterfactual preference checks and redundancy classifiers. The recursion halts once a saturation threshold—defined by the number of rejected proposals—signals diminishing returns in novelty.

Figure 1: The RRD framework recursively decomposes coarse rubrics, filters misaligned and redundant criteria, and assigns correlation-aware weights for robust aggregation.

Stage III: Rubric Weight Assignment

For tasks characterized by multiple, distributed preference axes, simple averaging or LLM-heuristic weighting fails due to criterion correlation and the absence of a clear dominant signal. RRD employs a correlation-aware scheme, assigning “whitened uniform” (WU) weights that homogenize the signal in whitened feature space, efficiently down-weighting correlated criteria and preventing overrepresentation.

This process yields rubrics tailored to the intrinsic complexity of each evaluation instance, with explicit control over informativeness, coverage, and statistical independence—key desiderata articulated and formalized in the theoretical analysis.

Theoretical Underpinnings

RRD is informed by an exponential upper bound on judge misclassification probability in terms of rubric edge and correlation structure. Specifically, the paper models each rubric’s contribution to preference determination under assumptions of positive edge and bounded pairwise correlation, deriving:

$$\Pr(\text{misclassification}) \leq \exp\left(-\min\left\{\frac{\Delta_m^2}{2V_m(+1)},\,\frac{\Delta_m^2}{2V_m(-1)}\right\}\right)$$

where $\Delta_m$ is aggregate rubric edge and $V_m$ is the variance proxy capturing rubric correlation. This formally motivates recursive expansion (to increase $\Delta_m$), misalignment and redundancy filtering (to minimize $V_m$), and correlation-aware weighting. Notably, when no dominant criterion exists, correlation-aware normalization (via the WU scheme) provably stabilizes aggregation, consistent with empirical improvements in judge reliability.

Empirical Results: Judge Accuracy

The effectiveness of RRD is evaluated on open-ended judging benchmarks—JudgeBench and PPE Preference—using both proprietary (GPT-4o) and open-weight (Llama3.1-405B) models.

• Substantial accuracy gains: On JudgeBench, $RRD_{\textrm{WU}}$ improves GPT-4o accuracy from $55.6\%$ (base) to $73.3\%$, a $+17.7$ point increase. Consistent improvements are observed across models and datasets.
• Mitigation of rubric noise: Naive LLM-generated rubrics, when not grounded in sample responses, degrade judge accuracy, while RRD recovers and exceeds base performance.

  Figure 2: RRD variants, especially with WU weighting, significantly outperform base and prior rubric-based judges in both accuracy and stability.

RRD also demonstrates adaptability: rubric counts grow and then plateau per instance, reflecting task complexity and avoiding over-decomposition (Figure 2b).

Ablation studies confirm that the WU weighting remains robust to deeper recursive exploration and that using diverse, high-quality sample generators during rubric proposal further increases accuracy.

Effectiveness for RL-Based Reward Modeling

When deployed as the basis for RLHF/RFT reward, RRD enables much stronger and more stable learning signals compared to baselines.

• Substantially higher reward improvements: On RFT with Qwen3-4B, policies trained with $RRD_{\text{WU}}$ achieve up to 160% reward improvement, compared to just 10–20% for naïve rubric or iterative baselines; Llama3.1-8B sees 60% improvement.
• Stability and continued reward climb: RRD-based rewards avoid early saturation and maintain learning progression throughout RFT.

Figure 3: RRD-derived rubrics yield consistently higher and more stable reward trajectories during RFT for both Qwen3-4B and Llama3.1-8B architectures.

Generalization and Downstream Policy Performance

Models fine-tuned with RRD-derived rewards exhibit strong generalization, as shown by evaluation on both in-domain (BiGGen Bench) and out-of-domain, high-stakes (HealthBench-Hard) benchmarks.

• State-of-the-art results on multiple axes: $RRD_{\text{WU}}$ policies outperform all baselines along Instruction Following, Completeness, and Reasoning without safety trade-off.
• Transfer to domain-specific evaluation: In clinical dialogue (HealthBench-Hard), RRD achieves substantial improvements in IF, accuracy, and completeness, confirming granular alignment with expert-authored rubrics.

  Figure 4: RRD-trained models demonstrate superior multidimensional scores across BiGGen Bench and HealthBench-Hard, showcasing robustness and transferability.

Practical and Theoretical Implications

The RRD framework provides a scalable, algorithmic alternative to hand-crafted or passively generated rubrics for both LLM judging and reward modeling. Its recursive structure adapts naturally to differing evaluation complexities, ensuring comprehensive yet parsimonious rubrics per instance. The WU weighting is significant for multi-criteria aggregation without labeled edge estimation, especially for open-ended, non-verifiable domains where human-like preference structure is inherently multidimensional.

Practically, RRD enables more reliable and interpretable model assessment, improved alignment in RLHF/RFT pipelines, and enhanced generalization across domains. Theoretically, it motivates further exploration of recursive, information-theoretic approaches to emergent property detection in LLM evaluation and reward specification.

Conclusion

Recursive Rubric Decomposition operationalizes a theoretically justified, multi-stage rubric generation pipeline that systematically addresses coverage, alignment, and redundancy in rubric-based LLM evaluation and fine-tuning. Empirical results demonstrate strong superiority over baselines in both evaluation accuracy and as reinforcement sources for open-ended generation tasks. By grounding the structure and aggregation of rubrics in rigorous statistical analysis, RRD represents a step toward scalable, interpretable, and reliable alignment for next-generation LLMs.

arXiv reference: (2602.05125)