Online Rubrics Elicitation from Pairwise Comparisons (2510.07284v1)

Abstract: Rubrics provide a flexible way to train LLMs on open-ended long-form answers where verifiable rewards are not applicable and human preferences provide coarse signals. Prior work shows that reinforcement learning with rubric-based rewards leads to consistent gains in LLM post-training. Most existing approaches rely on rubrics that remain static over the course of training. Such static rubrics, however, are vulnerable to reward-hacking type behaviors and fail to capture emergent desiderata that arise during training. We introduce Online Rubrics Elicitation (OnlineRubrics), a method that dynamically curates evaluation criteria in an online manner through pairwise comparisons of responses from current and reference policies. This online process enables continuous identification and mitigation of errors as training proceeds. Empirically, this approach yields consistent improvements of up to 8% over training exclusively with static rubrics across AlpacaEval, GPQA, ArenaHard as well as the validation sets of expert questions and rubrics. We qualitatively analyze the elicited criteria and identify prominent themes such as transparency, practicality, organization, and reasoning.

• The paper introduces OnlineRubrics, a framework that dynamically augments rubrics using LLM-based pairwise response comparisons during RL policy training.
• It employs an LLM-driven extractor for criteria elicitation, deduplication, and reward calculation across both generalist and expert domains.
• Experimental results demonstrate significant performance improvements over static baselines, with gains up to 25% on select benchmarks.

Online Rubrics Elicitation from Pairwise Comparisons: A Technical Analysis

Motivation and Context

Rubric-based reward modeling has become a central paradigm for post-training LLMs in open-ended domains where verifiable rewards are infeasible. While static rubrics—human-authored or synthetic—provide interpretable, multi-objective signals, they are inherently limited in coverage and susceptible to reward hacking. The "Online Rubrics Elicitation from Pairwise Comparisons" paper introduces OnlineRubrics, a framework for dynamic, online augmentation of rubrics via LLM-driven pairwise response comparisons. This approach is motivated by the need to continuously surface emergent errors and desiderata during RL-based policy optimization, thereby improving both alignment and sample efficiency.

Figure 1: OnlineRubrics workflow: at each training step, a pair of responses (current policy vs. control) is compared, new criteria are elicited and deduplicated, and the augmented rubric is used for reward calculation in policy gradient updates.

Methodology

Training Setup and Reward Formulation

OnlineRubrics is implemented atop the GRPO algorithm, which optimizes the following clipped policy gradient objective:

$$\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E}_{i, j}\left[ \min \left( r_{i,j}(\theta) \hat{A}_{i,j}^{\text{group}}, \text{clip}(r_{i,j}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_{i,j}^{\text{group}} \right) - \beta \mathbb{D}_{KL}(\pi_\theta || \pi_{ref}) \right]$$

where rewards $R_j$ are computed as weighted sums of binary grades from an LLM-based grader over the rubric criteria:

$$R_j = q(\text{LLM}_\text{grader}(o_j, x_i, \mathcal{C}_i))$$

with $q$ as the normalized weighted sum reduction.

Online Criteria Elicitation

At each training step, for each prompt, the system samples $M$ candidate responses from both the current policy and a control policy (either the reference or previous policy). For each pair, an LLM-based extractor (conditioned on a system prompt) identifies discriminative differences and proposes new criteria with associated weights. These criteria are deduplicated and merged with the existing rubric, forming the basis for reward calculation in the next policy update.

Figure 2: Abbreviated system prompt template for LLM-based criteria elicitation from pairwise response comparisons.

The process is formalized in Algorithm 1, which details sampling, extraction, deduplication, reward computation, and policy update steps.

Theoretical Justification

The paper provides a formal bound on the gradient estimation error induced by missing criteria in the rubric. Specifically, the difference between the true and estimated policy gradients is upper-bounded by the $\ell_1$ norm of the weights of unmodeled criteria, scaled by the expected squared norm of the policy score function. This motivates the continuous augmentation of rubrics to minimize implicit reward mass and improve training stability.

Dataset Construction and Grader Selection

Two datasets are curated: Generalist Rubrics (real-world, single-turn prompts with human-authored rubrics) and Expert Rubrics (domain-specific, expert-authored rubrics in Physics, Chemistry, Biology, and Math). Each rubric consists of weighted, binary-checkable criteria.

Figure 3: Example data sample from the Generalist Rubrics dataset.

Figure 4: Example data sample from the Expert Rubrics dataset.

LLM graders are benchmarked for rubric evaluation capability using human-annotated data. AUC scores and inference costs are compared, with GPT-4.1-mini selected as the default grader for its Pareto-optimal trade-off.

Figure 5: AUC vs. inference cost for LLM graders; models on the Pareto frontier are optimal for rubric grading.

Experimental Results

Baselines

The paper compares OnlineRubrics against several baselines:

• LLM-Judge Score (Likert scale grading)
• Offline Rubrics (Synthetic and Human-written)
• Universal Requirements (fixed criteria)
• Pointwise Elicitation (criteria extracted from single responses)

Main Findings

OnlineRubrics consistently outperforms all baselines across both generalist and expert domains, as well as public benchmarks (AlpacaEval, GPQA-Diamond, Arena-Hard, GSM8K). Notably, OnlineRubrics yields up to 8 percentage point improvements over static human-written rubrics on AlpacaEval and GPQA, and up to 25% absolute gains over the initial instruct model on select benchmarks.

Figure 6: Training curves on Generalist and Expert datasets; OnlineRubrics variants outperform static rubrics and LLM-Judge scores.

The pairwise elicitation approach is superior to pointwise extraction and universal requirements, indicating that sample-specific, discriminative criteria are more effective than static or generic augmentations.

Qualitative Analysis of Elicited Criteria

Cluster analysis of elicited criteria reveals several recurring themes:

• Evidence grounding and reproducibility
• Anti-gaming (avoiding over-specification)
• Practicality and real-world feasibility
• Structural organization, causal reasoning, and uncertainty handling

  Figure 7: Top-10 most frequent clusters of rubric criteria elicited via OnlineRubrics, sorted by sample proportion.

OnlineRubrics adaptively expands rubric coverage, surfacing both overlooked errors and emergent desired properties, leading to more comprehensive and resilient evaluation standards.

Implementation Considerations

Computational Requirements

Experiments are conducted on Qwen-2.5-7B-Instruct, trained with GRPO for three epochs, using 8 NVIDIA H100 GPUs. Each training step involves multiple LLM inference calls for both grading and criteria extraction, making inference cost a critical factor in grader selection.

Deployment Strategies

OnlineRubrics can be integrated into any rubric-based RL pipeline. The LLM-based extractor and deduplication prompts are modular and can be adapted to different domains or grader models. The framework is compatible with both human-written and synthetic rubrics, and supports dynamic augmentation during training.

Limitations

• The quality of elicited criteria depends on the extractor LLM and prompt engineering.
• Deduplication and clustering require careful design to avoid redundancy and drift.
• The approach does not guarantee full recovery of the true criteria set, but tightens the bound on unmodeled reward mass.

Implications and Future Directions

OnlineRubrics advances the state of multi-objective RLHF by enabling dynamic, sample-specific rubric augmentation. This mitigates reward hacking, improves alignment, and enhances sample efficiency. The framework is extensible to other domains and can be combined with advanced reward modeling techniques (e.g., conditional reward models, multi-gradient descent).

Future work may explore:

• Automated rubric refinement via meta-learning
• Integration with verifiable reward signals for hybrid supervision
• Scaling to multi-turn and conversational settings
• Robustness to adversarial or noisy criteria extraction

Conclusion

OnlineRubrics provides a principled and empirically validated framework for online rubric elicitation via pairwise response comparisons. By continuously augmenting evaluation criteria during RL-based LLM training, it addresses the limitations of static rubrics and improves alignment across diverse domains. The approach demonstrates strong quantitative gains and qualitative improvements in rubric coverage, with clear implications for scalable, interpretable, and robust LLM post-training.