Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Abstract: Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins $24$ of $30$ base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in $2.5$--$4\times$ fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.

• The paper introduces POW3R, a dynamic rubric weight reallocation strategy that enhances RLVR efficiency by concentrating on unsaturated, informative criteria.
• It employs per-criterion variance assessment and within-category reweighting to adapt reward signals while preserving human-designed rubric structure.
• Empirical results show 2.5–4× faster convergence and superior strict completion rates across multimodal and text-based tasks.

Policy-Aware Rubric Rewards for Reinforcement Learning with Verifiable Rewards

Motivation and Challenges of Rubric-Based RL

Reinforcement learning with verifiable rewards (RLVR) has driven advances in post-training alignment for LLMs and MLLMs by utilizing reward signals that are robustly checked for correctness via deterministic verifiers. However, many high-value tasks—such as medical advice, scientific generation, and visually grounded reasoning—require a more nuanced evaluation that decomposes output quality into multiple qualitative criteria, typically operationalized as a rubric. In this regime, the reward becomes a function of the model’s performance over several prompt-specific and weighted rubric items, often independently judged by LLMs.

A widely adopted practice is aggregating rubric scores into a single scalar using fixed human-assigned weights. This approach implicitly presumes that these weights reflect not only the desired downstream importance (evaluation) but also their utility for driving optimization in the current policy—a conflation shown to be statistically and empirically unsound in RLVR. The central insight of "Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR" (2605.20164) is that many crucial criteria are either saturated (always passed), dead (never passed), or do not provide contrastive signal for policy updates. As a result, a non-trivial fraction of the training signal is wasted along non-informative axes, leading to inefficient and sometimes ineffective learning.

Figure 1: Rubric-pressure diagnostic showing that static aggregation directs significant pressure to saturated and dead criteria, decoupling importance from learnability.

POW3R: Policy-Aware Reward Aggregation

The paper introduces Policy-Aware Rubric Reward (POW3R), a rubric reward aggregation framework designed to address the disconnect between static human-assigned weights and dynamically informative training signals within rubric-based RL. POW3R operates atop the standard Group Relative Policy Optimization (GRPO) regime. Rather than altering the underlying rubric definitions or their human-assigned weights (which encode task desiderata and evaluation protocols), POW3R adaptively re-allocates within-category reward weights based on rollout-level contrastiveness (variance). Its essential components are:

• Per-Criterion Rollout Variance: For each criterion, POW3R measures the empirical variance in pass/fail judgments across rollouts, identifying which criteria currently serve as informative differentiators.
• Within-Category Reweighting: Each criterion weight is dynamically scaled based on its relative variance within its category, using a stabilized and bounded procedure (EMA smoothing, min/max clipping, and blend with the human prior).
• Category Mass Preservation: Total category weight remains unchanged, preserving rubric design intent and allowing direct comparison in evaluation while focusing optimization on salient axes.

The construction ensures human-preferred objectives (category balance, final evaluation semantics) are maintained, but the training signal becomes highly adaptive, concentrating updates where the policy has headroom to improve.

Experimental Setup and Rubric RL Benchmarks

POW3R is benchmarked across two scales: a proprietary multimodal dataset (MM) with 10k hand-crafted prompts, images, rubrics, and a set of text-only tasks from HealthBench (HB). Each dataset supports integer-valued, prompt-specific rubric weights and category tags. Reward judgments are LLM-based for all comparisons, using carefully calibrated judge models to balance cost and verdict fidelity.

Rollouts for policy optimization are generated from Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct, Gemma3-4B-IT (for MM), and Qwen3-4B-Instruct, Qwen3-8B, Gemma3-4B-IT (for HB). All policy training follows standardized hyperparameters, prompt sampling, and judge interfaces. Baselines include traditional binary rewards (all-or-nothing), static scalar sums, and static category-normalized weights.

Figure 2: Illustrative rubric-judging interface for both multimodal and text-only rubric RL tasks—each response is scored per prompt-specific criterion and aggregated.

Quantitative Results and Analytic Insights

POW3R demonstrates systematic improvements in both mean rubric reward and strict completion fraction (success across all required criteria), achieving top performance in 24/30 base-policy/metric runs. Notably, POW3R’s dominance is consistent across both multimodal and text-only domains, models, and metrics.

• Efficiency: POW3R reaches comparable reward thresholds in 2.5–4× fewer training steps compared to static and category-normalized rewards, directly reflecting the benefit of focusing updates on learnable criteria.
• Granular Gains: Per-category analysis shows that POW3R’s gains concentrate in categories with contrastive variance (e.g., Visual Perception, Content Completeness, Truthfulness), while categories with already-saturated criteria see little deviation from the static baselines.

  Figure 3: Main result summary—on both the MM and HB test sets, POW3R yields higher rubric rewards and strict completion rates than static and binary reward constructions; left: MM, right: HB.

  

  Figure 4: Per-category reward learning dynamics (Qwen3-VL-4B/MM). POW3R rapidly exploits available contrast and saturates improvements first in categories with the largest unsatisfied criteria.

POW3R not only raises average performance but Pareto-dominates the baselines across the two main objectives, providing both higher-quality and more consistently complete outputs.

Theoretical and Practical Implications

These results challenge the convention of fixed scalarization in multi-objective RL. Fixed rubric weights suffice for evaluation but largely fail as static training signals when policies rapidly saturate a subset of objectives. By dynamically aligning reward aggregation with the current policy’s learning frontier, one can extract maximal value from existing judicious rubric design, leading to both compute savings and higher-quality policies without sacrificing the underlying behavioral target.

In terms of practical deployment, the aggregation framework is fully decoupled from the LLM policy optimizer and from the rubric authoring pipeline, requiring no architectural changes or additional reward modeling.

Limitations and Future Directions

The current framework is dependent on LLM-based rubric judges, which can introduce bias and entanglement in both the reward signal and evaluation protocol. RUBRIC datasets with fine-grained, static, human weights across additional domains—especially outside of the visual and medical context—are scarce. This limits immediate external validity. Furthermore, co-evolving rubrics and policies, judge diversity, and longer training schedules remain fertile grounds for expansion.

Conclusion

POW3R introduces a policy-aware aggregation approach that addresses the mismatch between human-designed rubric weights and the optimization requirements of LLMs trained with RLVR. By reallocating training pressure towards informative, unsaturated criteria—and preserving evaluation semantics—the method achieves both improved sample efficiency and model quality. This work positions reward aggregation as a key, adjustable lever for multi-objective alignment in contemporary RL for LLMs and MLLMs, establishing that effective post-training requires explicit attention not just to what should matter, but to what currently can teach.