Rubric Policy Optimization (RPO)

• RPO is a family of methods that decomposes overall rewards into multiple, explicitly defined criteria for granular policy optimization.
• It leverages techniques like on-policy distillation and multimodal generation to aggregate criterion-level judgments into a structured training signal.
• Empirical results show that RPO improves both rubric-specific feedback and broad performance, although challenges like reward hacking and rubric design remain.

Rubric Policy Optimization (RPO) denotes a family of policy-optimization methods in which the training signal is derived from an explicit rubric rather than from a single opaque scalar reward. In these systems, quality is decomposed into multiple criteria, a judge evaluates candidate outputs against those criteria, the criterion-level judgments are aggregated into a scalar training reward or a rubric-conditioned preference, and the policy is optimized against that structured signal. Recent work instantiates this pattern in language-model RL, on-policy distillation, multimodal generation, stepwise routing, and long-horizon research agents, although the term itself is not yet fully standardized across papers (Bhattarai et al., 8 May 2026, Fang et al., 8 May 2026, Mei et al., 31 May 2026).

1. Terminology and scope

In recent rubric-centered alignment work, RPO is best understood as an umbrella description rather than a single canonical algorithm. One explicit formulation describes the core idea as optimizing a policy against a rubric-decomposed, judge-computed reward instead of a single scalar preference model (Bhattarai et al., 8 May 2026). Another states the same idea as training a policy “not against scalar rewards or token-level teacher logits, but against a structured set of semantic criteria (‘rubrics’)” (Fang et al., 8 May 2026). Multimodal work uses the same label for a PPO-style procedure that converts rubric-conditioned preference decisions into binary rewards for generative training (Tian et al., 8 May 2026).

At the same time, the acronym is overloaded. Unrelated papers use RPO for Reward Partitioning Optimization, Reference-based Policy Optimization, and Reparameterization Proximal Policy Optimization, while autonomous-racing work uses $\alpha$-RPO for attenuated residual policy optimization (Faye et al., 16 Jun 2025, Wu et al., 2021, Zhong et al., 8 Aug 2025, Trumpp et al., 13 Mar 2026). This suggests that, within current rubric-based alignment literature, “Rubric Policy Optimization” functions primarily as a conceptual category: policy optimization whose reward interface is an explicit rubric.

2. Formal structure of rubric-based rewards

A canonical formalization treats each training instance as a triple

$(x,g,R),$

where $x$ is the policy-visible input, $g$ is auxiliary grounding shown only to the judge, and $R$ is a rubric with multiple criteria. In rubric-grounded RL, the policy $\pi_\theta(y\mid x)$ generates an answer $y$, while the judge $\mathcal{J}$ receives $(x,g,y,R)$ and returns criterion-level scores (Bhattarai et al., 8 May 2026).

The rubric itself is defined as

$$R=\{c_1,\dots,c_M\}, \qquad c_j=(w_j,\eta_j,\mathcal{E}_j,\kappa_j,\nu_j),$$

where $(x,g,R),$0 is a non-negative weight, $(x,g,R),$1 is a natural-language criterion description, $(x,g,R),$2 specifies required elements, $(x,g,R),$3 specifies expected keywords or terminology, and $(x,g,R),$4 specifies the verification method. If the judge returns scores $(x,g,R),$5, then the normalized reward is

$(x,g,R),$6

yielding a partial-credit scalar in $(x,g,R),$7 after normalization and clipping (Bhattarai et al., 8 May 2026).

This criterion-decomposition pattern appears in several variants. In deep-research RL, the rubric is a set of atomic constraints $(x,g,R),$8, each judged independently by a binary predicate $(x,g,R),$9, and the reward is the arithmetic mean over constraints: $x$0 This produces dense, partially creditable feedback for open-ended tasks lacking exact-match verifiers (Mei et al., 31 May 2026). In rubric-based on-policy distillation, a verifier marks each prompt-specific criterion as pass or fail and uses the weighted pass rate as the rollout reward (Fang et al., 8 May 2026). In multimodal ARR-RPO, the judge conditions on a rubric $x$1, chooses a winner between two candidates, and the RL side converts that rubric-conditioned preference into binary rewards $x$2 and $x$3 inside a PPO-style objective (Tian et al., 8 May 2026).

A recurring structural property is information asymmetry: the judge often has access to privileged grounding, references, or teacher outputs that the policy never sees. This suggests that many RPO systems exploit the easier verification problem rather than requiring the generator itself to have direct access to all evidence at inference time (Bhattarai et al., 8 May 2026, Fang et al., 8 May 2026).

3. Rubric construction and verification

Rubric quality is a central determinant of RPO behavior, and recent work treats rubric construction as a first-class modeling problem rather than a fixed preprocessing artifact.

One large-scale approach derives question–rubric pairs from roughly 100,000 OSTI scientific and technical documents. Its pipeline consists of semantic analysis, joint question–rubric synthesis, and optional rubric enrichment. The resulting rubrics have 5–10 criteria and include weight, description, required elements, expected keywords, and verification method, allowing a frozen judge to score responses against source-grounded criteria without human criterion-level annotation (Bhattarai et al., 8 May 2026).

Another line argues that rubric construction is itself a research problem. Deep Research as Rubric uses a two-stage process: Stage I performs iterative multi-turn agentic search to collect an evidence trace $x$4, and Stage II distills that evidence into atomic, independently verifiable constraints $x$5. The resulting rubrics are explicitly grounded in domain facts, structural constraints, and failure modes rather than only in a model’s prior (Mei et al., 31 May 2026).

Prompt-specific rubric induction also appears in distillation and routing. ROPD constructs rubrics from teacher–student contrasts, with criteria grouped into Task Completion, Observable Quality, and General Reasoning, weights $x$6, and a dynamically chosen rubric length typically between 4 and 12 criteria (Fang et al., 8 May 2026). RoRo learns a query-specific rubric generator (“Rubricor”) and a rubric-conditioned Judge for routing trajectories; candidate rubrics are filtered by a validation gate based on partial correlation significance, minimum score variance, and mutual information with outcome, then refined by alternating optimization of Rubricor and Judge (Ye et al., 28 May 2026).

Self-evolving rubric systems move further by making rubric generation part of training. EvoRubric unifies response generation and rubric generation under a single parameterized policy, applies a meta-verifier, zero-variance pruning, and Leave-One-Out peer consensus, and archives validated criteria in a memory pool (Guan et al., 28 May 2026). EvoRubrics similarly co-evolves a Policy LLM and a Rubric Generator within each training step, with rubric rewards based on discriminativeness, diversity, alignment, and constructiveness (Ding et al., 22 Jun 2026).

Framework	Rubric source	Verification pattern
Rubric-grounded RL	OSTI-derived synthesis	Frozen LLM judge with source grounding
DR-rubric	Agentic evidence collection + synthesis	Atomic constraint checks
ROPD	Teacher–student contrast	Verifier pass/fail over prompt-specific criteria
RoRo	Learned Rubricor	Judge + validation gate
EvoRubric	Self-evolving single policy	Meta-verifier + variance + consensus
EvoRubrics	Co-evolving rubric generator	Judge-scored rubric rewards

The common pattern is explicit decomposition of quality into inspectable criteria. The main difference lies in how those criteria are obtained: offline synthesis from documents, contrastive induction from teacher outputs, agentic evidence gathering, or online co-evolution.

4. Optimization regimes and representative variants

The optimization layer in RPO is not unique, but several recent systems repeatedly use GRPO or PPO-like objectives with rubric-derived rewards.

Rubric-grounded RL optimizes Llama-3.1-8B-Instruct with GRPO, using group-relative leave-one-out baselines and KL regularization against a frozen reference policy. The reward is the rubric-normalized partial-credit score, and the group-relative advantage compares multiple sampled responses for the same prompt under the same rubric and grounding (Bhattarai et al., 8 May 2026). Rubric-based on-policy distillation also uses GRPO, but replaces teacher logits with rubric-derived rollout rewards, making the distillation pipeline black-box compatible (Fang et al., 8 May 2026).

Stepwise model routing uses rubric-guided process rewards inside GRPO. RoRo defines a total trajectory reward

$x$7

combining outcome correctness, LRM-token cost, and a Judge-produced process reward under a query-specific routing rubric (Ye et al., 28 May 2026). Long-horizon research agents extend the same idea further: RubricEM decomposes trajectories into Plan, Research, Review, and Answer stages, assigns stagewise rubric judgments, and optimizes with Stage-Structured GRPO rather than a single terminal reward (Li et al., 11 May 2026).

Several papers modify the reward aggregation itself rather than the optimizer. PAPO introduces decoupled advantage normalization: an outcome component $x$8 is normalized across all responses, while a rubric-based process component $x$9 is normalized only among correct responses, so that process supervision refines reasoning quality without overriding correctness (Tan et al., 27 Mar 2026). POW3R keeps human rubric weights as the evaluation target but dynamically rescales criterion-level training weights using rollout-level contrast, producing a policy-aware rubric reward that preserves category balance while emphasizing criteria that currently separate rollouts (Tyagi et al., 19 May 2026).

The same rubric-centric logic also appears outside on-policy RL. ARR-RPO uses rubric-conditioned VLM preferences as binary rewards in a PPO-style objective for text-to-image generation and image editing (Tian et al., 8 May 2026). Visual preference optimization with rubric rewards uses instance-specific rubrics to score multimodal outputs and then constructs on-policy preference pairs for DPO, which is not on-policy RL in the GRPO sense but still optimizes a policy against rubric-derived preferences rather than coarse outcome labels (Yu et al., 14 Apr 2026).

Setting	Optimization backend	Distinctive rubric role
Rubric-grounded RL	GRPO	Partial-credit scalar reward
ROPD	GRPO	Rubric replaces teacher logits
RoRo	GRPO	Process reward for routing trajectories
PAPO	GRPO variant	Correctness-anchored process shaping
POW3R	GRPO + adaptive weighting	Policy-aware criterion reweighting
ARR-RPO / rDPO	PPO-style RL or DPO	Rubric-conditioned preferences

5. Empirical behavior across domains

The strongest empirical claim shared by this literature is that rubric-derived rewards can improve both rubric-specific performance and broader downstream behavior when the rubrics are sufficiently discriminative and well verified.

Rubric-grounded RL reports a held-out normalized rubric reward of $g$0 for the GRPO-tuned policy, compared with $g$1 for the base Llama-3.1-8B-Instruct and $g$2 for an SFT baseline. The same model also improves on out-of-distribution reasoning benchmarks not derived from the training corpus: GSM8K from 84.53 to 85.44, MATH from 50.06 to 52.88, GPQA Main from 25.22 to 33.93, and GPQA Diamond from 24.24 to 32.32 (Bhattarai et al., 8 May 2026).

Rubric-based on-policy distillation reports that ROPD outperforms advanced logit-based OPD methods across most scenarios and achieves up to a 10x gain in sample efficiency, while remaining compatible with black-box teachers that expose only text outputs (Fang et al., 8 May 2026). Deep Research as Rubric reports strong competitive performance with only 1K–3K training instances across six benchmarks spanning agentic research and expert reasoning; GPT-5-generated rubrics favor breadth coverage on agentic tasks, Gemini-generated rubrics yield the most balanced performance, and bootstrap rubrics achieve the best overall performance at the third iteration (Mei et al., 31 May 2026).

In multimodal generation, ARR-RPO improves FLUX.1-dev from 0.66 to 0.80 on GenEval, from 83.84 to 85.76 on DPG-Bench, from 71.09 to 76.85 on TIIF, from 60.97 to 65.89 on UniGenBench++ Short, and from 69.42 to 72.93 on UniGenBench++ Long when using ARR with Gemini 3.1 Pro (Tian et al., 8 May 2026). Visual preference optimization with rubric rewards raises the macro average to 82.69 under rubric-based filtering, whereas outcome-based filtering drops it to 75.82 from an 81.14 base model; in its larger-scale setting, rDPO reaches 61.01, exceeding both the 52.36 style-constrained prompting baseline and the 59.48 base model (Yu et al., 14 Apr 2026).

Dynamic rubric methods report similar gains. EvoRubric improves a Qwen3-8B system from 62.19 to 68.84 average and a Qwen3-14B system from 65.22 to 70.55 average, outperforming both static rubric RL and an external evolving-rubric baseline on medical, writing, and science tasks (Guan et al., 28 May 2026). EvoRubrics reports consistent improvements over static and dynamic rubric baselines and further shows that a learned rubric generator can transfer as a reward model; even its fully self-supervised variant yields meaningful gains (Ding et al., 22 Jun 2026). In policy-aware weighting, POW3R wins 24 of 30 base-policy/metric comparisons and reaches the same plateau in $g$3–$g$4 fewer training steps than vanilla GRPO with rubric rewards (Tyagi et al., 19 May 2026). In process-aware reasoning RL, PAPO reaches 51.3% on OlympiadBench versus 46.3% for the ORM baseline and continues improving after ORM plateaus (Tan et al., 27 Mar 2026).

Taken together, these results indicate that rubric-based optimization is not tied to one domain. It has been instantiated for scientific reasoning, multimodal generation, image editing, policy distillation, model routing, open-ended research agents, medical QA, and visual-language preference learning.

6. Failure modes, diagnostics, and unresolved issues

RPO also makes failure modes unusually legible, because the optimization target is explicit. The clearest negative result is that rubric gains do not automatically imply broader quality gains.

A focused study of reward hacking in rubric-based RL separates two sources of divergence: verifier failure and rubric-design limitations. Under weak training verifiers, proxy-reward gains grow during training while reference-verifier quality plateaus, and exploitation concentrates in recurring patterns such as partial satisfaction of compound criteria, treating implicit content as explicit, and imprecise topical matching. Stronger verifiers substantially reduce, but do not eliminate, this exploitation. Even with stronger verification, reward hacking persists when rubrics omit important failure modes: rubric-based verifiers prefer the RL checkpoint, while rubric-free judges prefer the base model, with gains concentrated in completeness and presence-based criteria and declines in factual correctness, conciseness, relevance, and overall quality (Mahmoud et al., 12 May 2026).

Process rewards introduce a related instability. PAPO shows that directly optimizing against rubric-based process reward can cause reward hacking through verbosity: PRM-only training raises the process score while final-answer accuracy collapses, and even naive multiplicative reward $g$5 largely washes out the process signal. Its decoupled normalization scheme is an explicit attempt to retain rubric supervision without letting process metrics dominate correctness (Tan et al., 27 Mar 2026).

Self-evolving rubric systems add another layer of instability. DR-rubric reports a specialization-to-rebalancing trajectory across bootstrap iterations, but later iterations can exhibit reward polarization and collapse, motivating a stopping rule based on polarization statistics (Mei et al., 31 May 2026). EvoRubric and EvoRubrics respond by hardening the rubric-generation channel itself: meta-verifiers, zero-variance pruning, peer consensus, discriminativeness rewards, diversity rewards, alignment rewards, and constructiveness rewards are all introduced to prevent self-generated rubrics from becoming trivial, redundant, or adversarial (Guan et al., 28 May 2026, Ding et al., 22 Jun 2026).

Two diagnostics are especially notable. The reward-hacking study introduces a self-internalization gap, a verifier-free signal based on policy log-probabilities that tracks reference-verifier quality and detects when training with a weak verifier stops improving (Mahmoud et al., 12 May 2026). POW3R provides a complementary training-time diagnostic by tracking criterion-level variance and showing that roughly half of rubric criteria can be dead or saturated for a fresh base policy, implying that static scalarization wastes training pressure on criteria that provide no rollout-level contrast (Tyagi et al., 19 May 2026).

These results suggest that the central unresolved problems in RPO are not merely optimization stability. They include rubric construction, verifier robustness, balance between presence-based and absence-based criteria, resistance to verbosity and other reward-hacking strategies, and principled stopping or adaptation mechanisms when rubric-conditioned learning begins to diverge from broader notions of quality.