Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

Abstract: Scalar reward models compress multi-dimensional human preferences into a single opaque score, creating an information bottleneck that often leads to brittleness and reward hacking in open-ended alignment. We argue that robust alignment for non-verifiable tasks is fundamentally a principle generalization problem: reward should not be a learned function internalized into a judge, but an explicit reasoning process executed under inspectable principles. To operationalize this view, we present the Open Rubric System (OpenRS), a plug-and-play, rubrics-based LLM-as-a-Judge framework built around Pairwise Adaptive Meta-Rubrics (PAMR) and lightweight Pointwise Verifiable Rubrics (PVRs), which provide both hard-constraint guardrails and verifiable reward components when ground-truth or programmatic checks are available. OpenRS uses an explicit meta-rubric -- a constitution-like specification that governs how rubrics are instantiated, weighted, and enforced -- and instantiates adaptive rubrics on the fly by conditioning on the semantic differences between two candidate responses. It then performs criterion-wise pairwise comparisons and aggregates criterion-level preferences externally, avoiding pointwise weighted scalarization while improving discriminability in open-ended settings. To keep principles consistent yet editable across various domains, we introduce a two-level meta-rubric refinement pipeline (automated evolutionary refinement for general principles and a reproducible human-in-the-loop procedure for domain principles), complemented with pointwise verifiable rubrics that act as both guardrails against degenerate behaviors and a source of verifiable reward for objective sub-tasks. Finally, we instantiate OpenRS as reward supervision in pairwise RL training.

• The paper introduces OpenRS, a reward system that executes adaptive pairwise rubrics and combines them with pointwise verifiable checks instead of collapsing preferences into a single scalar score.
• OpenRS achieves an 89.4 average across four reward-modeling benchmarks, outperforming the cited baseline by 5.1 points and remaining more stable on difficult RM-Bench pairs.
• Replacing a scalar reward model with OpenRS improves end-to-end RL performance from 68.4 to 71.3 across five benchmarks, while pairwise difference grounding, meta-rubrics, and local asynchronous serving support practical scaling.

Motivation: the scalar reward bottleneck

The paper's central argument is that robust reward supervision for non-verifiable tasks is a principle generalization problem rather than a function-approximation problem. Scalar reward models (SRMs) compress multi-dimensional human preferences into a single score, creating an information bottleneck that invites reward hacking and brittle out-of-distribution behavior. Generative reward models (GenRMs) trained on preference data improve transparency through rationales, but the evaluation principles remain implicitly internalized in model weights, so they inherit SRM-like brittleness and are expensive to iterate. Existing rubric-based reward pipelines, while promising for extending RLVR-style optimization to open-ended domains, mostly rely on static rubrics and produce their final signal via pointwise scoring followed by weighted scalar aggregation — a step the authors identify as an intrinsic ceiling on discriminability and a persistent reward-gaming surface. OpenRS rejects this final scalarization: rubric-level preferences are aggregated externally, and reward is executed as an explicit reasoning process under a constitution-like meta-rubric rather than learned into a judge.

System design: PAMR and pointwise verifiable rubrics

OpenRS decomposes the reward $R(q, o_i)$ into two pathways. The Pairwise Adaptive Meta-Rubric (PAMR) pathway first computes the semantic differences $\Delta_{ij}$ between a candidate pair, then instantiates a pair-specific adaptive rubric $\mathcal{R}_{ij} = \{(c_k, w_k)\}_{k=1}^K$ by conditioning a hierarchical meta-rubric (a General Meta Rubric $\mathcal{M}_{\text{gen}}$ plus Domain Meta Rubrics $\mathcal{M}_{\text{dom}}$) on those differences. Each criterion receives a comparative score $v_k \in \{-2,\dots,2\}$, and the pairwise score $s_{ij}$ is a weight-normalized criterion-level average. The Pointwise Verifiable Rubric (PVR) pathway supplies deterministic signals $\phi_c(o) \in \{+1, -1\}$ for checkable constraints (format, length, ground-truth correctness), acting both as hard guardrails against degenerate behaviors and as verifiable reward components. The final RL reward is the superposition $$R(q, o_i) = s_{i,\text{ref}} + \gamma \sum_{c \in \mathcal{V}_q} \phi_c(o_i)$$, with each rollout compared against a bootstrapped reference anchor following BRPO (Jia et al., 30 May 2025), reducing judge calls from $O(N^2)$ to $\Delta_{ij}$0.

Two-level meta-rubric refinement

The General Meta Rubric is refined via a beam-style evolutionary search in which a refinement policy $\Delta_{ij}$1 rolls out edit sequences over $\Delta_{ij}$2 actions, with candidates scored by an oracle measuring alignment accuracy on a balanced preference set. The refinement policy is trained with a proposed Asymmetric GRPO (Asym-GRPO) variant: because rewards are sparse and skewed, standard group-normalized GRPO can assign positive advantage to "less-bad" failures; Asym-GRPO backpropagates only through the top-$\Delta_{ij}$3 rollouts and masks the rest. Empirically, Frozen refinement plateaus, standard GRPO is volatile, and Asym-GRPO yields the most stable sustained improvement. Domain Meta Rubrics are adapted separately through a reproducible human-in-the-loop error-analysis workflow, with edits constrained to abstract, reusable principles and validated on held-out splits — a deliberate contrast to gradient-based adaptation that avoids catastrophic forgetting of judge competence.

Reward-modeling results

OpenRS achieves the best average across four benchmarks (RM-Bench, JudgeBench, RewardBench v2, PPE Preference Chinese subset), with OpenRS (Qwen3-235B-A22B-Instruct-2507) reaching an average of 89.4 versus 84.3 for Skywork-Reward-V2-Llama-3.1-8B, a +5.1 improvement. Per-benchmark gains include +11.3 on JudgeBench (93.3 with gpt-oss-120b vs. 82.0 for the strongest reported baseline), +6.6 on RewardBench v2, +2.7 on PPE Preference, and +0.2 on RM-Bench — the latter a notably narrow margin over Skywork-Reward-V2, so the RM-Bench advantage should not be overstated. Fine-grained breakdowns show SRMs degrading sharply on RM-Bench Hard pairs (sometimes below 50% from >90% on Easy) while OpenRS remains stable (96.6/95.0/87.4 across Easy/Normal/Hard). Ablations attribute the gains to three ingredients: pairwise over pointwise evaluation (86.1 vs. 89.4 average), the diff-first grounding mechanism (removing it costs ~1.9 average points), and domain meta-rubrics. A Pareto analysis over judge backbones motivates Qwen3-235B-A22B as the default, and the system scales monotonically with backbone strength (84.9 with Qwen3-30B-A3B to 89.4 with Qwen3-235B-A22B), indicating evaluation quality is bounded by the judge's underlying reasoning ability — a dependency the paper acknowledges rather than resolves.

End-to-end RL and the claimed "Aha Moment"

Replacing the SRM with OpenRS as the reward interface — holding policy warm-start and training data fixed — improves the average over five public benchmarks from 68.4 to 71.3, with the largest gains on IFScale (54.8 → 59.0) and JudgeMark-v2 (43.8 → 54.4); pointwise OpenRS yields an intermediate 69.4, supporting the claim that pairwise, difference-grounded execution matters beyond rubric formatting alone. Throughput is addressed by serving the judge locally on 128 H20 GPUs with asynchronous batching, sustaining ~10,000 concurrent requests with small wall-clock overhead relative to an SRM setup.

The paper's most speculative claim concerns emergence: the authors argue that SRMs induce a "double approximation loss" (first in reward modeling, then in policy fitting) that caps policy performance, and that rubric-based rewards act as a "soft" rule-based verifier preserving the multimodality of human preferences. They report a distinctive entropy trajectory — an initial decrease followed by a rise and stabilization at a higher level, unlike monotonic entropy collapse in standard RLHF — and qualitative case studies showing stronger subjective stance and emotional expression. This evidence is observational and partly qualitative; the paper itself concedes that larger-scale studies across more policy sizes and tasks are needed to validate whether rubric-based rewards reliably unlock "Aha Moment"-style emergence beyond verifiable domains.

Limitations and open questions

Several limitations are stated or apparent. The in-house LMArena-style generalization evaluation (1,000 real user queries each for QA and Writing) was incomplete at writing time, with conclusions explicitly deferred. The "Same"-rate mechanism, which filters bidirectionally inconsistent judgments rather than forcing a decision, reaches ~0.45 during RL training — acceptable under DAPO-style filtering, but it means a substantial fraction of rollout signal is discarded, and the interaction between rising same-rates and effective sample efficiency over long training runs is not characterized. Judge quality is bounded by the backbone model, and the refinement pipeline depends on oracle alignment scores on preference sets, so oracle noise propagates into the meta-rubric. Adversarial robustness of rubric execution (e.g., prompt injection against the judge) and automated monitoring of rubric drift during training are identified as open problems rather than solved. Finally, the end-to-end RL comparison uses a single in-house policy and a single training-data configuration, leaving the generality of the 68.4 → 71.3 gain across policy scales untested.

Conclusion

OpenRS operationalizes the thesis that reward for non-verifiable tasks should be an inspectable reasoning process rather than a learned scalar function, combining difference-grounded adaptive pairwise rubrics, externally aggregated criterion preferences, and verifiable pointwise guardrails under a hierarchically refined meta-rubric. The empirical record — state-of-the-art reward-modeling results with a +5.1 average margin, consistent RL gains from a drop-in reward replacement, and throughput-feasible deployment — substantiates the discriminability argument against pointwise scalarization. The open questions that remain are chiefly about cost reduction (distilling rubric execution to smaller judges), adversarial robustness, and whether the observed entropy and behavioral shifts constitute durable emergence at scale.