RewardRank: Relative Reward Shaping in LLM RL

• RewardRank is a method that converts absolute reward scores into bounded, within-group relative rewards for more stable LLM training.
• It employs a listwise ranking reward model using a pretrained LLM encoder to compare rollouts while enforcing rule-based correctness.
• Empirical results show improved verifiable reasoning accuracy and better handling of sparse supervision and high variance in open-ended tasks.

RewardRank, in the terminology used for “From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning,” denotes the combination of Reinforcement Learning with Relative Rewards (RLRR) and a listwise Ranking Reward Model for group-based reinforcement learning of LLMs (Niu et al., 30 Jan 2026). The central shift is from absolute numerical scoring to within-group relative ranking. In the formulation given there, raw evaluations from either rule-based verifiers or scalar reward models are converted into bounded relative rewards, which are then used in group-wise advantage estimation and a clipped GRPO objective. The stated motivation is twofold: under absolute scoring, verifiable tasks can suffer sparse supervision when sampled groups are unanimously correct or incorrect, while open-ended tasks can suffer unstable updates because scalar reward models produce unbounded scores with high variance over time (Niu et al., 30 Jan 2026).

1. Formal motivation and problem setting

The RewardRank formulation begins from group-based reinforcement learning for LLMs, where a prompt $p$ yields $k$ rollouts $o_1,\dots,o_k$ with raw scores $r_1,\dots,r_k$ (Niu et al., 30 Jan 2026). The paper identifies two failure modes of absolute scoring.

In verifiable tasks such as math proofs, rule-based verifiers give binary feedback $s_i^{\mathrm{rule}} \in \{0,1\}$. As the policy improves, entire sampled groups can become unanimously correct or unanimously incorrect. The result is zero intra-group variance and vanishing policy gradients. The paper describes this as sparse supervision under absolute scoring and uses it as a primary motivation for replacing magnitude-based rewards with rank-based signals (Niu et al., 30 Jan 2026).

In open-ended tasks, the failure mode is different. Scalar reward models produce unbounded scores $s_i$ with high variance over time, and the standard group-based advantage estimate

$A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$

is described as being dominated by shifting scales, which leads to unstable updates (Niu et al., 30 Jan 2026). RewardRank addresses this by transforming raw evaluations into relative signals defined only within each sampled group.

This framing places RewardRank inside a broader family of relative-reward methods. An earlier precedent is Ranked Reward (R2), which ranks a single agent’s raw returns against a buffer of recent returns to create a relative performance metric for single-agent combinatorial optimization (Laterre et al., 2018). A plausible implication is that RewardRank extends the general idea of ranking-based reward shaping from episode-level self-comparison to group-wise comparison among contemporaneous LLM rollouts.

2. Relative-reward formulation

RewardRank converts raw scores into relative rewards through a monotonic mapping $f$ based on the within-group rank $\mathrm{rnk}(o_i)$ (Niu et al., 30 Jan 2026). Two variants are defined.

For open-ended tasks, Pure Relative Reward (PRR) is

$$R_i^{\mathrm{rel}} = f_{\mathrm{PRR}}(\mathrm{rnk}_i) = \frac{k-\mathrm{rnk}_i}{k-1}, \qquad \mathrm{rnk}_i\in\{1,\dots,k\},$$

with $k$0 denoting best and $k$1 worst (Niu et al., 30 Jan 2026).

For verifiable tasks, Hybrid Relative Reward (HRR) is

$k$2

This construction preserves rule-based correctness while adding a small rank-derived shaping term within the correctness class (Niu et al., 30 Jan 2026).

Once relative rewards are computed, advantages are formed group-wise:

$k$3

with optional normalization by the group standard deviation $k$4,

$k$5

The paper further applies correctness-aware clipping:

$k$6

Its stated purpose is to avoid penalizing correct but lower-ranked samples (Niu et al., 30 Jan 2026).

The theoretical stability argument follows directly from boundedness. PRR maps rewards to $k$7, while HRR maps them to $k$8. The paper states that variance is therefore bounded by Popoviciu’s inequality, $k$9, and presents this as an explanation for more stable advantage estimation (Niu et al., 30 Jan 2026).

3. Ranking Reward Model

The Ranking Reward Model is the component that directly predicts relative ordering for group-based optimization (Niu et al., 30 Jan 2026). Its backbone is a pretrained LLM encoder. A classification head takes as input the concatenation or interleaving of $o_1,\dots,o_k$0 candidates $o_1,\dots,o_k$1 plus prompt context, and emits $o_1,\dots,o_k$2 logits $o_1,\dots,o_k$3 (Niu et al., 30 Jan 2026).

The model uses a listwise Plackett–Luce formulation:

$o_1,\dots,o_k$4

Given a ground-truth permutation $o_1,\dots,o_k$5, the ranking loss is the cross-entropy over the permutation distribution:

$o_1,\dots,o_k$6

The paper states that each term encourages the model to place the true $o_1,\dots,o_k$7-th best response above all remaining candidates (Niu et al., 30 Jan 2026).

The training data pipeline is hierarchical. It first collects $o_1,\dots,o_k$8 pairs. It then enforces a strict rule-based hierarchy, correct $o_1,\dots,o_k$9 incorrect. Within the same correctness group, it uses a strong SRM or a stronger LLM judge to produce a silver-standard ranking. Cases where the SRM ranking contradicts ground-truth correctness are discarded (Niu et al., 30 Jan 2026). This design makes correctness the primary ordering criterion and uses learned or judged quality only as a secondary criterion within correctness-equivalent subsets.

A related but distinct use of rank-plus-reward modeling appears in slate recommendation. “Probabilistic Rank and Reward” models joint feedback over whether the user interacted with the slate and which rank was selected, using a categorical likelihood over non-click and clicked positions (Aouali et al., 2022). That work is not an RLHF method, but it shows that combining rank and reward signals can be formalized probabilistically rather than through scalar proxy losses alone.

4. Training and deployment algorithm

The paper describes RewardRank as a training and deployment procedure built around GRPO-style policy optimization (Niu et al., 30 Jan 2026). The Ranking Reward Model is first pretrained on offline preference data. During RL fine-tuning of the policy $r_1,\dots,r_k$0, each step samples a batch of prompts, samples $r_1,\dots,r_k$1 rollouts per prompt from $r_1,\dots,r_k$2, and computes rule-based scores $r_1,\dots,r_k$3 and/or SRM scores (Niu et al., 30 Jan 2026).

For ranking inference, the $r_1,\dots,r_k$4 candidates are partitioned into subgroups of size $r_1,\dots,r_k$5, with $r_1,\dots,r_k$6. The Ranking Reward Model is applied to each subgroup to obtain local raw ranks $r_1,\dots,r_k$7. A hierarchical re-ranking stage then enforces $r_1,\dots,r_k$8 lexicographic order to produce global $r_1,\dots,r_k$9 (Niu et al., 30 Jan 2026). The paper therefore does not use model ranking in isolation; it constrains the final order using explicit correctness and conciseness criteria.

Reward shaping then applies either HRR or PRR to the ranks and rule-based scores. Advantages are computed as

$s_i^{\mathrm{rule}} \in \{0,1\}$0

and correctness-aware clipping is applied. Policy optimization maximizes the clipped GRPO objective

$s_i^{\mathrm{rule}} \in \{0,1\}$1

The procedure can optionally collect new preference data from the improved policy for further Ranking Reward Model refinement (Niu et al., 30 Jan 2026).

The hierarchical re-ranking stage is empirically important. On the AIME/Olympiad suite, removing the correctness constraint drops average performance from $s_i^{\mathrm{rule}} \in \{0,1\}$2 to $s_i^{\mathrm{rule}} \in \{0,1\}$3. Removing length re-ranking slightly increases accuracy but at the cost of longer outputs (Niu et al., 30 Jan 2026). This indicates that the ranking system is intended not merely to sort by perceived quality but to preserve hard constraints and output-shape preferences during optimization.

5. Empirical results and ablations

The principal reported experiments cover verifiable reasoning, open-ended writing, and reward-model evaluation (Niu et al., 30 Jan 2026). The following table summarizes the concrete numerical results explicitly given.

Setting	Baseline	RewardRank result
Avg. accuracy, 1.5B, math+logic	GRPO: 53.4%	RLRR (HRR): 55.4%; RLRR (PRR): 55.5%
Avg. accuracy, 8B backbone	62.9%	RLRR(H): 65.9%
WritingBench	baseline reward models	RLRR(PRR): +0.7–1.8 points across domains
WritingBench with fine-tuned Ranking RM	pointwise SRMs	+1.5–2.0 overall improvements
Best-of-$s_i^{\mathrm{rule}} \in \{0,1\}$4 reward-model evaluation	SRM	Ranking RM selects better responses as $s_i^{\mathrm{rule}} \in \{0,1\}$5 grows

For verifiable reasoning, the experiments use Qwen-1.5B and LLaMA-8B on 8 math and 2 logic benchmarks. Average accuracy for the 1.5B setting increases from $s_i^{\mathrm{rule}} \in \{0,1\}$6 under GRPO to $s_i^{\mathrm{rule}} \in \{0,1\}$7 with RLRR (HRR) and $s_i^{\mathrm{rule}} \in \{0,1\}$8 with RLRR (PRR). On the 8B backbone, RLRR(H) increases the average from $s_i^{\mathrm{rule}} \in \{0,1\}$9 to $s_i$0 (Niu et al., 30 Jan 2026).

For open-ended writing on WritingBench, the paper reports that baseline reward models combined with RLRR(PRR) yield $s_i$1 to $s_i$2 points across domains such as Academic, Finance, and Politics. When a fine-tuned Ranking Reward Model is used, the gains become $s_i$3 to $s_i$4 overall relative to pointwise SRMs (Niu et al., 30 Jan 2026).

For reward-model evaluation through Best-of-$s_i$5 test-time scaling, the Ranking Reward Model selects better responses than the SRM as $s_i$6 grows, with a wider gap at $s_i$7 (Niu et al., 30 Jan 2026). The paper presents this as evidence that listwise relative ranking is itself a stronger selection primitive than scalar pointwise scoring for grouped candidate sets.

The ablation results isolate several mechanisms. GRPO’s “effective” groups, defined as those with non-zero variance, fall below $s_i$8 late in training, whereas RLRR uses $s_i$9 of rollouts. GRPO improves with larger group size $A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$0, but RLRR outperforms it at all $A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$1. Random ranking plus correctness re-ranking improves over GRPO from $A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$2 to $A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$3, and the full Ranking Reward Model adds another $A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$4 to $A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$5 (Niu et al., 30 Jan 2026). This indicates that merely restoring relative differentiation already helps, while more accurate intra-group ordering provides additional gains.

A separate but related LLM-RL line is R$A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$6, whose Structural Entropy Ranking Reward assigns relative rewards to truncated or failed samples using entropy-based dominance ranking. In that setting, removing SERR drops AIME24 from $A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$7 to $A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$8 and MATH from $A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}$9 to $f$0 (Jiang et al., 27 Jan 2026). This does not establish equivalence to RewardRank, but it supports the broader proposition that ranking-derived rewards can preserve informative gradients when ordinary group advantages collapse.

6. Conceptual position and related uses of the term

RewardRank, as defined here, is specific to the shift from absolute to relative rewards in group-based reinforcement learning for LLMs (Niu et al., 30 Jan 2026). It should be distinguished from several adjacent but non-identical uses of ranking-based reward ideas.

The most direct historical precursor is Ranked Reward (R2), which uses a buffer of recent returns and a percentile threshold $f$1 to map raw episodic returns into binary targets $f$2 for single-agent combinatorial optimization (Laterre et al., 2018). R2 compares the current episode to recent episodes of the same agent, whereas RewardRank compares multiple rollouts within a prompt-conditioned group and preserves finer order information through listwise ranking and bounded relative rewards. This suggests a lineage from self-comparison across time to comparative evaluation within a sampled cohort.

In robotics, Rank2Reward learns a temporally ranked progress function from passive video demonstrations and converts it into a shaped reward for reinforcement learning (Yang et al., 2024). Its ranking target is temporal order within expert trajectories, rather than quality order among contemporaneous sampled outputs. The commonality is the use of ranking to generate dense or stable reward signals when direct reward supervision is weak or unavailable.

In recommendation and learning-to-rank, the name “RewardRank” appears again with a different meaning. “RewardRank: Optimizing True Learning-to-Rank Utility” learns a deep utility model over entire permutations and then optimizes a ranking policy via differentiable soft permutation operators (Bhatt et al., 19 Aug 2025). There, the reward is list-level user engagement such as click or purchase probability rather than RL advantage estimation over LLM rollouts. Likewise, “Combining Reward and Rank Signals for Slate Recommendation” and “Probabilistic Rank and Reward” use reward and rank jointly to estimate slate utility in recommendation settings (Aouali et al., 2021); (Aouali et al., 2022).

A common misconception would be to treat all of these methods as instances of the same algorithm. The shared motif is the use of rank information to reshape or supplement reward, but the objects being ranked differ: recent episodes in R2, generated responses in RewardRank for LLM RL, token-entropy patterns in R$f$3, video frames in Rank2Reward, and item permutations or clicked positions in recommendation work (Laterre et al., 2018); (Niu et al., 30 Jan 2026); (Jiang et al., 27 Jan 2026); (Yang et al., 2024); (Bhatt et al., 19 Aug 2025). Their training objectives, data assumptions, and deployment settings are therefore distinct.

7. Significance, limitations, and plausible implications

The significance claimed for RewardRank is that it replaces magnitude-based reward shaping with pure intra-group preference signals while remaining compatible with group-based policy optimization (Niu et al., 30 Jan 2026). Empirically, the method is reported to deliver more stable gradients, higher data utilization, and consistent gains over absolute-scoring baselines. Theoretically, bounded relative rewards provide a direct variance-control argument, and practically, the Ranking Reward Model supplies a mechanism for obtaining comparative supervision even when scalar rewards are sparse or unstable (Niu et al., 30 Jan 2026).

At the same time, the formulation embeds several design choices that function as implicit constraints. The training data pipeline enforces a strict rule-based hierarchy, discards cases where SRM ranking contradicts ground-truth correctness, and uses hierarchical re-ranking with correctness and length lexicographic order at deployment time (Niu et al., 30 Jan 2026). This suggests that RewardRank is not simply a learned preference system; it is a hybrid of learned ordering and manually imposed structure. A plausible implication is that its effectiveness depends not only on ranking quality but also on how well those structural constraints encode the intended task semantics.

Another limitation, stated indirectly through the ablations, is that ranking quality within correctness-equivalent sets still matters. Random ranking with correctness re-ranking already improves over GRPO, but the full Ranking Reward Model adds a further improvement (Niu et al., 30 Jan 2026). This indicates that relative rewards alone are not sufficient; the comparative model must recover meaningful fine-grained order. Similarly, the need for subgroup partitioning with $f$4 and hierarchical re-ranking suggests an implementation trade-off between listwise fidelity and tractable ranking over larger candidate sets (Niu et al., 30 Jan 2026).

More broadly, the method contributes to an emerging view that advantage estimation in LLM reinforcement learning may be more robust when derived from relative structure rather than from raw score magnitudes. RewardRank operationalizes that view through bounded HRR/PRR mappings, a listwise Ranking Reward Model, and correctness-aware clipping inside GRPO (Niu et al., 30 Jan 2026). The surrounding literature indicates that this rank-based perspective has analogues in self-play for single-agent optimization, failure handling in reasoning models, passive-video reward construction, and listwise recommendation, but RewardRank’s specific contribution is to make relative ranking the central reward-shaping primitive for group-based LLM RL (Laterre et al., 2018); (Jiang et al., 27 Jan 2026); (Yang et al., 2024); (Bhatt et al., 19 Aug 2025).