---
title: 'RewardRank: Relative Reward Shaping in LLM RL'
url: https://www.emergentmind.com/topics/rewardrank
type: topic
---

# RewardRank: Relative Reward Shaping in LLM RL

RewardRank, in the terminology used for “From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning,” denotes the combination of Reinforcement Learning with Relative Rewards (RLRR) and a listwise Ranking Reward Model for group-based reinforcement learning of large language models [2601.23058]. The central shift is from absolute numerical scoring to within-group relative ranking. In the formulation given there, raw evaluations from either rule-based verifiers or scalar reward models are converted into bounded relative rewards, which are then used in group-wise advantage estimation and a clipped GRPO objective. The stated motivation is twofold: under absolute scoring, verifiable tasks can suffer sparse supervision when sampled groups are unanimously correct or incorrect, while open-ended tasks can suffer unstable updates because scalar reward models produce unbounded scores with high variance over time [2601.23058].

## 1. Formal motivation and problem setting

The RewardRank formulation begins from group-based reinforcement learning for LLMs, where a prompt $p$ yields $k$ rollouts $o_1,\dots,o_k$ with raw scores $r_1,\dots,r_k$ [2601.23058]. The paper identifies two failure modes of absolute scoring.

In verifiable tasks such as math proofs, rule-based verifiers give binary feedback $s_i^{\mathrm{rule}} \in \{0,1\}$. As the policy improves, entire sampled groups can become unanimously correct or unanimously incorrect. The result is zero intra-group variance and vanishing policy gradients. The paper describes this as sparse supervision under absolute scoring and uses it as a primary motivation for replacing magnitude-based rewards with rank-based signals [2601.23058].

In open-ended tasks, the failure mode is different. Scalar reward models produce unbounded scores $s_i$ with high variance over time, and the standard group-based advantage estimate
$$
A_k = \frac{s_k-\mathrm{mean}(s)}{\mathrm{std}(s)}
$$
is described as being dominated by shifting scales, which leads to unstable updates [2601.23058]. RewardRank addresses this by transforming raw evaluations into relative signals defined only within each sampled group.

This framing places RewardRank inside a broader family of relative-reward methods. An earlier precedent is Ranked Reward (R2), which ranks a single agent’s raw returns against a buffer of recent returns to create a relative performance metric for single-agent combinatorial optimization [1807.01672]. A plausible implication is that RewardRank extends the general idea of ranking-based reward shaping from episode-level self-comparison to group-wise comparison among contemporaneous LLM rollouts.

## 2. Relative-reward formulation

RewardRank converts raw scores into relative rewards through a monotonic mapping $f$ based on the within-group rank $\mathrm{rnk}(o_i)$ [2601.23058]. Two variants are defined.

For open-ended tasks, Pure Relative Reward (PRR) is
$$
R_i^{\mathrm{rel}} = f_{\mathrm{PRR}}(\mathrm{rnk}_i) = \frac{k-\mathrm{rnk}_i}{k-1},
\qquad \mathrm{rnk}_i\in\{1,\dots,k\},
$$
with $1$ denoting best and $k$ worst [2601.23058].

For verifiable tasks, Hybrid Relative Reward (HRR) is
$$
R_i^{\mathrm{rel}} = s_i^{\mathrm{rule}} + \tau\cdot\tanh\!\left(\frac{k-\mathrm{rnk}_i}{k-1}\right),
\qquad 0<\tau\ll1.
$$
This construction preserves rule-based correctness while adding a small rank-derived shaping term within the correctness class [2601.23058].

Once relative rewards are computed, advantages are formed group-wise:
$$
A_i = R_i^{\mathrm{rel}} - \bar R^{\mathrm{rel}},
$$
with optional normalization by the group standard deviation $\sigma_G$,
$$
\hat A_i = \frac{R_i^{\mathrm{rel}} - \bar R^{\mathrm{rel}}}{\sigma_G}.
$$
The paper further applies correctness-aware clipping:
$$
A_i^{\mathrm{clip}} =
\begin{cases}
\max(A_i,\xi^-) & \text{if } o_i \text{ is correct} \\
\min(A_i,\xi^+) & \text{if } o_i \text{ is incorrect.}
\end{cases}
$$
Its stated purpose is to avoid penalizing correct but lower-ranked samples [2601.23058].

The theoretical stability argument follows directly from boundedness. PRR maps rewards to $[0,1]$, while HRR maps them to $[0,1+\tau]$. The paper states that variance is therefore bounded by Popoviciu’s inequality, $\le (b-a)^2/4$, and presents this as an explanation for more stable advantage estimation [2601.23058].

## 3. Ranking Reward Model

The Ranking Reward Model is the component that directly predicts relative ordering for group-based optimization [2601.23058]. Its backbone is a pretrained LLM encoder. A classification head takes as input the concatenation or interleaving of $k$ candidates $o_1,\dots,o_k$ plus prompt context, and emits $k$ logits $\ell_1,\dots,\ell_k$ [2601.23058].

The model uses a listwise Plackett–Luce formulation:
$$
P(\pi\mid o_1\dots o_k) = \prod_{i=1}^k
\frac{\exp(\ell_{\pi_i})}{\sum_{j=i}^k \exp(\ell_{\pi_j})}.
$$
Given a ground-truth permutation $\pi^\*$, the ranking loss is the cross-entropy over the permutation distribution:
$$
L_{\mathrm{rank}} = - \log P(\pi^\*\mid o_1\dots o_k)
= - \sum_{i=1}^k \left[
\ell_{\pi^\*_i} - \log \sum_{j=i}^k \exp(\ell_{\pi^\*_j})
\right].
$$
The paper states that each term encourages the model to place the true $i$-th best response above all remaining candidates [2601.23058].

The training data pipeline is hierarchical. It first collects $(\text{prompt}, k \text{ candidates})$ pairs. It then enforces a strict rule-based hierarchy, correct $>$ incorrect. Within the same correctness group, it uses a strong SRM or a stronger LLM judge to produce a silver-standard ranking. Cases where the SRM ranking contradicts ground-truth correctness are discarded [2601.23058]. This design makes correctness the primary ordering criterion and uses learned or judged quality only as a secondary criterion within correctness-equivalent subsets.

A related but distinct use of rank-plus-reward modeling appears in slate recommendation. “Probabilistic Rank and Reward” models joint feedback over whether the user interacted with the slate and which rank was selected, using a categorical likelihood over non-click and clicked positions [2208.06263]. That work is not an RLHF method, but it shows that combining rank and reward signals can be formalized probabilistically rather than through scalar proxy losses alone.

## 4. Training and deployment algorithm

The paper describes RewardRank as a training and deployment procedure built around GRPO-style policy optimization [2601.23058]. The Ranking Reward Model is first pretrained on offline preference data. During RL fine-tuning of the policy $\pi_\theta$, each step samples a batch of prompts, samples $k$ rollouts per prompt from $\pi_{\theta_{\mathrm{old}}}$, and computes rule-based scores $s_i^{\mathrm{rule}}$ and/or SRM scores [2601.23058].

For ranking inference, the $k$ candidates are partitioned into subgroups of size $n$, with $n=4$. The Ranking Reward Model is applied to each subgroup to obtain local raw ranks $\mathrm{rnk}_i^{\mathrm{raw}}$. A hierarchical re-ranking stage then enforces $(\text{correctness}, \text{length})$ lexicographic order to produce global $\mathrm{rnk}_i$ [2601.23058]. The paper therefore does not use model ranking in isolation; it constrains the final order using explicit correctness and conciseness criteria.

Reward shaping then applies either HRR or PRR to the ranks and rule-based scores. Advantages are computed as
$$
A_i = R_i^{\mathrm{rel}} - \mathrm{mean}(R^{\mathrm{rel}})
$$
and correctness-aware clipping is applied. Policy optimization maximizes the clipped GRPO objective
$$
J(\theta)= \frac{1}{k} \sum_{i,t}
\min\!\bigl(\rho_{i,t}A_i,\,
\mathrm{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)A_i\bigr).
$$
The procedure can optionally collect new preference data from the improved policy for further Ranking Reward Model refinement [2601.23058].

The hierarchical re-ranking stage is empirically important. On the AIME/Olympiad suite, removing the correctness constraint drops average performance from $47.1$ to $45.7$. Removing length re-ranking slightly increases accuracy but at the cost of longer outputs [2601.23058]. This indicates that the ranking system is intended not merely to sort by perceived quality but to preserve hard constraints and output-shape preferences during optimization.

## 5. Empirical results and ablations

The principal reported experiments cover verifiable reasoning, open-ended writing, and reward-model evaluation [2601.23058]. The following table summarizes the concrete numerical results explicitly given.

| Setting | Baseline | RewardRank result |
|---|---:|---:|
| Avg. accuracy, 1.5B, math+logic | GRPO: 53.4% | RLRR (HRR): 55.4%; RLRR (PRR): 55.5% |
| Avg. accuracy, 8B backbone | 62.9% | RLRR(H): 65.9% |
| WritingBench | baseline reward models | RLRR(PRR): +0.7–1.8 points across domains |
| WritingBench with fine-tuned Ranking RM | pointwise SRMs | +1.5–2.0 overall improvements |
| Best-of-$k$ reward-model evaluation | SRM | Ranking RM selects better responses as $k$ grows |

For verifiable reasoning, the experiments use Qwen-1.5B and LLaMA-8B on 8 math and 2 logic benchmarks. Average accuracy for the 1.5B setting increases from $53.4\%$ under GRPO to $55.4\%$ with RLRR (HRR) and $55.5\%$ with RLRR (PRR). On the 8B backbone, RLRR(H) increases the average from $62.9\%$ to $65.9\%$ [2601.23058].

For open-ended writing on WritingBench, the paper reports that baseline reward models combined with RLRR(PRR) yield $+0.7$ to $1.8$ points across domains such as Academic, Finance, and Politics. When a fine-tuned Ranking Reward Model is used, the gains become $+1.5$ to $2.0$ overall relative to pointwise SRMs [2601.23058].

For reward-model evaluation through Best-of-$k$ test-time scaling, the Ranking Reward Model selects better responses than the SRM as $k$ grows, with a wider gap at $k=8,16$ [2601.23058]. The paper presents this as evidence that listwise relative ranking is itself a stronger selection primitive than scalar pointwise scoring for grouped candidate sets.

The ablation results isolate several mechanisms. GRPO’s “effective” groups, defined as those with non-zero variance, fall below $40\%$ late in training, whereas RLRR uses $100\%$ of rollouts. GRPO improves with larger group size $k$, but RLRR outperforms it at all $k\in\{4,8,16\}$. Random ranking plus correctness re-ranking improves over GRPO from $36.3\%$ to $37.5\%$, and the full Ranking Reward Model adds another $+0.5\%$ to $38.0\%$ [2601.23058]. This indicates that merely restoring relative differentiation already helps, while more accurate intra-group ordering provides additional gains.

A separate but related LLM-RL line is R$^3$, whose Structural Entropy Ranking Reward assigns relative rewards to truncated or failed samples using entropy-based dominance ranking. In that setting, removing SERR drops AIME24 from $47.50$ to $36.88$ and MATH from $89.27$ to $86.84$ [2601.19620]. This does not establish equivalence to RewardRank, but it supports the broader proposition that ranking-derived rewards can preserve informative gradients when ordinary group advantages collapse.

## 6. Conceptual position and related uses of the term

RewardRank, as defined here, is specific to the shift from absolute to relative rewards in group-based reinforcement learning for LLMs [2601.23058]. It should be distinguished from several adjacent but non-identical uses of ranking-based reward ideas.

The most direct historical precursor is Ranked Reward (R2), which uses a buffer of recent returns and a percentile threshold $r_\alpha$ to map raw episodic returns into binary targets $z\in\{+1,-1\}$ for single-agent combinatorial optimization [1807.01672]. R2 compares the current episode to recent episodes of the same agent, whereas RewardRank compares multiple rollouts within a prompt-conditioned group and preserves finer order information through listwise ranking and bounded relative rewards. This suggests a lineage from self-comparison across time to comparative evaluation within a sampled cohort.

In robotics, Rank2Reward learns a temporally ranked progress function from passive video demonstrations and converts it into a shaped reward for reinforcement learning [2404.14735]. Its ranking target is temporal order within expert trajectories, rather than quality order among contemporaneous sampled outputs. The commonality is the use of ranking to generate dense or stable reward signals when direct reward supervision is weak or unavailable.

In recommendation and learning-to-rank, the name “RewardRank” appears again with a different meaning. “RewardRank: Optimizing True Learning-to-Rank Utility” learns a deep utility model over entire permutations and then optimizes a ranking policy via differentiable soft permutation operators [2508.14180]. There, the reward is list-level user engagement such as click or purchase probability rather than RL advantage estimation over LLM rollouts. Likewise, “Combining Reward and Rank Signals for Slate Recommendation” and “Probabilistic Rank and Reward” use reward and rank jointly to estimate slate utility in recommendation settings [2107.12455]; [2208.06263].

A common misconception would be to treat all of these methods as instances of the same algorithm. The shared motif is the use of rank information to reshape or supplement reward, but the objects being ranked differ: recent episodes in R2, generated responses in RewardRank for LLM RL, token-entropy patterns in R$^3$, video frames in Rank2Reward, and item permutations or clicked positions in recommendation work [1807.01672]; [2601.23058]; [2601.19620]; [2404.14735]; [2508.14180]. Their training objectives, data assumptions, and deployment settings are therefore distinct.

## 7. Significance, limitations, and plausible implications

The significance claimed for RewardRank is that it replaces magnitude-based reward shaping with pure intra-group preference signals while remaining compatible with group-based policy optimization [2601.23058]. Empirically, the method is reported to deliver more stable gradients, higher data utilization, and consistent gains over absolute-scoring baselines. Theoretically, bounded relative rewards provide a direct variance-control argument, and practically, the Ranking Reward Model supplies a mechanism for obtaining comparative supervision even when scalar rewards are sparse or unstable [2601.23058].

At the same time, the formulation embeds several design choices that function as implicit constraints. The training data pipeline enforces a strict rule-based hierarchy, discards cases where SRM ranking contradicts ground-truth correctness, and uses hierarchical re-ranking with correctness and length lexicographic order at deployment time [2601.23058]. This suggests that RewardRank is not simply a learned preference system; it is a hybrid of learned ordering and manually imposed structure. A plausible implication is that its effectiveness depends not only on ranking quality but also on how well those structural constraints encode the intended task semantics.

Another limitation, stated indirectly through the ablations, is that ranking quality within correctness-equivalent sets still matters. Random ranking with correctness re-ranking already improves over GRPO, but the full Ranking Reward Model adds a further improvement [2601.23058]. This indicates that relative rewards alone are not sufficient; the comparative model must recover meaningful fine-grained order. Similarly, the need for subgroup partitioning with $n=4$ and hierarchical re-ranking suggests an implementation trade-off between listwise fidelity and tractable ranking over larger candidate sets [2601.23058].

More broadly, the method contributes to an emerging view that advantage estimation in LLM reinforcement learning may be more robust when derived from relative structure rather than from raw score magnitudes. RewardRank operationalizes that view through bounded HRR/PRR mappings, a listwise Ranking Reward Model, and correctness-aware clipping inside GRPO [2601.23058]. The surrounding literature indicates that this rank-based perspective has analogues in self-play for single-agent optimization, failure handling in reasoning models, passive-video reward construction, and listwise recommendation, but RewardRank’s specific contribution is to make relative ranking the central reward-shaping primitive for group-based LLM RL [1807.01672]; [2601.19620]; [2404.14735]; [2508.14180].

Source: https://www.emergentmind.com/topics/rewardrank