Parallel Slice Ranking Reward (PSRR)

• The paper introduces PSRR as a mechanism that converts coarse batch-level rewards into fine-grained, per-completion feedback for precise credit assignment.
• PSRR reorganizes output into parallel slices, enabling independent ranking and local reward computation that refines both pairwise and listwise reward components.
• Empirical results demonstrate that PSRR’s sliced rewards significantly boost performance, achieving higher Spearman correlations compared to naive listwise reinforcement learning.

Parallel Slice Ranking Reward (PSRR) is a ranking-reward mechanism introduced in PoLi-RL for Conditional Semantic Textual Similarity (C-STS). Its defining operation is to compute ranking rewards in parallel slices, where each slice consists of same-indexed completions from different samples. In that formulation, PSRR converts a coarse batch-level listwise signal into per-completion ranking feedback, with the stated goal of providing “a precise, differentiated learning signal for each individual completion” and thereby enabling granular credit assignment during reinforcement learning. Within PoLi-RL, PSRR is not a standalone objective; it is the mechanism by which the Stage II pairwise and listwise reward terms are computed inside a two-stage RL curriculum (Song et al., 5 Oct 2025).

1. Definition and problem setting

PSRR is defined in the setting of C-STS, where each example is a tuple

$x = (t_1, t_2, c, y),$

with $t_1$ and $t_2$ denoting two text segments, $c$ a natural-language condition, and $y \in [1,5]$ a human similarity score. The label space is fine-grained: 1 denotes completely dissimilar, 2 thematically related but dissimilar, 3 roughly equivalent with important differences, 4 mostly equivalent with unimportant differences, and 5 completely equivalent. The condition changes the semantics of similarity itself, so the task is not ordinary STS but conditional judgment over specific aspects (Song et al., 5 Oct 2025).

The PoLi-RL paper motivates reinforcement learning on two grounds. First, C-STS requires condition-dependent reasoning rather than pure lexical matching. Second, the main evaluation metric is Spearman correlation, which is non-differentiable and ranking-based. The optimization target is presented as

$$\theta^* = \underset{\theta}{\operatorname{arg\,max}} \ \mathbb{E}_{x \sim \mathcal{D},\, o \sim \pi_{\theta}(p)} [R(x, o)],$$

where the policy $\pi_\theta$ receives a prompt $p=\mathrm{INS}(x)$, generates an output sequence $o$, and the predicted score is parsed from the output as $\tilde y = \mathrm{Parse}(o)$ (Song et al., 5 Oct 2025).

PSRR is motivated by a negative result: naively applying listwise RL from scratch does not materially improve performance. The paper attributes this to two structural problems. The ranking objective is too complex at the start of training, and a single reward computed over the whole batch is too coarse to assign credit to individual completions. In C-STS, where distinctions such as 3 versus 4 or 4 versus 5 matter, a batch-wide list-level reward entangles unrelated predictions and obscures which completion actually caused a ranking error. PSRR is introduced precisely as the remedy for that granularity failure (Song et al., 5 Oct 2025).

2. Position within the PoLi-RL framework

PSRR appears inside PoLi-RL, a “Point-to-List Reinforcement Learning” framework built as a two-stage curriculum. The first stage is foundational skill acquisition. Its reward is described as a weighted combination of pointwise, binary, and format rewards. The exact pointwise term is

$t_1$0

with $t_1$1 and $t_1$2. The binary term rewards whether the prediction falls on the correct side of the similar-versus-dissimilar boundary at 3:

$t_1$3

The format reward enforces the required answer structure, although no explicit formula is given (Song et al., 5 Oct 2025).

The second stage performs fine-grained semantic refinement. Its reward is described as a weighted combination of pointwise, pairwise, and listwise terms. PSRR enters at exactly this point: it is the mechanism that reorganizes generated completions so that the pairwise and listwise rewards are computed within parallel slices rather than over a single flat batch. The framework therefore retains pointwise supervision as a stabilizing anchor while adding sliced ranking signals for local and global ordering refinement (Song et al., 5 Oct 2025).

Optimization is implemented with DAPO, described as an extension of GRPO. For each sample, the policy generates $t_1$4 completions, each receives a scalar reward, and the reward is converted into a relative advantage by within-group normalization:

$t_1$5

This detail is important for interpreting PSRR: PSRR produces more precise reward values, but those values still enter a group-relative policy optimization pipeline rather than replacing it (Song et al., 5 Oct 2025).

A common misconception is to treat PSRR as a pure listwise objective. The paper does not support that reading. PSRR is better characterized as a sliced reward-computation mechanism embedded in the Stage II hybrid reward. Both the pairwise and the listwise components are explicitly computed within slices, and both depend on the PSRR reorganization of outputs (Song et al., 5 Oct 2025).

3. Parallel slices and reward construction

The formal definition of a parallel slice is the core of PSRR. For a batch of $t_1$6 samples, the policy generates $t_1$7 completions for each sample:

$t_1$8

Each completion is parsed into a predicted score $t_1$9. Instead of flattening all $t_2$0 outputs into one list, PSRR forms $t_2$1 slices by fixing the completion index $t_2$2 and varying the sample index $t_2$3:

$t_2$4

The batch labels are

$t_2$5

A slice is therefore vertical across samples, not horizontal within a single sample. “Same-indexed completions from different samples” is the paper’s operative definition (Song et al., 5 Oct 2025).

Within each slice, PSRR computes ranking rewards at the level of individual completions. The central listwise term is

$t_2$6

This reward is high when the predicted rank of completion $t_2$7 within slice $t_2$8 is close to the ideal rank implied by the gold label $t_2$9 within the batch. Dividing by $c$0 normalizes the rank error to $c$1. The paper presents this as the mathematical heart of PSRR, because it yields a separate listwise reward for each completion rather than a single reward for the entire batch (Song et al., 5 Oct 2025).

The pairwise reward is also defined within slices. The paper introduces

$c$2

and then applies a piecewise rule: if the sign of the predicted difference disagrees with the sign of the true difference, the reward is 0; if the sign agrees, the reward consists of a base reward plus an additional term that decreases with the discrepancy between $c$3 and $c$4. The paper specifies $c$5 for paired samples. This pairwise term is local and exploits the dataset’s paired structure, whereas the listwise term is the more general PSRR mechanism (Song et al., 5 Oct 2025).

The significance of the construction is the paper’s “two-level decomposition.” First, the full set of generated outputs is decomposed into slices. Second, each completion within each slice receives its own ranking reward. This design localizes both the comparison set and the credit-assignment target. A plausible implication is that PSRR is best understood as a per-completion listwise surrogate, not merely as a smaller-batch variant of ordinary listwise RL.

4. Optimization procedure and implementation

The Stage II PSRR pipeline is operationally explicit. A batch of $c$6 C-STS samples is drawn. Each sample is converted into a prompt and sampled $c$7 times under the current policy. The outputs are parsed into scalar predictions, pointwise rewards are computed per completion, the predictions are reorganized into $c$8 slices, and pairwise and listwise ranking rewards are then computed within each slice. These reward components are combined with the pointwise term, normalized into within-sample advantages over the $c$9 completions, and used for a DAPO policy update (Song et al., 5 Oct 2025).

This means that PSRR changes the structure of reward computation without changing the underlying policy-gradient principle. The completion-level rewards generated by PSRR become the $y \in [1,5]$0 values that enter the standard group-normalization stage. In other words, PSRR sharpens the supervision before advantage estimation rather than replacing group-relative RL (Song et al., 5 Oct 2025).

The paper also discusses the computational profile of slice-based ranking. A sufficiently large slice size $y \in [1,5]$1 is described as crucial for a stable and meaningful ranking signal, but generating all $y \in [1,5]$2 completions and scoring them jointly is memory-intensive. The implementation therefore uses gradient accumulation: completions are generated across devices and accumulation steps, rewards and advantages are computed globally over the collected outputs, and forward/backward passes are then performed over smaller sub-batches while gradients are accumulated. This is a practical detail rather than a theoretical component, but it constrains how PSRR can be deployed at scale (Song et al., 5 Oct 2025).

Another implementation constraint is dependence on multiple completions per sample. PSRR presupposes a nontrivial $y \in [1,5]$3, because the method relies on grouping same-indexed completions across the batch. A plausible implication is that PSRR is most natural in on-policy RL settings that already sample multiple outputs per prompt, such as GRPO- or DAPO-style training.

5. Empirical behavior and ablations

The strongest empirical evidence for PSRR comes from the contrast with naive listwise RL. On the official C-STS benchmark, few-shot inference scores 37.9 Spearman, while “Naive RL (Listwise only)” reaches only 38.19. By contrast, PoLi-RL Stage I reaches 44.77, and the full Stage II system with pointwise, pairwise, and listwise components reaches 48.18 Spearman and 48.27 Pearson. The paper reports this as a new state of the art for the cross-encoder architecture and as the first successful application of RL to C-STS (Song et al., 5 Oct 2025).

The ablation results isolate the importance of ranking refinement and, within it, the listwise term enabled by PSRR. Removing the listwise component yields 46.71, while removing the pairwise component yields 47.6, compared with the full 48.18. The larger drop from removing listwise supervision suggests that the sliced listwise signal is the more consequential Stage II ingredient. The paper’s interpretation is that PSRR makes listwise RL informative enough to be useful, whereas naive batch-wide listwise reward is too coarse (Song et al., 5 Oct 2025).

Slice-size sensitivity further characterizes the method. Performance peaks at $y \in [1,5]$4 with 48.18, while $y \in [1,5]$5 gives 47.16, $y \in [1,5]$6 gives 47.44, $y \in [1,5]$7 gives 47.18, and $y \in [1,5]$8 gives 46.78. The authors interpret this as evidence that slices that are too small yield unstable ranking signals, whereas slices that are too large make the ranking task overly complex. That finding is central to PSRR as a design principle: localization helps, but over-expanding the ranking set reintroduces the difficulty that PSRR is intended to avoid (Song et al., 5 Oct 2025).

The paper also reports distributional analysis showing the highest density of perfect predictions and the lowest density in large-error regions for PoLi-RL. This does not isolate PSRR in a single statistic, but it is consistent with the claim that the sliced hybrid reward produces more stable and reliable fine-grained predictions than either few-shot prompting or naive listwise RL (Song et al., 5 Oct 2025).

6. Relation to adjacent ranking-reward paradigms

PSRR belongs to a broader family of reward constructions that replace absolute scalar supervision with relative order information, but its specific slice construction is distinctive. A close analogue is Reinforcement Learning with Relative Rewards (RLRR), which replaces absolute reward values with within-group relative ordering signals and maps ranks into bounded rewards before standard group-based advantage computation. For verifiable tasks, it uses Hybrid Relative Reward,

$y \in [1,5]$9

and for open-ended tasks it uses Pure Relative Reward,

$$\theta^* = \underset{\theta}{\operatorname{arg\,max}} \ \mathbb{E}_{x \sim \mathcal{D},\, o \sim \pi_{\theta}(p)} [R(x, o)],$$0

The paper explicitly evaluates fixed-size subgroups with a Ranking Reward Model and then hierarchically re-ranks them, which makes it a close conceptual match to PSRR-style reward shaping even though it does not use the term “Parallel Slice Ranking Reward” (Niu et al., 30 Jan 2026).

A second adjacent design appears in R$$\theta^* = \underset{\theta}{\operatorname{arg\,max}} \ \mathbb{E}_{x \sim \mathcal{D},\, o \sim \pi_{\theta}(p)} [R(x, o)],$$1-SQL, where ranking is performed over execution-result equivalence classes rather than individual candidates. Candidates that yield the same execution result are grouped into $$\theta^* = \underset{\theta}{\operatorname{arg\,max}} \ \mathbb{E}_{x \sim \mathcal{D},\, o \sim \pi_{\theta}(p)} [R(x, o)],$$2, and group ranking combines a cross-group preference term,

$$\theta^* = \underset{\theta}{\operatorname{arg\,max}} \ \mathbb{E}_{x \sim \mathcal{D},\, o \sim \pi_{\theta}(p)} [R(x, o)],$$3

with a pointwise-group utility term,

$$\theta^* = \underset{\theta}{\operatorname{arg\,max}} \ \mathbb{E}_{x \sim \mathcal{D},\, o \sim \pi_{\theta}(p)} [R(x, o)],$$4

Groups are then sorted lexicographically by $$\theta^* = \underset{\theta}{\operatorname{arg\,max}} \ \mathbb{E}_{x \sim \mathcal{D},\, o \sim \pi_{\theta}(p)} [R(x, o)],$$5. This is not PSRR in terminology or mechanics, but it is another instance of partition-aware ranking reward, where the unit of comparison is a structured subset rather than an isolated candidate (Han et al., 28 Apr 2026).

A third related direction is R4, which learns rewards from ordinal trajectory ratings by sampling one trajectory per class, predicting returns, applying a differentiable soft-rank operator, and minimizing the ranking mean squared error

$$\theta^* = \underset{\theta}{\operatorname{arg\,max}} \ \mathbb{E}_{x \sim \mathcal{D},\, o \sim \pi_{\theta}(p)} [R(x, o)],$$6

The paper processes multiple sampled ranking sets per update, effectively performing repeated parallel setwise ranking. Its theoretical contribution is that, under stated assumptions, the minimizers of the rMSE objective are exactly the reward functions that preserve class ordering. This suggests a broader interpretation of PSRR: ranking rewards can be organized around slices, groups, or class-balanced subsets so long as the reward construction preserves ordinal structure and supports stable credit assignment (Kharyal et al., 14 Jan 2026).

The limitations of PSRR are correspondingly clear. It depends on a tunable slice size $$\theta^* = \underset{\theta}{\operatorname{arg\,max}} \ \mathbb{E}_{x \sim \mathcal{D},\, o \sim \pi_{\theta}(p)} [R(x, o)],$$7; it incurs nontrivial computational and memory overhead; it relies on multiple completions per sample; its pairwise term exploits the paired structure of C-STS and is therefore task-specific; and, despite its Spearman-oriented motivation, it is not an exact Spearman objective. The paper instead constructs rank-aligned surrogate rewards whose empirical value comes from finer credit assignment rather than from direct optimization of the evaluation metric (Song et al., 5 Oct 2025).