Stepwise Reward Centering in RLHF

• Stepwise reward centering is a per-training-step transformation that recalibrates raw reward scores by subtracting a prompt-specific value estimate, yielding a contrastive signal for reward model improvement.
• It employs a modular framework with a shared encoder for both the value head and reward head, filtering responses via a curriculum-driven threshold to balance positive and negative advantages.
• The approach addresses RLHF challenges by converting on-policy rollouts into self-supervised supervision, demonstrating improved reward calibration and policy performance across multiple benchmarks.

Searching arXiv for papers on “stepwise reward centering” and closely related reward-model / RLHF formulations. arXiv search query: "stepwise reward centering" arXiv search query: "reward model on-policy value anchored feedback RLHF reward centering" “Stepwise reward centering” (Editor's term) denotes a per-training-step transformation in which raw reward-model scores for on-policy responses are centered by a prompt-specific value estimate before being reused as supervision for reward-model and policy updates. In the formulation introduced by SAVE—“Self-supervised reward model improvement via Value-Anchored On-policy feedback”—the centered quantity is the response-level RM advantage,

$a^{\RM}(x,y)=R_{\omega,\phi}(x,y)-V_{\omega,\psi}(x),$

where $R_{\omega,\phi}(x,y)$ is the reward head and $V_{\omega,\psi}(x)$ is a prompt-specific value head sharing the same encoder. This construction addresses two stated RLHF bottlenecks: distributional drift as the policy $\pi_\theta$ evolves beyond the static RM training distribution, and the labeling bottleneck associated with fresh human preferences or external judges (Wang et al., 29 May 2026).

1. Conceptual role in on-policy reward-model training

Modern RLHF pipelines are described as relying on a fixed reward model $r_\xi(x,y)$ trained on offline preference data. Two failure modes follow from this setup. First, as the policy improves, it visits regions that are under-supervised by the static RM, which can produce miscalibration and reward hacking. Second, obtaining new supervision from human annotation or judge models remains costly. SAVE addresses both issues by turning the policy’s own rollouts into self-supervised supervision for the RM.

Within that framework, stepwise reward centering consists of comparing each sampled response’s scalar reward against an adaptive anchor rather than against another externally labeled response. The anchor is prompt-specific and is recomputed during RL training. This suggests that the centering operation is not merely a normalization heuristic; it is the mechanism that converts on-policy reward scores into a contrastive signal usable for RM improvement without extra labels.

The centering is “stepwise” in a literal algorithmic sense. At each RL training step, the current policy samples a batch of on-policy responses for each prompt, and the current value head provides the anchor against which those responses are evaluated. The procedure is therefore tied to the current policy distribution rather than to a static offline corpus.

2. Mathematical formulation of the centered signal

The SAVE framework augments the RM with a prompt-specific value head

$V_{\omega,\psi}(x)=v_\psi\bigl(f_\omega(x)\bigr),$

where $f_\omega(\cdot)$ is the shared encoder and $v_\psi$ is a lightweight MLP. The corresponding on-policy value is

$V^{\pi_\theta}(x)=\E_{y\sim\pi_\theta(\cdot\mid x)}\bigl[R_{\omega,\phi}(x,y)\bigr].$

The centered reward signal is the response-level RM advantage

$a^{\RM}(x,y)=R_{\omega,\phi}(x,y)-V_{\omega,\psi}(x).$

Responses are then filtered by a curriculum-driven threshold $R_{\omega,\phi}(x,y)$0: $R_{\omega,\phi}(x,y)$1 where $R_{\omega,\phi}(x,y)$2 is the logistic sigmoid and $R_{\omega,\phi}(x,y)$3 decays from a strict initial margin $R_{\omega,\phi}(x,y)$4 to $R_{\omega,\phi}(x,y)$5. The retained responses are partitioned into

$R_{\omega,\phi}(x,y)$6

This formulation makes the centering explicit. Positive-advantage responses are those above the prompt-specific anchor; negative-advantage responses are those below it. Prompts for which either $R_{\omega,\phi}(x,y)$7 or $R_{\omega,\phi}(x,y)$8 are skipped, because they do not provide contrastive supervision. A plausible implication is that the anchor is used both as a calibration device and as a selection boundary for confidence-aware self-training.

3. Per-step algorithmic workflow

SAVE proceeds in three stages per RL step. First, in adaptive feedback filtering, a batch of prompts $R_{\omega,\phi}(x,y)$9 is sampled, and for each $V_{\omega,\psi}(x)$0, $V_{\omega,\psi}(x)$1 responses are drawn from $V_{\omega,\psi}(x)$2. RM advantages are computed, the threshold $V_{\omega,\psi}(x)$3 is applied, and the retained responses are split into $V_{\omega,\psi}(x)$4 and $V_{\omega,\psi}(x)$5. Only prompts with both subsets non-empty are kept.

Second, in self-supervised reward-model improvement, the value anchor is fixed and $V_{\omega,\psi}(x)$6 is updated by descending the value-anchored contrastive loss averaged over the retained prompts. The value-head parameters $V_{\omega,\psi}(x)$7 are then updated by descending a regression loss over the full sampled groups. The paper specifies that these are block-coordinate steps with stop-gradient separation to prevent cross-contamination.

Third, policy model optimization is performed using the updated RM. Advantages and filtered sets are recomputed, rewards are evaluated on the filtered responses, and $V_{\omega,\psi}(x)$8 is updated with a standard on-policy RL algorithm using those rewards and KL regularization (Wang et al., 29 May 2026).

This ordering is central to the notion of stepwise centering. The centering operation is not computed once and reused indefinitely; it is recalibrated, consumed for RM updating, and then recomputed before the policy step. That repeated recentering aligns reward supervision with the evolving policy distribution.

4. Loss design and the function of the anchor

Given a prompt $V_{\omega,\psi}(x)$9 with $\pi_\theta$0 and $\pi_\theta$1, the reward head is updated using the value-anchored contrastive loss

$\pi_\theta$2

Positive-advantage responses are pushed above the anchor and negative-advantage ones below it.

Independently, the value head is trained to match the group mean: $\pi_\theta$3 The paper characterizes this as yielding a calibrated baseline for subsequent advantage computation.

These two losses clarify what “centering” accomplishes. The anchor is neither an auxiliary critic detached from RM training nor a purely diagnostic statistic. It is itself learned to track the mean reward for a prompt, while the reward head is optimized relative to that learned center. This suggests a division of labor: the value head defines the local center of mass of reward scores for a prompt, and the reward head learns the ordering structure around that center.

A common misconception is to treat such a value estimate as only a policy-optimization baseline. In SAVE, the value head also supplies supervision structure for the RM itself. The centered signal therefore operates at the level of reward-model improvement, not solely at the level of policy-gradient variance control.

5. Compatibility with on-policy RL and empirical performance

The framework is described as largely agnostic to the underlying on-policy optimizer. In the reported experiments it is paired with GRPO, RLOO, and GSPO, and the only modification to the standard RL loop is the interleaved RM update using on-policy rollouts and the recomputation of advantages and filtered rewards before the policy step. Grouped rollouts produced by these critic-free methods are directly repurposed for advantage partitioning.

Evaluation covers six RM benchmarks using two policy backbones, Qwen2.5-3B and Qwen3-4B. The six benchmarks are RewardBench, RewardBench 2, RM-Bench, PPE Preference, PPE Correctness, and JudgeBench. On the Qwen3-4B backbone, the average across the six tasks is reported as 76.0% for the initial RM baseline, 76.4% for continual offline RM, 77.3% for SAVE with GRPO, 76.6% for SAVE without curriculum, and 76.5% for SAVE without policy update. The gains are stated to hold across GRPO, RLOO, and GSPO within $\pi_\theta$4 points (Wang et al., 29 May 2026).

Downstream policy performance is also reported. For Qwen3-4B on AlpacaEval 2 and Arena-Hard v2.0, GRPO baseline reaches LC 51.68%, WR 59.81%, and Arena 30.2%; adding SAVE in co-training yields LC 53.28%, WR 62.69%, and Arena 33.5%; using the improved RM to train a fresh policy yields LC 54.24%, WR 62.40%, and Arena 33.9%. These results are presented as evidence that improved reward modeling translates into stronger policies.

From the standpoint of stepwise reward centering, the empirical significance is that the centered, anchor-based transformation is not only a filtering heuristic. It is implicated in benchmark-level RM gains and in policy-level improvements across multiple RL algorithms and policy backbones.

6. Interpretation, scope, and limitations

The strengths attributed to the framework are that it is fully self-supervised, adaptive, and modular. It is fully self-supervised because no extra human or external judge labels are required. It is adaptive because on-policy rollouts focus RM training on current weaknesses. It is modular because it is compatible with critic-free on-policy RL methods.

The same source also identifies several limitations. Experiments are conducted on 3–4B-parameter models, so behavior on $\pi_\theta$5 larger LLMs is untested. Evaluation relies on benchmarks and LLM-based judges that may not capture subjective or safety-critical nuances. The method introduces extra RM updates and response sampling, increasing memory and compute by approximately 20% (Wang et al., 29 May 2026).

Several interpretive points follow. First, stepwise reward centering is not equivalent to collecting new preference labels; rather, it converts reward-graded on-policy responses into supervision through a prompt-specific value anchor. Second, it is not a static recalibration of an RM after RL training; it is an interleaved process that updates the RM inside the on-policy loop. Third, it is not tied to a single optimization algorithm, since the reported results span GRPO, RLOO, and GSPO.

Future directions proposed for the framework include multi-scale feedback such as step-level signals, more efficient sampling, human-in-the-loop calibration, and theoretical analysis of convergence in the RM–policy minimax game. This suggests that stepwise reward centering, as instantiated by value-anchored on-policy feedback, is best understood not as an isolated trick but as a broader design principle for continual reward-model adaptation under policy drift.