Reward Normalization (ReNorm)

• Reward Normalization (ReNorm) is a collection of techniques that transform rewards, returns, or advantages in reinforcement learning to ensure scale alignment, boundedness, and improved credit assignment.
• It encompasses methods like per-prompt Z-scoring, min–max normalization, and decoupled normalization, each tailored to preserve task semantics and mitigate reward hacking.
• Empirical studies demonstrate that ReNorm stabilizes training and enhances performance by effectively balancing dense process rewards with sparse outcome signals in diverse RL settings.

Reward normalization (ReNorm) denotes a family of transformations applied to rewards, returns, or advantages before policy optimization. Across recent reinforcement-learning work for LLMs and related RL settings, the term covers several distinct operations: affine coupling of dense process rewards with sparse outcome rewards, per-prompt Z-scoring, per-reward decoupled normalization, verifier-stratified min–max normalization, skill-conditional normalization, adaptive Beta-based normalization, and advantage-preserving reward balancing on the MDP itself (Xu et al., 29 Sep 2025, Ge et al., 30 Jan 2026, Liu et al., 8 Jan 2026, Tao et al., 8 Oct 2025). What unifies these uses is the attempt to control scale, preserve task semantics, stabilize optimization, and improve credit assignment without discarding information that is relevant to policy improvement.

1. Scope and principal formulations

The literature uses “reward normalization” at several granularities. In some works, the normalized object is a step-wise reward attached to individual tokens; in others it is a per-prompt sequence-level advantage, a reward component in a multi-objective vector, or the reward function of an MDP itself. Earlier work also used the term for intrinsic-reward rescaling and Q-value normalization in exploration-driven RL, defining $R_t^{\text{intrinsic}} = \phi(loss_t)/\mathcal U$, a softmax-normalized Q-value $\mathcal Q_t(a)$, and a clipped target $y_t^{ns}\le 1$ (Choi et al., 2019).

Formulation	Normalization rule	Conditioning domain
PPR / ReNorm	$r_{p,t}=\hat r_{p,t}+r_o-1$	step-wise process score plus final correctness (Xu et al., 29 Sep 2025)
GRPO	$A_i(o)=\frac{r-\mu_i}{\sigma_i}$	per prompt (Ge et al., 30 Jan 2026)
GDPO	normalize each $r_k$ over the group, then batch-normalize the sum	per reward, per prompt, then batch (Liu et al., 8 Jan 2026)
PAPO	$A_{\text{out}}$ over all responses, $A_{\text{proc}}$ only among correct responses	all responses vs. correct subset (Tan et al., 27 Mar 2026)
HERO	min–max RM normalization within verifier strata, mapped to bounded bands	incorrect vs. correct verifier groups (Tao et al., 8 Oct 2025)
PDCR	min–max normalization within visual or textual clusters	per skill cluster (Yoon et al., 13 May 2026)
BNPO	$\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$	per-question Bernoulli success probability with Beta modeling (Xiao et al., 3 Jun 2025)

These formulations differ less in surface syntax than in the subset over which statistics are computed. Some methods normalize over all samples for a prompt, some only within reward-defined or skill-defined strata, and some avoid empirical moments entirely by using fixed transforms or analytically chosen parametric normalizers. This suggests that ReNorm is best understood as a design space organized by conditioning structure rather than as a single algorithm.

2. Hybrid reward normalization for process-supervised non-verifiable tasks

The most explicit recent use of the name “ReNorm” appears in Principle Process Reward (PPR), which trains a search-agent LLM with a sparse outcome reward $r_o\in\{0,1\}$ and dense process rewards $\mathcal Q_t(a)$0 produced by a principle-based process reward model (Xu et al., 29 Sep 2025). The stated problem is that naively mixing the two produces instability and reward hacking: the binary outcome reward is delayed and sparse, while the process rewards are continuous and appear at many steps; optimizing the latter in isolation can improve “local quality” without improving final correctness.

The implemented normalization is

$\mathcal Q_t(a)$1

It is motivated by the centered form $\mathcal Q_t(a)$2 with $\mathcal Q_t(a)$3 and $\mathcal Q_t(a)$4, but the implementation uses the fixed symmetric prior $\mathcal Q_t(a)$5. The consequences are explicit. If $\mathcal Q_t(a)$6, then $\mathcal Q_t(a)$7, so all step rewards are non-positive. If $\mathcal Q_t(a)$8, then $\mathcal Q_t(a)$9, so step rewards are non-negative. The paper therefore characterizes the construction as providing sign alignment, scale alignment, boundedness, and centering for stable GAE/PPO optimization.

Token-level reward placement is also part of the scheme. For token index $y_t^{ns}\le 1$0,

$y_t^{ns}\le 1$1

These rewards are fed into GAE and PPO with $y_t^{ns}\le 1$2. ReNorm is applied at training time for every trajectory and every step, but not at evaluation time, where performance is measured by exact match on final answers. The paper also emphasizes that the implemented normalization is global and fixed: it does not maintain running statistics, perform per-task re-estimation, or apply additional z-scoring or min–max scaling beyond the native $y_t^{ns}\le 1$3 bounds and the affine shift to $y_t^{ns}\le 1$4.

The central claim is semantic rather than merely numeric. ReNorm ensures that failed trajectories never receive positive step credit, while successful trajectories retain dense within-trajectory differentiation. The theoretical appendix further states that the broadcast-and-centering scheme makes the expected objective proportional to $y_t^{ns}\le 1$5, with $y_t^{ns}\le 1$6 and $y_t^{ns}\le 1$7. A plausible implication is that process supervision becomes useful precisely when it is at least weakly informative about eventual success.

3. Conditional, decoupled, and structured normalization

A major theme in subsequent work is that reward normalization becomes more effective when it respects the internal structure of the reward signal rather than collapsing heterogeneous quantities into a single scalar too early. Several recent methods differ mainly in how they choose the domain of normalization.

GDPO begins from the observation that directly applying GRPO to summed multi-reward rollouts causes distinct reward combinations to collapse into identical advantage values (Liu et al., 8 Jan 2026). Its remedy is to normalize each reward component separately within the prompt group,

$y_t^{ns}\le 1$8

then sum the normalized components, optionally with weights, and finally batch-normalize the aggregate advantage. The paper’s constructive examples show that GRPO can map $y_t^{ns}\le 1$9, $r_{p,t}=\hat r_{p,t}+r_o-1$0, and $r_{p,t}=\hat r_{p,t}+r_o-1$1 reward pairs to the same normalized advantage pair $r_{p,t}=\hat r_{p,t}+r_o-1$2, whereas GDPO preserves more distinct advantage groups.

PAPO applies a closely related decoupling to outcome and process supervision (Tan et al., 27 Mar 2026). It defines

$r_{p,t}=\hat r_{p,t}+r_o-1$3

over all responses in the group, but normalizes process scores only over the correct subset $r_{p,t}=\hat r_{p,t}+r_o-1$4: $r_{p,t}=\hat r_{p,t}+r_o-1$5 The combined advantage is $r_{p,t}=\hat r_{p,t}+r_o-1$6. This design preserves correctness anchoring while differentiating reasoning quality only where correctness has already been achieved.

HERO instead uses verifier-stratified normalization for hybrid verifier-plus-RM training (Tao et al., 8 Oct 2025). Responses are split into $r_{p,t}=\hat r_{p,t}+r_o-1$7 and $r_{p,t}=\hat r_{p,t}+r_o-1$8 by the verifier label $r_{p,t}=\hat r_{p,t}+r_o-1$9, and reward-model scores are min–max normalized within each stratum. Incorrect responses are mapped to $A_i(o)=\frac{r-\mu_i}{\sigma_i}$0, and correct responses to $A_i(o)=\frac{r-\mu_i}{\sigma_i}$1. The stated effect is that the dense reward model can refine ranking within each verifier group but cannot invert correctness ordering, because even the best incorrect response remains below the worst correct response.

PDCR makes the conditioning variable latent rather than verifier-defined (Yoon et al., 13 May 2026). It uses a Visual Dependence Score and Otsu thresholding to partition reasoning steps into visual and textual clusters, then normalizes discounted confidence-gain returns separately within $A_i(o)=\frac{r-\mu_i}{\sigma_i}$2 and $A_i(o)=\frac{r-\mu_i}{\sigma_i}$3. The paper characterizes the failure of a single global normalization as “mixture-induced signal degradation,” because global min–max statistics are dominated by the majority textual steps and therefore distort the minority visual ones.

HPO and A-HPO modify a different axis of normalization: the interaction between advantage sign and response length in sparse-reward GRPO-style training (Sana et al., 28 May 2026). They replace response-level length normalization $A_i(o)=\frac{r-\mu_i}{\sigma_i}$4 with mean-length normalization $A_i(o)=\frac{r-\mu_i}{\sigma_i}$5, remove per-group standard deviation from the advantage, and introduce sign-asymmetric weights

$A_i(o)=\frac{r-\mu_i}{\sigma_i}$6

A-HPO sets $A_i(o)=\frac{r-\mu_i}{\sigma_i}$7 adaptively from the empirical sign ratio $A_i(o)=\frac{r-\mu_i}{\sigma_i}$8. In this setting, normalization is explicitly tied to update symmetry rather than only to variance.

4. Theoretical interpretations

Theoretical work on ReNorm increasingly treats normalization as more than a variance heuristic. One line of analysis shows that GRPO’s per-prompt standard-deviation normalization behaves like an adaptive gradient method (Ge et al., 30 Jan 2026). Under Bernoulli rewards and a log-linear policy, the paper derives

$A_i(o)=\frac{r-\mu_i}{\sigma_i}$9

and proves that local curvature satisfies

$r_k$0

Because the normalization scales each prompt’s gradient by the inverse reward variance, it implicitly approximates a curvature-adaptive step size. The same work identifies three training phases—early acceleration, transition, and late interference—under which the benefit of standard-deviation normalization changes as reward variance and feature orthogonality evolve.

A second line replaces empirical standardization with parametric distributional normalization. BNPO models the per-question success probability $r_k$1 as Beta-distributed and defines

$r_k$2

Under the paper’s assumptions, the variance of the resulting policy-gradient estimator is finite iff $r_k$3 and $r_k$4, and is uniquely minimized at

$r_k$5

Within that framework, REINFORCE with baseline corresponds to $r_k$6, while GRPO corresponds, up to scaling, to $r_k$7 (Xiao et al., 3 Jun 2025). The paper therefore presents BNPO as a generalization of both methods under binary-valued rewards.

A third line studies normalization at the MDP level rather than at the advantage-estimator level. “MDP Geometry, Normalization and Reward Balancing Solvers” defines transformations that shift value at one state by $r_k$8 while preserving all action advantages, and uses them to map any MDP to a normal form in which the optimal value function is identically zero (Mustafin et al., 2024). “On Reward-Balancing Methods for Reinforcement Learning” formalizes the same family as

$r_k$9

and proves that if $A_{\text{out}}$0, then $A_{\text{out}}$1 and $A_{\text{out}}$2 for all policies (Baroncini et al., 22 Apr 2026). In this setting, normalization is not scaling in the statistical sense at all; it is an advantage-preserving transformation along an affine orbit of equivalent rewards.

These theories are not mutually exclusive. They point to distinct invariants: curvature matching in GRPO, estimator-variance minimization in BNPO, and exact preservation of advantages in reward-balancing methods. Together they show that “normalization” can denote adaptive preconditioning, structured rescaling, or canonicalization of the reward function itself.

5. Empirical behavior and observed effects

The empirical case for ReNorm depends strongly on the reward structure. In PPR, the ablation reported on NQ, TriviaQA, and PopQA for 3B-Instruct shows that ReNorm outperforms the alternatives listed in the paper: “w/o Norm” reaches average EM $A_{\text{out}}$3, “Rescale Norm” $A_{\text{out}}$4, “Reduced Reward” $A_{\text{out}}$5, and “ReNorm (ours)” $A_{\text{out}}$6 (Xu et al., 29 Sep 2025). The same paper reports that Figure 1(b) shows monotonically increasing and stable reward curves for ReNorm, whereas “w/o Norm,” “Rescale Norm,” and “Reduced Reward” stagnate, fluctuate strongly, or collapse.

PAPO’s decoupled normalization yields a different empirical signature: it keeps training informative after outcome-only GRPO exhausts its variance (Tan et al., 27 Mar 2026). On OlympiadBench, PAPO reaches $A_{\text{out}}$7 versus $A_{\text{out}}$8 for ORM. The paper also reports that the zero-advantage ratio rises from approximately $A_{\text{out}}$9 to $A_{\text{proc}}$0 for ORM-only GRPO on Qwen2.5-7B, whereas PAPO holds it at approximately $A_{\text{proc}}$1. In the authors’ interpretation, correct-subset process normalization continues to differentiate among correct responses precisely when outcome-only normalization collapses.

HERO’s verifier-stratified normalization is most visible when verifier signals are uniform within a prompt group (Tao et al., 8 Oct 2025). For Qwen3-4B-Base trained on easy-to-verify data, the paper reports average performance $A_{\text{proc}}$2 for HERO on easy-to-verify evaluation, compared with $A_{\text{proc}}$3 for verifier-only and $A_{\text{proc}}$4 for RM-only. On hard-to-verify evaluation under the same training regime, HERO reaches $A_{\text{proc}}$5, compared with $A_{\text{proc}}$6 for verifier-only and $A_{\text{proc}}$7 for RM-only. The paper also states that naive weighted sums of rule-based and RM rewards underperform HERO substantially, which it attributes to the absence of stratified normalization and bounded correctness-preserving bands.

For multi-reward RL, GDPO’s principal empirical claim is that decoupling normalization per reward dimension improves both resolution and stability (Liu et al., 8 Jan 2026). It consistently outperforms GRPO on tool calling, math reasoning, and coding reasoning, and the paper’s toy examples and “distinct advantage group” counts make the mechanism visually explicit: scalar normalization after reward summation collapses many more reward configurations than per-reward normalization before aggregation.

In sparse-reward GRPO-style training, HPO and A-HPO show that the choice of normalization can materially change early learning dynamics (Sana et al., 28 May 2026). On TeleLogs, A-HPO reaches final reward $A_{\text{proc}}$8, compared with $A_{\text{proc}}$9 for SAPO, $\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$0 for GSPO, and $\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$1 for GRPO, while maintaining a comparable response length. Fixed-$\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$2 ablations show that $\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$3 yields reward $\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$4 with mean length $\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$5, $\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$6 yields $\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$7 with mean length $\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$8, $\frac{R(q,o)-p(q)}{f_N(p(q);\alpha,\beta)}$9 yields $r_o\in\{0,1\}$0 with mean length $r_o\in\{0,1\}$1, and A-HPO yields $r_o\in\{0,1\}$2 with mean length $r_o\in\{0,1\}$3. The paper interprets this as evidence that neither positive-only nor fully symmetric updates are optimal under early sparse reward.

Across these studies, the same empirical pattern recurs. Normalization helps most when raw reward geometry is misleading: dense signals can outrun correctness, heterogeneous reward dimensions can collapse into indistinguishable scalar advantages, minority skills can be washed out by majority-skill statistics, and sparse binary rewards can induce strongly imbalanced positive/negative update counts.

6. Limitations, misconceptions, and open directions

A recurrent misconception is that reward normalization is merely a generic stabilizer that can be bolted onto any reward design. The recent literature is more restrictive. PPR explicitly argues that optimizing raw process rewards can improve “local quality” without improving final correctness, and therefore makes final-answer correctness enter the sign of every step reward (Xu et al., 29 Sep 2025). PAPO likewise assigns $r_o\in\{0,1\}$4 to incorrect responses, and HERO bounds reward-model influence inside verifier-defined bands (Tan et al., 27 Mar 2026, Tao et al., 8 Oct 2025). These constructions indicate that normalization alone does not resolve reward hacking; it must be paired with a conditioning structure that anchors the dense signal to a trusted objective.

Another misconception is that global normalization is always sufficient. PDCR shows that global min–max normalization across mixed visual and textual steps can statistically distort the minority perception signal because the global extrema are dominated by textual steps (Yoon et al., 13 May 2026). GDPO makes an analogous point for multi-reward scalarization: normalization after summing heterogeneous rewards can collapse distinct reward combinations into identical advantages (Liu et al., 8 Jan 2026). This suggests that the subset over which statistics are computed is often more consequential than the algebraic form of the normalization itself.

The methods also have concrete limitations. BNPO’s theoretical guarantees are derived for binary-valued rewards and rely on Bernoulli/Beta modeling; the paper leaves continuous-reward generalization open (Xiao et al., 3 Jun 2025). HERO depends on a reasonably good verifier and on tuning $r_o\in\{0,1\}$5 and the parameters of its difficulty-weighting function (Tao et al., 8 Oct 2025). PAPO requires enough correct responses in a group for correct-subset normalization to be informative, otherwise it degenerates to ORM-only GRPO (Tan et al., 27 Mar 2026). PDCR incurs extra forward passes to compute visual dependence and may face noisy min–max estimates in small-cluster regimes (Yoon et al., 13 May 2026). GRPO-style standard-deviation normalization itself is not uniformly benign: the local-curvature analysis notes a speed-versus-calibration trade-off and diminishing returns when cross-prompt interference rises late in training (Ge et al., 30 Jan 2026).

Open problems therefore cluster around richer outcome types, stronger conditional structure, and more faithful uncertainty modeling. Several papers explicitly propose multi-cluster or hierarchical extensions beyond binary or two-skill decompositions, adaptive weighting between outcome and process signals, generalization beyond binary rewards, and integrated theories for normalization together with clipping, KL penalties, or entropy bonuses (Yoon et al., 13 May 2026, Tan et al., 27 Mar 2026, Xiao et al., 3 Jun 2025, Ge et al., 30 Jan 2026). A plausible implication is that future ReNorm research will move further away from monolithic “subtract mean, divide by std” recipes and toward normalization schemes that are conditional on reward source, skill, task, verifier state, or model-estimated uncertainty.