Adaptive Reward Normalization (ARN)

• Adaptive Reward Normalization (ARN) is a suite of techniques that dynamically rescale, center, and adjust reward signals in reinforcement learning to achieve invariance and stable updates.
• These methods employ online exponential moving averages and parameter adjustments to normalize targets and prevent common issues like exploding or vanishing gradients.
• ARN frameworks, including ART, POP, and BNPO, have been shown to enable faster convergence and robust performance across varied tasks such as Atari, continuous control, and LLM reinforcement learning.

Adaptive Reward Normalization (ARN) refers to a family of techniques for rescaling, centering, or otherwise adapting the reward or target signal in reinforcement learning (RL)—and by extension, other learning systems—so that network outputs, value function updates, or policy gradients are invariant (in law or function) to the arbitrary scale and shift of the underlying rewards. ARN methods are motivated by both optimization stability and invariance desiderata: in unconstrained RL tasks, rewards or value targets may vary by orders of magnitude, leading to exploding or vanishing gradients, high variance in policy updates, and sensitivity of outcomes to domain heuristics. Core ARN schemes maintain online (typically exponential moving average) estimates of reward or return statistics, rescale targets or advantages accordingly, and, where relevant, adjust network parameters to preserve output continuity. This article synthesizes the main ARN frameworks, underlying theory, algorithmic forms, and empirical findings across value-based RL, policy-gradient RL, process- and outcome-driven LLM RL, and MDP analytic normalization.

1. Scale Invariance and the Objectives of Adaptive Reward Normalization

The primary rationale for ARN arises from the lack of intrinsic scale invariance in most learning algorithms. In value-based RL, as in Atari benchmarks, the magnitude of returns and TD targets changes during policy improvement and varies dramatically between tasks. Absent normalization, large-magnitude targets force the network into regimes where gradient-based updates require intractably small learning rates, while small-magnitude targets result in slow, inefficient learning. Early approaches address this by hand-tuned clipping of rewards—a non-invariant, domain-specific heuristic that can distort policy behavior.

ARN generalizes and replaces such heuristics by embedding two desiderata:

• Adaptive Rescaling of Targets (ART): Maintain a normalization such that targets (or TD returns, advantages, or per-step rewards) are centered to zero mean and rescaled to unit variance (or any prescribed scale), rendering downstream optimization insensitive to absolute magnitudes.
• Preserving Outputs Precisely (POP): When the normalization changes, adjust network parameters or reward landscapes so that unnormalized predictions remain unchanged, preventing destructive oscillations or “forgetting” caused by normalization drift.

Together, ART and POP render the algorithm invariant (modulo bookkeeping) to affine transformations of the reward or return signal and eliminate step-size sensitivity across domains (Hasselt et al., 2016).

2. Mathematical Formulations and Algorithmic Implementations

Target Normalization in Value-Based RL

Consider a value-function approximator $f(x;\theta,W,b) = W h(x;\theta) + b$, where $h(\cdot;\theta)$ is the penultimate network. Introduce normalization parameters $(\mu_t,\sigma_t)$, compute normalized targets $\tilde Y_t = (Y_t - \mu_t)/\sigma_t$, and minimize the squared error loss $\ell_t = \frac{1}{2}\|g(X_t) - \tilde Y_t\|^2$ with $g(x) = W h(x) + b$. On normalization parameter update, rescale $(W,b)$ via

$$W \leftarrow \frac{\sigma}{\sigma_{\text{new}}} W, \quad b \leftarrow \frac{\sigma b + \mu - \mu_{\text{new}}}{\sigma_{\text{new}}},$$

so that $f(x)$ is unchanged (Hasselt et al., 2016). Exponential moving averages are used to track $\mu_t$ and $h(\cdot;\theta)$0:

$h(\cdot;\theta)$1

This ensures bounded normalized targets and avoids instability from heavy-tailed returns.

Policy-Gradient ARN: Z-Score and Beyond

In group-wise/sequence-level RL (e.g., Group Relative Policy Optimization, GRPO), the Arnold estimator is constructed by prompt/sample:

• Compute the mean and standard deviation of obtained rewards for each context batch: $h(\cdot;\theta)$2.
• Normalize each advantage: $h(\cdot;\theta)$3. The policy gradient is then:

$h(\cdot;\theta)$4

This Z-score normalization implements a gradient scaling analogous to local-curvature-aware step size adaptation (Ge et al., 30 Jan 2026).

Affine Normalization via MDP State-Wise Shifts

A fully analytic ARN approach operates at the MDP level, performing state-wise affine reward shifts that are advantage-preserving. For each state $h(\cdot;\theta)$5, the normalization

$h(\cdot;\theta)$6

guarantees that all policy advantages remain invariant. Iterative application of these shifts (as in the VFS algorithm) yields reward balancing solvers that can converge more rapidly than classical value iteration in environments with strong bottlenecks or self-loops (Mustafin et al., 2024).

Binary/Process/Outcome Reward Normalization

For process-supervised RL (e.g., PPR+ReNorm for LLM tasks), normalization is performed by centering process and outcome rewards:

$h(\cdot;\theta)$7

where $h(\cdot;\theta)$8 and $h(\cdot;\theta)$9 are running averages (potentially adapted online), ensuring zero-mean per-step rewards and anchoring local to final outcome supervision (Xu et al., 29 Sep 2025).

3. Advanced ARN: Distributional and Adaptive Scaling Approaches

Beyond standard Z-scoring, recent developments treat the normalization distribution itself as part of the adaptation protocol. Beta Normalization Policy Optimization (BNPO) uses a dynamically updated Beta distribution to model the evolving probability of correct answers (for binary rewards), with policy-gradient normalization:

$(\mu_t,\sigma_t)$0

where $(\mu_t,\sigma_t)$1 is the Beta PDF with parameters $(\mu_t,\sigma_t)$2 updated via method-of-moments to minimize the variance of the gradient estimator. This generalizes both baseline-subtracted REINFORCE and static GRPO, leading to provable variance reduction and greater optimization stability (Xiao et al., 3 Jun 2025).

In continuous control with deep ReLU nets, Adaptive Network Scaling (ANS) searches for an optimal reward scale $(\mu_t,\sigma_t)$3, transferring trained network weights by matching homogenous scaling across layers:

$(\mu_t,\sigma_t)$4

and using returns’ EMA to guide scale adaptation, thus preventing ReLU neuron death and maximizing agent performance relative to fixed normalization (Wu et al., 2018).

4. Empirical Evidence and Comparative Analysis

ARN methods have demonstrated empirical superiority and stability across a range of RL domains:

Domain/Task	Baseline/Comparison	ARN Variant	Outcome/Metric
Atari (Double-DQN, unclipped)	Reward clipping	Pop-Art (ARN)	Same/better final scores; cross-task LR, 2 orders of mag. smaller grad norm spread (Hasselt et al., 2016)
Binary regression with rare spikes	SGD, ART-only	ART+POP (ARN)	Lowest RMSE, stable on rare spikes (Hasselt et al., 2016)
GSM8K, MATH (GRPO), LLM RL	GRPO w/o normalization	ARN (Z-score)	Faster convergence, +3–7 pp accuracy gain in stable regime (Ge et al., 30 Jan 2026)
LLM agentic tasks (PPR)	Process+outcome, no norm	ReNorm (ARN)	+10–15 absolute points on QA, stable learning (Xu et al., 29 Sep 2025)
Mujoco continuous control	Fixed scale, Pop-Art	ANS (ARN)	ANS outperforms all baselines; mitigates dying ReLU (Wu et al., 2018)
Reasoning LMs, binary reward	REINFORCE, GRPO	BNPO (Beta norm)	0.7–1% increase in pass@1; variance reduction (Xiao et al., 3 Jun 2025)

Notably, Pop-Art normalization in actor–critic settings may degrade performance, whereas ANS—using scale search and weight adaptation—is specifically effective for ReLU activations (Wu et al., 2018). In settings with heavy-tailed, sparse, or hybrid rewards, bounded normalization (as in ReNorm (Xu et al., 29 Sep 2025) or BNPO (Xiao et al., 3 Jun 2025)) is critical to prevent reward hacking and runaway updates.

5. Practical Considerations, Stability, and Hyperparameters

Implementing ARN requires bookkeeping for normalization constants and (in “POP” or weight-scaling variants) closed-form parameter transformations to assure invariance. Core hyperparameters include:

• EMA factors (e.g., $(\mu_t,\sigma_t)$5 for mean/variance tracking): small for slow, stable updates; large for rapid adaptation in non-stationary tasks.
• Normalization scale: preset (e.g., unit variance) or domain-tuned.
• Adaptive step size: with ARN, a fixed learning rate suffices across domains.
• Initialization: network output and normalization constants must match at initialization to avoid discontinuities.
• Global scaling: in some contexts (e.g., process-outcome hybrid), an outer scaling-of-rewards knob helps further smooth updates.

The computational overhead is minimal—mostly a few additional scalar ops and parameter transformations per update (Pop-Art), or distribution moment tracking (BNPO). ARN can wrap any optimizer and is orthogonal to the underlying choice of SGD, Adam, RMSProp, or advantage estimator.

6. Limitations, Extensions, and Theoretical Insights

While ARN delivers scale invariance, faster convergence, and greater robustness:

• Extra hyperparameters (e.g., $(\mu_t,\sigma_t)$6, learning rate, reward scale) must be monitored, and in low-variance tasks normalization may attenuate the learning signal.
• Theoretical convergence: In log-linear policies, ARN-augmented GRPO provably accelerates convergence by a factor governed by the mean within-prompt reward variance (Ge et al., 30 Jan 2026). State-wise affine MDP normalization is advantage-preserving and—when used in reward balancing solvers—can enable rapid convergence even in high self-loop settings (Mustafin et al., 2024).
• Process/outcome blending: Joint normalization of dense step-wise process rewards and sparse outcome rewards