Smooth Advantage Standardization (SAS)

• Smooth Advantage Standardization (SAS) is a reward processing technique that normalizes rewards using smoothed step-wise and cumulative statistics to prevent zero-gradient updates.
• It leverages dynamic interpolation between recent and global reward statistics to reduce variance and stabilize policy updates in reinforcement learning.
• Empirical results show SAS boosts response utilization ratios by over 20% compared to traditional methods, validating its integration in the DCPO framework.

Smooth Advantage Standardization (SAS) is a reward processing technique introduced in the Dynamic Clipping Policy Optimization (DCPO) framework for reinforcement learning from verifiable rewards (RLVR) in LLMs. SAS systematically addresses limitations of single-step reward normalization, especially in the context of Generative Reward Policy Optimization (GRPO) and Dynamic Advantage Policy Optimization (DAPO), by reducing zero-gradient updates and batch variance, thus substantially improving response utilization and data efficiency (Yang et al., 2 Sep 2025).

1. Motivation and Failure Modes of Existing Standardization

Standard approaches such as those used in GRPO and DAPO standardize $G$ sample rewards for each RL step $i$ only against each other: $$\hat A^i_{j} = \frac{R^i_j - \mu^i_{\rm new}}{\sigma^i_{\rm new}}$$ where $\mu^i_{\rm new}$ and $\sigma^i_{\rm new}$ are the sample mean and standard deviation of the rewards $R^i_{1,\ldots,G}$ at RL step $i$.

Two predominant failure cases arise from this approach:

1. If all rewards are identical ($R^i_{1\ldots G}$ either all correct or all incorrect), the standard deviation vanishes ($\sigma^i_{\rm new} = 0$), leading to zero standardized advantage ($\hat A^i_j = 0$) for all samples, and thus no learning signal.
2. With high-entropy sampling, the standardized advantages can fluctuate drastically across steps, with large or even sign-reversed updates, destabilizing training.

SAS addresses these issues by maintaining running global statistics of rewards across steps, smoothing between recent and cumulative statistics, and selecting the minimal-magnitude advantage as the learning signal. This mitigates zero-gradient batches and reduces update variance.

2. Mathematical Framework

At RL step $i$, for each generated sample $j$ ($j = 1, ..., G$), let $R^i_j$ denote the reward. SAS proceeds as follows:

A. Step-wise statistics:

$\mu^i_{\rm new} = \frac{1}{G} \sum_{j=1}^G R^i_j$

$$\sigma^i_{\rm new} = \sqrt{\frac{1}{G} \sum_{j=1}^G (R^i_j - \mu^i_{\rm new})^2}$$

B. Cumulative (old) statistics (previous $i-1$ steps):

$$\mu^i_{\rm old} = \frac{1}{G(i-1)} \sum_{k=1}^{i-1} \sum_{j=1}^G R^k_j$$

$$\sigma^i_{\rm old} = \sqrt{\frac{1}{G(i-1)} \sum_{k=1}^{i-1} \sum_{j=1}^G (R^k_j - \mu^i_{\rm old})^2}$$

C. Aggregate (all-steps) statistics:

$$\mu^i_{\rm total} = \frac{1}{i}(\mu^i_{\rm new} + (i-1)\mu^i_{\rm old})$$

$$(\sigma^i_{\rm total})^2 = \frac{1}{i} \left( (\sigma^i_{\rm new})^2 + (i-1)(\sigma^i_{\rm old})^2 + \frac{i-1}{i}(\mu^i_{\rm old} - \mu^i_{\rm new})^2 \right)$$

$$\sigma^i_{\rm total} = \sqrt{(\sigma^i_{\rm total})^2}$$

D. Computation of standardized advantages:

• Step-only:

$$\hat A^i_{\rm new,j} = \frac{R^i_j - \mu^i_{\rm new}}{\sigma^i_{\rm new} + \varepsilon_{\rm st}}$$

• Cumulative:

$$\hat A^i_{\rm total,j} = \frac{R^i_j - \mu^i_{\rm total}}{\sigma^i_{\rm total} + \varepsilon_{\rm st}}$$

where $\varepsilon_{\rm st}$ is a small constant (e.g., $10^{-8}$).

E. Smoothed interpolation:

$$\hat{SA}^i_{\rm new,j} = \frac{i-1}{i}\hat A^i_{\rm new,j} + \frac{1}{i}\hat A^i_{\rm total,j}$$

$$\hat{SA}^i_{\rm total,j} = \frac{1}{i}\hat A^i_{\rm new,j} + \frac{i-1}{i}\hat A^i_{\rm total,j}$$

F. Signal selection for stability:

$$\hat A^i_j = \begin{cases} \hat{SA}^i_{\rm new,j} & \text{if } |\hat{SA}^i_{\rm new,j}| < |\hat{SA}^i_{\rm total,j}| \ \hat{SA}^i_{\rm total,j} & \text{otherwise} \end{cases}$$

This regime ensures no batch is fully zeroed out, dampens variance, and provides a consistent training signal.

3. Integration and Pseudocode

SAS is implemented as an augmenting module to the RLVR update cycle in DCPO. The complete iteration is:

initialize mu_old ← 0, σ_old ← 0 for i in 1…N_steps: # Generate G responses, collect rewards R[1..G] for j in 1..G: Rj ← reward_of_response(j) # Compute step-wise stats μ_new ← (1/G) * sum_j Rj σ_new ← sqrt((1/G)*sum_j (Rj−μ_new)^2) # Compute cumulative stats if i==1: μ_old ← 0; σ_old ← 0 μ_total ← (μ_new + (i-1)*μ_old)/i σ_total² ← (σ_new² + (i-1)*σ_old² + ((i-1)/i)*(μ_old−μ_new)²)/i σ_total ← sqrt(σ_total²) # Standardize & smooth for each response for j in 1..G: A_new_j ← (Rj − μ_new) / (σ_new + ε_st) A_total_j ← (Rj − μ_total) / (σ_total + ε_st) SA_new_j ← ((i-1)/i)*A_new_j + (1/i)*A_total_j SA_tot_j ← (1/i)*A_new_j + ((i-1)/i)*A_total_j # pick the smaller-abs signal if |SA_new_j| < |SA_tot_j|: A_hat_j ← SA_new_j else: A_hat_j ← SA_tot_j # Use A_hat_j for all tokens of response j in the clipped surrogate loss # (rest of DCPO loss / gradient descent) # Update cumulative stats μ_old ← μ_total σ_old ← σ_total

No extra learnable parameters or momentum variables are required, and the entire module is parameter-free aside from $\varepsilon_{\rm st}$, which is fixed.

4. Hyper-parameters and Practical Considerations

• Standardization jitter ($\varepsilon_{\rm st}$): Typically set to $1 \times 10^{-8}$, sufficient to avoid division-by-zero.
• Smoothing schedule ($\tfrac{i-1}{i}, \tfrac{1}{i}$): These weights gradually favor step-wise statistics early on and cumulative statistics as training progresses, requiring no manual tuning.
• Parameterless design: No additional learnable or momentum terms are introduced, ensuring a stable inductive bias throughout optimization.

5. Empirical Impact and Ablation Evidence

The empirical effect of incorporating SAS into DCPO is defined primarily in terms of the Response Utilization Ratio (RUR)—the proportion of samples with nonzero advantage values that contribute to policy gradients. Comparative benchmarks reveal:

Model	GRPO RUR	DCPO (with SAS) RUR
Qwen2.5-1.5B	45.6%	67.1%
Qwen2.5-3B	48.3%	74.3%
Qwen2.5-7B	37.4%	73.2%
Qwen2.5-14B	43.9%	72.4%
Average	43.8%	71.8%

Further, ablation on Avg@32 for Qwen2.5-7B shows the following pattern:

• Incorporating Only-Token-Mean (OTM) yields modest improvement over GRPO.
• Replacing step-only with smoothed standardization (SAS) yields further enhancement, closely matching DAPO.
• Adding Dynamic-Adaptive Clipping (DAC) yields the largest gain and stabilizes training.
• The full combination (OTM+SAS+DAC, i.e., DCPO) yields the highest and most robust performance.

Training curves and entropy plots further indicate that SAS substantially diminishes variance, prevents the zero-advantage collapse, and supports sustained learning across RL steps (Yang et al., 2 Sep 2025).

6. Role within DCPO and Broader Significance

SAS is a core element within the DCPO framework, directly mitigating two failure points of reward standardization in previous RLVR approaches: zero-gradient updates arising from degenerate batch rewards, and elevated variance from high-entropy sampling dynamics. By promoting stable, adaptive advantage estimation with negligible computational expense and no hyper-parameter overhead, SAS enables improved data efficiency, higher effective batch utilization, and enhanced final accuracy.

A plausible implication is that similar smoothing and signal selection strategies could have broad applicability for other RL scenarios that feature sparse or skewed reward distributions, especially where batchwise reward degeneracy impedes stable policy optimization. However, concrete evidence for such generalization is not provided within the cited work.

7. Summary

Smooth Advantage Standardization offers a mathematically robust, implementation-minimal approach to reward normalization in RLVR training for LLMs. It provides a principled solution to zeroing and instability in reward-driven policy updates and demonstrates substantial improvements in empirical Response Utilization and accuracy on benchmark tasks, with a concise and parameter-free algorithmic footprint (Yang et al., 2 Sep 2025).