ReLU-Clipped Advantages & Spectral Normalization (VSOP)

• The paper demonstrates that applying ReLU-clipped advantage estimation in actor-critic methods maximizes a provable lower bound on expected returns while mitigating high-variance updates.
• It employs spectral normalization to enforce a global 1-Lipschitz constraint on value approximators, leading to improved policy robustness and generalization.
• The integration of dropout-based Bayesian regularization facilitates state-aware exploration via Thompson sampling, thereby enhancing sample efficiency on benchmark tasks.

ReLU-clipped Advantages and Spectral Normalization (VSOP) refers to a reinforcement learning framework that integrates ReLU-clipped policy advantages, spectral normalization of network weights, and Bayesian regularization via dropout within on-policy actor-critic methods. The approach, detailed in "ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages" (Jesson et al., 2023), is theoretically motivated to optimize for provable value improvements by maximizing a lower bound on the value function, while controlling the Lipschitz constant of value approximators. The algorithm also leverages spectral normalization, originally developed for stabilizing generative adversarial networks (Miyato et al., 2018), as a lightweight and differentiable tool for ensuring global smoothness of deep neural critics. This combination yields empirical gains in policy robustness, generalization, and sample efficiency across standard continuous control and exploration benchmarks.

1. ReLU-clipped Advantage Estimation

In on-policy actor-critic reinforcement learning, the advantage function under policy $\pi$ is defined as $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$. In practice, bootstrapped (e.g., GAE-based) estimators are used to compute $\hat{A}_t$, which VSOP then transforms by applying a rectified linear unit (ReLU) clipping:

$A^+(s,a) = \max(A^\pi(s,a), 0).$

Only non-negative advantages contribute to the policy gradient update, resulting in the actor update:

$$\nabla_\theta J^+ = \mathbb{E}_{(s,a)\sim\pi}[A^+(s,a)\nabla_\theta\log\pi_\theta(a|s)].$$

In stochastic mini-batch gradient ascent, this takes the concrete form:

$$\theta \gets \theta + \eta \frac{1}{B} \sum_{i=1}^B \max(0, \hat{A}_i)\nabla_\theta \log \pi_\theta(a_i|s_i).$$

The motivation is to reinforce actions demonstrably better than the mean while avoiding high-variance penalization of actions with insufficient evidence of suboptimality. Clipping negative advantages prevents updating the policy in unexplored or statistically ambiguous directions.

2. Theoretical Underpinnings: Value Lower Bound and Lipschitz Control

VSOP provides a formal justification for ReLU-clipped advantages by showing that ascent with $\nabla_\theta J^+$ maximizes a lower bound on the expected return, with the lower bound gap explicitly tied to the Lipschitz constant $K$ of the value function:

$$V^{\pi'}(s) \geq V^\pi(s) + \mathbb{E}_{(s,a)\sim d^\pi}[A^+(s,a)] - K\cdot C(s).$$

Here, $C(s)$ is a term controlled by the Lipschitz smoothness of $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$0. The analysis employs a surrogate value function $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$1, accumulated only from positive advantages. The additive residual $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$2, which can be harmful if large, is bounded by the Lipschitz constant of $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$3. This directly motivates strategies that rigorously constrain the Lipschitz constant of value approximators to shrink the lower bound gap.

3. Spectral Normalization in Deep Networks

Spectral normalization (SN), introduced originally for GAN discriminators (Miyato et al., 2018), constrains the weight matrix $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$4 of each linear or convolutional layer so that its spectral norm $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$5 (largest singular value) does not exceed 1. In VSOP, SN is enforced using the power iteration method, which estimates $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$6 efficiently:

• Maintain estimate $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$7.
• Iterate:

  $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$8

• Approximate $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$9.
• Update $\hat{A}_t$0.

This guarantees (up to approximation error) that $\hat{A}_t$1 and thus, for layered networks with 1-Lipschitz activation functions (e.g., ReLU), the global Lipschitz constant is directly controlled. For convolutional layers, weights are flattened to matrices prior to SN.

Constraining $\hat{A}_t$2 to unity in all critic layers simultaneously reduces the constant $\hat{A}_t$3 in the value function lower bound, thereby tightening theoretical guarantees and, empirically, reducing overfitting and spurious oscillations in value estimates.

4. Integration with Dropout-Based Bayesian Regularization

VSOP incorporates dropout for both critic and actor networks at training and inference, interpreting each random dropout mask as a Monte-Carlo sample from an implicit variational posterior $\hat{A}_t$4. This naturally provides an approximate evidence lower bound (ELBO) maximization framework for Bayesian inference:

$\hat{A}_t$5

At action-selection time, random dropout instantiates model uncertainty—yielding "state-aware" exploration via Thompson sampling. Each environment step executes a new dropout mask $\hat{A}_t$6, drawing $\hat{A}_t$7, meaning that exploration concentrates around the learned policy's mode but is adaptively diffuse where uncertainty is high.

5. Empirical Performance and Ablation

On MuJoCo continuous control and ProcGen generalization benchmarks, VSOP demonstrates:

• Median normalized return improvements of $\hat{A}_t$8 over A3C, $\hat{A}_t$9 over PPO, and $A^+(s,a) = \max(A^\pi(s,a), 0).$0–$A^+(s,a) = \max(A^\pi(s,a), 0).$1 over SAC/TD3.
• Interquartile mean consistently $A^+(s,a) = \max(A^\pi(s,a), 0).$2–$A^+(s,a) = \max(A^\pi(s,a), 0).$3 higher than PPO/A3C across tasks.
• In ProcGen, VSOP surpasses PPO on all normalized metrics, with typical IQM gains of $A^+(s,a) = \max(A^\pi(s,a), 0).$4–$A^+(s,a) = \max(A^\pi(s,a), 0).$5.

Ablation studies confirm that the largest marginal performance gains derive from the use of ReLU-clipped advantages and a single-action policy update, while spectral normalization and Thompson sampling provide further improvements in stability and exploration (Jesson et al., 2023).

6. Relation to Other Stabilization Methods

Spectral normalization differs from other Lipschitz-enforcement methods common in both RL and GANs:

• Weight clipping: Forces each element of $A^+(s,a) = \max(A^\pi(s,a), 0).$6 into $A^+(s,a) = \max(A^\pi(s,a), 0).$7, which biases all singular values, results in low-rank (collapsed) matrices, and underutilizes feature capacity.
• Frobenius/Row normalization: Normalizes the sum-of-squares or rows, constraining $A^+(s,a) = \max(A^\pi(s,a), 0).$8 and often leading to spectrum collapse.
• Orthonormal regularization: Adds a penalty for deviation from $A^+(s,a) = \max(A^\pi(s,a), 0).$9; expensive and abolishes spectral ordering (all singular values pushed to one).
• Gradient penalty (WGAN-GP): Regularizes the gradient norm $$\nabla_\theta J^+ = \mathbb{E}_{(s,a)\sim\pi}[A^+(s,a)\nabla_\theta\log\pi_\theta(a|s)].$$0 at interpolated points; local and computationally costly.
• Spectral normalization: Directly enforces $$\nabla_\theta J^+ = \mathbb{E}_{(s,a)\sim\pi}[A^+(s,a)\nabla_\theta\log\pi_\theta(a|s)].$$1; computationally efficient (1–2 matrix-vector products per update), preserves expressivity except in the top singular direction, and introduces only minor runtime overhead.

A summary comparison is provided below:

Method	Strategy	Drawbacks/Notes
Weight Clipping	Clamp W to $$\nabla_\theta J^+ = \mathbb{E}_{(s,a)\sim\pi}[A^+(s,a)\nabla_\theta\log\pi_\theta(a|s)].$$2	Collapses rank, biases all modes
Frobenius Norm	$$\nabla_\theta J^+ = \mathbb{E}_{(s,a)\sim\pi}[A^+(s,a)\nabla_\theta\log\pi_\theta(a|s)].$$3	Drives top singular value up, rank loss
Orthonormal Reg.	$$\nabla_\theta J^+ = \mathbb{E}_{(s,a)\sim\pi}[A^+(s,a)\nabla_\theta\log\pi_\theta(a|s)].$$4 penalty	Expensive, destroys spectrum
Gradient Penalty	Regularize $$\nabla_\theta J^+ = \mathbb{E}_{(s,a)\sim\pi}[A^+(s,a)\nabla_\theta\log\pi_\theta(a|s)].$$5	Doubles backward pass, local only
Spectral Normalization	$$\nabla_\theta J^+ = \mathbb{E}_{(s,a)\sim\pi}[A^+(s,a)\nabla_\theta\log\pi_\theta(a|s)].$$6	Preserves rank, efficient, global

7. Significance and Limitations

VSOP demonstrates that restricting policy updates to empirically justified, provably advantageous actions, combined with rigorous regularization of value networks via spectral norms, yields statistically robust and theoretically grounded reinforcement learning algorithms (Jesson et al., 2023). Spectral normalization provides a lightweight global Lipschitz constraint, improving generalization and training stability compared to prior weight clipping or gradient-penalty alternatives (Miyato et al., 2018).

Limitations include potential looseness of the global Lipschitz bound in deep networks (due to the product of layer norms), and the power-iteration estimator may drift if network parameters change rapidly—although using previously stored $$\nabla_\theta J^+ = \mathbb{E}_{(s,a)\sim\pi}[A^+(s,a)\nabla_\theta\log\pi_\theta(a|s)].$$7 typically suffices with a single iteration per batch. Spectral normalization alone does not enforce pointwise gradient constraints, which local gradient penalties provide. A plausible implication is that combined or hybrid approaches may be required in extremely deep or highly nonstationary settings.