Clipped Action Policy Gradient (CAPG)

• CAPG is a policy gradient estimator designed for reinforcement learning in bounded action spaces by integrating the effect of action clipping into gradient computations.
• It replaces stochastic gradients outside the legal range with deterministic gradients based on cumulative distribution functions, ensuring unbiasedness and strict variance reduction.
• CAPG enhances sample efficiency and stability in applications like robotics and high-dimensional control benchmarks, leading to faster and more robust learning.

Clipped Action Policy Gradient (CAPG) is a policy gradient estimator designed for reinforcement learning (RL) in continuous control settings where actions are restricted to bounded domains. Standard policy gradient methods typically ignore the mismatch between unbounded policy distributions (e.g., Gaussians) and the inherently bounded action spaces encountered in practical problems—especially in robotics and control systems—leading to unnecessary estimator variance. CAPG addresses this by incorporating knowledge of action clipping directly into the gradient estimation process, resulting in an unbiased estimator with strictly reduced variance, improved learning stability, and enhanced sample efficiency (Fujita et al., 2018, Eisenach et al., 2018).

1. Motivation and Background

Many RL frameworks model actions via unbounded distributions like the Gaussian, even though the actual environment only allows action values within finite bounds, implemented via clipping. This mismatch introduces uncontrolled noise in the policy gradient estimator, as samples from outside the allowed range are mechanically projected (clipped) to the interval, while the gradient estimator proceeds as if every action, no matter how extreme, is executed as sampled. This results in higher estimator variance and potentially degraded sample efficiency.

The CAPG method was introduced to mitigate this issue by leveraging the deterministic nature of action clipping and the constancy of the Q-value outside the allowed action interval (Fujita et al., 2018). CAPG can be viewed as a member of a broader class of marginal policy gradient estimators that operate on the effective (post-transformation) action distribution (Eisenach et al., 2018).

2. Mathematical Formulation and Estimator Construction

The standard policy gradient estimator for a parameterized policy $\pi_\theta$ is given by

$$\nabla_\theta \eta(\pi_\theta) = \mathbb{E}_{s \sim d^\pi, u \sim \pi_\theta} \left[ Q^{\pi_\theta}(s,u) \, \nabla_\theta \log \pi_\theta(u|s) \right].$$

When actions are clipped to $[u_l, u_h]$, the true Q-value is

$$Q^{\pi_\theta}(s,u) = \begin{cases} Q^{\pi_\theta}(s,u_l) & u \leq u_l \ Q^{\pi_\theta}(s,u) & u_l < u < u_h \ Q^{\pi_\theta}(s,u_h) & u \geq u_h \end{cases}$$

and the expectation can be assessed separately over three regions. For a policy $\pi_\theta$ with cumulative distribution function $\Pi_\theta(u|s)$, the gradient is decomposed as: $$\mathbb{E}_u\big[ Q(s,u) \nabla_\theta \log \pi_\theta(u|s) \big] = Q(s,u_l)\nabla_\theta \log \Pi_\theta(u_l|s) + \mathbb{E}_{u \in (u_l, u_h)}[ Q(s,u) \nabla_\theta \log \pi_\theta(u|s) ] + Q(s,u_h) \nabla_\theta \log (1-\Pi_\theta(u_h|s))$$ Thus, the gradient estimator in CAPG replaces the stochastic score function $\nabla_\theta \log \pi_\theta(u|s)$ outside the interval with deterministic CDF-based gradients: $$\overline{\psi}(s,u) = \begin{cases} \nabla_\theta \log \Pi_\theta(u_l|s) & u \leq u_l \ \nabla_\theta \log \pi_\theta(u|s) & u_l < u < u_h \ \nabla_\theta \log(1-\Pi_\theta(u_h|s)) & u \geq u_h \end{cases}$$ The CAPG estimator thus is: $\mathbb{E}_u [Q(s,u) \overline{\psi}(s,u)]$ This construction yields an unbiased estimator for the policy gradient (Fujita et al., 2018, Eisenach et al., 2018).

3. Variance Reduction and Theoretical Guarantees

Rigorous proofs in the primary literature establish both unbiasedness and strict variance reduction for the CAPG estimator relative to the conventional, unclipped estimator.

Unbiasedness follows from the “compatible density” assumption—that the policy density allows the exchange of derivative and expectation—and application of integration by parts to the partitioned regions of the action domain.

Variance reduction is guaranteed because, in regions outside $[u_l, u_h]$, the stochastic gradient $\nabla_\theta \log \pi_\theta(u|s)$ is replaced by the fixed deterministic value $\nabla_\theta \log \Pi_\theta(u_l|s)$ or $\nabla_\theta \log(1-\Pi_\theta(u_h|s))$, eliminating variance due to sampling in these "dead" regions. Formally, for each region,

$$\operatorname{Var}[Q(s,u) \overline{\psi}(s,u)] \leq \operatorname{Var}[Q(s,u) \nabla_\theta \log \pi_\theta(u|s)]$$

with the difference quantifiable as an expected scaled Fisher information in the clipped region (Eisenach et al., 2018).

The reduction in variance becomes particularly pronounced when the policy's mean or variance leads to a high likelihood of sampling out-of-bound actions, as is commonly the case in early training or in highly constrained tasks (Fujita et al., 2018).

4. Relationship to Related Policy Gradient Methods

CAPG is best understood in context with other policy gradient estimators:

• Standard (REINFORCE-type) estimator directly applies the likelihood-ratio trick, accumulating variance from samples both inside and outside the feasible region.
• Marginal Policy Gradient (MPG) estimators operate on the distribution induced by action transformations (e.g., clipping, normalization), of which CAPG is a special case for the clipping transformation (Eisenach et al., 2018).
• Rao–Blackwellization: The CAPG estimator can be seen as a Rao–Blackwellized variant, integrating out redundant action-space randomness that does not affect the returned control due to clipping.
• Comparison with All-Action estimators: While CAPG achieves variance reduction by collapsing the gradient in clipped regions, all-action estimators reduce variance by conditioning on the entire set of possible actions using numerical integration or Monte Carlo averaging (Petit et al., 2019).

The structural modification in CAPG enables its use with both on-policy and off-policy actor-critic algorithms, provided the Q-value (or advantage estimator) is appropriately chosen for the clipped action.

5. Practical Integration and Implementation

The CAPG estimator is simple to implement within the automatic differentiation frameworks commonly used in deep RL. For each sampled action $u$,

• If $u \leq u_l$, reconstruct the loss using the log CDF: $\log \Pi_\theta(u_l|s)$.
• If $u \in (u_l, u_h)$, apply the standard log-probability: $\log \pi_\theta(u|s)$.
• If $u \geq u_h$, use the log of the upper-tail mass: $\log (1 - \Pi_\theta(u_h|s))$.

This modification is made at the per-sample loss construction level, often as a drop-in replacement for the log-probability operation in policy loss computation. Public reference implementations are available (Fujita et al., 2018).

Table: Summary of Score Functions Used in CAPG

Region	Replacement in Loss	Score Function
$u \leq u_l$	$\log \Pi_\theta(u_l|s)$	$\nabla_\theta \log \Pi_\theta(u_l|s)$
$u \in (u_l,u_h)$	$\log \pi_\theta(u|s)$	$\nabla_\theta \log \pi_\theta(u|s)$
$u \geq u_h$	$\log (1-\Pi_\theta(u_h|s))$	$\nabla_\theta \log (1 - \Pi_\theta(u_h|s))$

CAPG integrates into existing policy gradient methods such as PPO and TRPO simply by substituting the gradient computation stage without needing to alter the rest of the algorithm (optimization, trust region, etc.) (Fujita et al., 2018).

6. Empirical Results and Practical Impact

Empirical validation is provided in both controlled continuum-armed bandit problems and high-dimensional continuous RL benchmarks (e.g., MuJoCo). Key findings include:

• Variance: CAPG exhibits strictly lower gradient estimation variance, measured across parameter settings (mean and variance of the policy), relative to standard estimators.
• Learning Speed: Faster convergence or higher asymptotic rewards, particularly in constrained action environments or when the policy frequently samples outside feasible bounds.
• Batch Size Sensitivity: CAPG provides more robust updates with small batch sizes, where variance reduction is most critical (Fujita et al., 2018, Eisenach et al., 2018).
• Stability across Algorithms: Benefits observed in both first-order (PPO) and second-order (TRPO) optimization procedures, without added complexity.

These experiments confirm the theoretical properties, showing consistent or stronger gains as policy parameters are initialized farther from the legal action domain or as action dimensionality increases.

7. Extensions, Generalizations, and Limitations

The marginal policy gradient formulation (Eisenach et al., 2018) generalizes CAPG to actions transformed by arbitrary mappings (e.g., normalization or projection). CAPG’s framework applies as long as the action-value is constant on irrelevant regions (i.e., those mapped to the same output under the transformation $T$). This perspective enables application to a broader class of bounded action problems.

However, CAPG’s construction presumes that the action-value for a clipped action equals the Q-value at the bound (i.e., $Q(s, u_l)$ or $Q(s, u_h)$), which may be violated in non-Markovian or atypical environment setups. Additionally, CAPG assumes that the policy distribution is differentiable and that the CDF and its gradient are tractable, which is the case for standard parametric families like Gaussians.

Summary

Clipped Action Policy Gradient (CAPG) is a variance-reduced, unbiased policy gradient estimator tailored to continuous control tasks with bounded action domains. By replacing stochastic score components outside the feasible range with deterministic gradients of the cumulative tail probabilities, CAPG eliminates variance arising from action samples that are deterministically clipped, yielding more stable and sample-efficient learning. CAPG is analytically grounded, easy to integrate into deep RL algorithms, robust to policy parameterization choices, and validated both theoretically and empirically as an improvement over conventional policy gradient estimators in bounded action settings (Fujita et al., 2018, Eisenach et al., 2018).