POP3D: Penalized Point Probability Distance

• The paper introduces POP3D, a policy-gradient method that uses a symmetric point probability penalty to stabilize policy updates in reinforcement learning.
• It replaces traditional KL-divergence and clipping approaches with a loss based on the squared difference of action probabilities, ensuring a lower-bound on the total variation divergence.
• Empirical tests on Atari and Mujoco benchmarks show competitive performance, reduced variance, and improved exploration compared to PPO and TRPO.

Policy Optimization with Penalized Point Probability Distance (POP3D) is a first-order policy-gradient method for reinforcement learning that proposes an alternative regularization scheme to Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). POP3D introduces the point probability distance as a symmetric penalty and establishes it as a lower bound on the squared total variation divergence, thereby providing a theoretically grounded, empirically competitive approach for stabilizing policy updates in both discrete and continuous action domains (Chu, 2018).

1. Point Probability Distance: Formal Definition and Properties

POP3D pivots on the point probability distance, denoted $D_{pp}$, as a regularization term in policy optimization. For discrete distributions $p$ and $q$ over $K$ actions, the total variation (TV) divergence is defined as:

$$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$

TRPO leverages $$\alpha = \max_s D_{TV}(\pi_{\theta_{\rm old}}(\cdot|s)\|\pi_\theta(\cdot|s))$$ and controls policy improvement using the squared TV divergence.

POP3D introduces $D_{pp}$ as follows. Let $a$ be the action taken at state $s$ by the agent. Then,

$$D_{pp}(\pi_{\theta_{\rm old}}(\cdot|s),\,\pi_\theta(\cdot|s)) = \bigl[\pi_{\theta_{\rm old}}(a|s) - \pi_\theta(a|s)\bigr]^2.$$

This penalty is symmetric and bounded ($p$0 in the discrete case). Importantly, for any index $p$1,

$p$2

so $p$3 is a lower bound for the squared TV divergence.

2. POP3D Objective: Formulation and Optimization

The POP3D objective modifies the policy-gradient surrogate by incorporating the $p$4 penalty. Using standard notation:

• $p$5,
• $p$6 is the estimated advantage at time $p$7.

The vanilla surrogate, in comparison with PPO's clipped surrogate, is replaced by the following loss (Equation 12 in the original work):

$p$8

Expressed as a minimization problem:

$p$9

In practice, the total training loss adds the value function regression and entropy bonus:

$q$0

where $q$1 and $q$2 are the respective coefficients.

3. Algorithmic Implementation

POP3D follows a minic-batch gradient-based optimization with the following structure:

1. Data Collection: Parallel actors ($q$3) collect rollouts of length $q$4 using the current policy, recording transitions $q$5, and computing GAE advantages ($q$6).
2. Surrogate Optimization: For $q$7 epochs per update, random mini-batches of size $q$8 are drawn from the collected samples. The gradient of the POP3D loss is computed and used to update $q$9 via Adam or SGD.
3. Old Policy Update: After optimization, update $K$0.

Advantage estimation is consistently performed using Generalized Advantage Estimation, GAE($K$1).

4. Manifold Perspective of the Regularization Effect

Deep neural policy parameterizations are highly over-complete, so many parameter sets yield the same high-probability action selections for a state—a solution manifold. Full distribution-matching penalties such as KL-divergence (used in TRPO and fixed-KL PPO) force agreement across all actions, effectively shrinking the solution manifold.

POP3D, by restricting the penalty to the sampled action, leaves the other action probabilities unconstrained during optimization. This leads to:

• Expansion of the effective solution manifold,
• Reduced penalty noise from random mini-batches,
• More "optimistic" updates focused on the sampled action,
• Enhanced exploration capacity on unpenalized actions,
• Alignment with PPO's principle that only the sampled ratio affects the update.

A plausible implication is that POP3D provides a balance between stability and flexibility, permitting policy innovation along directions that do not harm sampled action probabilities.

5. Empirical Evaluation

The performance of POP3D was assessed on 49 Atari games (40M frames) and 7 Mujoco continuous control tasks (10M frames). All algorithms used OpenAI Gym wrappers and identical neural network architectures:

Atari (discrete), final 100-episode wins (Score₁₀₀):

• POP3D: 32 games
• PPO: 11
• BASELINE (fixed-KL): 5
• TRPO: 1

Mujoco (continuous), final 100-episode wins:

• POP3D: 5
• PPO: 2

Sample Atari results (mean score over last 100 episodes, 3 seeds):

Game	POP3D	PPO	BASELINE	TRPO
Alien	1510.80	1431.17	1311.23	1110.40
Assault	5400.13	4438.82	1846.75	1363.46
Breakout	458.41	281.93	67.70	40.65

Overall, POP3D matches or surpasses PPO in final performance and stability (lower across-seed variance), with comparable competitiveness on continuous control tasks, though PPO may learn faster initially (Chu, 2018).

6. Practical Usage and Hyperparameters

POP3D adopts most default PPO hyperparameters, with a notable exception for the penalty coefficient $K$2:

• Atari: $K$3, actors=8, epochs=3, minibatch=256, Adam step size linearly annealed ($K$4), $K$5, $K$6, $K$7, $K$8, $K$9 (fixed), PPO $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$0, BASELINE KL-penalty $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$1.
• Mujoco: $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$2, epochs=10, minibatch=64, Adam $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$3 constant, $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$4, $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$5, no learning-rate decay, $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$6, PPO $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$7.

Random seeds were set to $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$8 for Atari and $$D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.$$9 for Mujoco. Full code is available at https://github.com/paperwithcode/pop3d.

7. Significance and Theoretical Implications

By substituting the KL-penalty (TRPO/fixed-KL PPO) or PPO's clipping with a point probability penalty, POP3D provides a symmetric, lower-bounded regularization on the most informative part of the action space for sampled data. This approach maintains first-order optimization efficiency and reduces the dimensionality of constrained policy updates, which may lead to more robust training and improved exploration. POP3D requires tuning only a single penalty parameter, and empirical evidence shows competitive or improved results compared to widely adopted baselines (Chu, 2018).