---
title: 'POP3D: Penalized Point Probability Distance'
url: https://www.emergentmind.com/topics/penalized-point-probability-distance-pop3d
type: topic
---

# POP3D: Penalized Point Probability Distance

Policy Optimization with Penalized Point Probability Distance (POP3D) is a first-order policy-gradient method for reinforcement learning that proposes an alternative regularization scheme to Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). POP3D introduces the point probability distance as a symmetric penalty and establishes it as a lower bound on the squared total variation divergence, thereby providing a theoretically grounded, empirically competitive approach for stabilizing policy updates in both discrete and continuous action domains [1807.00442].

## 1. Point Probability Distance: Formal Definition and Properties

POP3D pivots on the point probability distance, denoted $D_{pp}$, as a regularization term in policy optimization. For discrete distributions $p$ and $q$ over $K$ actions, the total variation (TV) divergence is defined as:
$$
D_{TV}(p\|q) = \frac{1}{2}\sum_{i=1}^K |p_i - q_i|.
$$
TRPO leverages $\alpha = \max_s D_{TV}(\pi_{\theta_{\rm old}}(\cdot|s)\|\pi_\theta(\cdot|s))$ and controls policy improvement using the squared TV divergence.

POP3D introduces $D_{pp}$ as follows. Let $a$ be the action taken at state $s$ by the agent. Then,
$$
D_{pp}(\pi_{\theta_{\rm old}}(\cdot|s),\,\pi_\theta(\cdot|s)) = \bigl[\pi_{\theta_{\rm old}}(a|s) - \pi_\theta(a|s)\bigr]^2.
$$
This penalty is symmetric and bounded ($0 \le D_{pp} \le 1$ in the discrete case). Importantly, for any index $\ell$,
$$
D_{TV}^2(p\|q) \geq (p_\ell - q_\ell)^2 = D_{pp}(p\|q),
$$
so $D_{pp}$ is a lower bound for the squared TV divergence.

## 2. POP3D Objective: Formulation and Optimization

The POP3D objective modifies the policy-gradient surrogate by incorporating the $D_{pp}$ penalty. Using standard notation:
- $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\rm old}}(a_t|s_t)}$,
- $\hat{A}_t$ is the estimated advantage at time $t$.

The vanilla surrogate, in comparison with PPO's clipped surrogate, is replaced by the following loss (Equation 12 in the original work):
$$
\max_\theta \mathbb{E}_t \bigl[ r_t(\theta) \hat{A}_t - \beta D_{pp}(\pi_{\theta_{\rm old}}(\cdot|s_t), \pi_\theta(\cdot|s_t)) \bigr].
$$
Expressed as a minimization problem:
$$
L^{\rm POP3D}(\theta) = -\mathbb{E}_t[r_t(\theta) \hat{A}_t ] + \beta\mathbb{E}_t[\bigl(\pi_{\theta_{\rm old}}(a_t|s_t) - \pi_\theta(a_t|s_t)\bigr)^2].
$$
In practice, the total training loss adds the value function regression and entropy bonus:
$$
L(\theta) = L^{\rm POP3D}(\theta) + c_v\,\mathbb{E}_t[(V_\theta(s_t)-V_{\rm target}(s_t))^2] - c_e\,\mathbb{E}_t[\mathcal{H}(\pi_\theta(\cdot|s_t))],
$$
where $c_v$ and $c_e$ are the respective coefficients.

## 3. Algorithmic Implementation

POP3D follows a minic-batch gradient-based optimization with the following structure:
1. **Data Collection:** Parallel actors ($N$) collect rollouts of length $T$ using the current policy, recording transitions $(s_t, a_t, r_t)$, and computing GAE advantages ($\hat A_t$).
2. **Surrogate Optimization:** For $K$ epochs per update, random mini-batches of size $M$ are drawn from the collected samples. The gradient of the POP3D loss is computed and used to update $\theta$ via Adam or SGD.
3. **Old Policy Update:** After optimization, update $\theta_{\rm old} \leftarrow \theta$.

Advantage estimation is consistently performed using Generalized Advantage Estimation, GAE($\gamma, \lambda$).

## 4. Manifold Perspective of the Regularization Effect

Deep neural policy parameterizations are highly over-complete, so many parameter sets yield the same high-probability action selections for a state—a solution manifold. Full distribution-matching penalties such as KL-divergence (used in TRPO and fixed-KL PPO) force agreement across all actions, effectively shrinking the solution manifold.

POP3D, by restricting the penalty to the sampled action, leaves the other action probabilities unconstrained during optimization. This leads to:
- Expansion of the effective solution manifold,
- Reduced penalty noise from random mini-batches,
- More "optimistic" updates focused on the sampled action,
- Enhanced exploration capacity on unpenalized actions,
- Alignment with PPO's principle that only the sampled ratio affects the update.

A plausible implication is that POP3D provides a balance between stability and flexibility, permitting policy innovation along directions that do not harm sampled action probabilities.

## 5. Empirical Evaluation

The performance of POP3D was assessed on 49 Atari games (40M frames) and 7 Mujoco continuous control tasks (10M frames). All algorithms used OpenAI Gym wrappers and identical neural network architectures:

**Atari (discrete), final 100-episode wins (Score₁₀₀):**
- POP3D: 32 games
- PPO: 11
- BASELINE (fixed-KL): 5
- TRPO: 1

**Mujoco (continuous), final 100-episode wins:**
- POP3D: 5
- PPO: 2

Sample Atari results (mean score over last 100 episodes, 3 seeds):

| Game      | POP3D   | PPO     | BASELINE | TRPO    |
|-----------|---------|---------|----------|---------|
| Alien     | 1510.80 | 1431.17 | 1311.23  | 1110.40 |
| Assault   | 5400.13 | 4438.82 | 1846.75  | 1363.46 |
| Breakout  | 458.41  | 281.93  | 67.70    | 40.65   |

Overall, POP3D matches or surpasses PPO in final performance and stability (lower across-seed variance), with comparable competitiveness on continuous control tasks, though PPO may learn faster initially [1807.00442].

## 6. Practical Usage and Hyperparameters

POP3D adopts most default PPO hyperparameters, with a notable exception for the penalty coefficient $\beta$:
- **Atari:** $T=128$, actors=8, epochs=3, minibatch=256, Adam step size linearly annealed ($2.5 \times 10^{-4} \times \alpha$), $\gamma=0.99$, $\lambda=0.95$, $c_v=1$, $c_e=0.01$, $\beta=5.0$ (fixed), PPO $\epsilon=0.1\times\alpha$, BASELINE KL-penalty $10.0$.
- **Mujoco:** $T=2048$, epochs=10, minibatch=64, Adam $3 \times 10^{-4}$ constant, $\gamma=0.99$, $\lambda=0.95$, no learning-rate decay, $\beta=5.0$, PPO $\epsilon=0.2$.

Random seeds were set to $\{10,100,1000\}$ for Atari and $\{0,10,100\}$ for Mujoco. Full code is available at https://github.com/paperwithcode/pop3d.

## 7. Significance and Theoretical Implications

By substituting the KL-penalty (TRPO/fixed-KL PPO) or PPO's clipping with a point probability penalty, POP3D provides a symmetric, lower-bounded regularization on the most informative part of the action space for sampled data. This approach maintains first-order optimization efficiency and reduces the dimensionality of constrained policy updates, which may lead to more robust training and improved exploration. POP3D requires tuning only a single penalty parameter, and empirical evidence shows competitive or improved results compared to widely adopted baselines [1807.00442].

Source: https://www.emergentmind.com/topics/penalized-point-probability-distance-pop3d