---
title: 'CIM-PPO: Correntropy-Induced Metric PPO'
url: https://www.emergentmind.com/topics/correntropy-induced-metric-cim-ppo
type: topic
---

# CIM-PPO: Correntropy-Induced Metric PPO

Correntropy-Induced Metric Proximal Policy Optimization (CIM-PPO) is a Proximal Policy Optimization (PPO) variant that replaces the standard Kullback-Leibler (KL) divergence penalty with a Liu–Correntropy Induced Metric (CIM), yielding a symmetric, bounded, and computationally efficient trust-region penalty. This approach rectifies the pathological behavior introduced by the asymmetric, unbounded nature of KL divergence in PPO-KL variants, leading to improved update efficiency and policy stability across continuous-control tasks [2110.10522].

## 1. Liu–Correntropy Induced Metric: Definition and Properties

The Liu–Correntropy Induced Metric is constructed from the correntropy between two vectors $x, y \in \mathbb{R}^n$ using a (normalized) Gaussian kernel $\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$. The correntropy is $V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$, and the induced metric is:
$$
\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}
$$
This metric is symmetric, positive-definite, bounded ($\mathrm{CIM}_\sigma(x, y) \leq 1$ when $\kappa_\sigma(0) = 1$), and satisfies the triangle inequality. The feature map $\Phi$ into a Hilbert space $H$ enables an inner-product view:
$$
\kappa_\sigma(x - y) = \langle \Phi(x), \Phi(y) \rangle_H
$$
such that
$$
\mathrm{CIM}_\sigma(x, y) = \| \Phi(x) - \Phi(y) \|_H
$$
Verifying that $\mathrm{CIM}_\sigma$ is a bona-fide metric establishes a geometric foundation for trust-region analysis [2110.10522].

## 2. KL Divergence as a Special Case and Asymmetry Issue

Local approximations relate the KL divergence penalty to the Fisher information:
$$
D_{\mathrm{KL}}(\pi_\text{old} \| \pi_\text{new}) \approx \frac{1}{2} \delta\theta^\top F(\theta) \delta\theta
$$
However, for general distributions, especially multivariate Gaussians with mismatched covariances, the KL divergence exhibits pronounced asymmetry, given by:
$$
D_{\mathrm{KL}}(\pi_1 \| \pi_2) - D_{\mathrm{KL}}(\pi_2 \| \pi_1) = \log\frac{|\Sigma_2|}{|\Sigma_1|} + \frac{1}{2} [ \mathrm{tr}(\Sigma_1^{-1}\Sigma_2 + \Sigma_2^{-1}(\mu_2 - \mu_1)(\mu_2 - \mu_1)^\top) - \mathrm{tr}(\Sigma_2^{-1}\Sigma_1 + \Sigma_1^{-1}(\mu_1 - \mu_2)(\mu_1 - \mu_2)^\top) ]
$$
The paper establishes that this asymmetry can be arbitrarily large in high-dimensional settings or with differing variances. Theorems 1 and 2 demonstrate that such asymmetry can misguide the surrogate gradient, causing updates in a direction that actually decreases the advantage, reducing policy improvement efficiency [2110.10522].

## 3. Extension to Reproducing Kernel Hilbert Space and Trust-Region Guarantee

CIM is extended to the case of distributions in Reproducing Kernel Hilbert Space (RKHS). For two policies $\pi_\text{old}(\cdot|s)$ and $\pi_\text{new}(\cdot|s)$, define:
$$
D_{\mathrm{CIM}}(\pi_\text{old}, \pi_\text{new}) = \mathbb{E}_{s \sim d_\pi} [ \mathrm{CIM}_\sigma(\pi_\text{old}(\cdot|s), \pi_\text{new}(\cdot|s)) ]
$$
with
$$
\mathrm{CIM}_\sigma(P, Q) = \sqrt{2\kappa(0) - 2 \mathbb{E}_{a} [\kappa(P(a), Q(a))]}
$$
This metric's triangle-inequality and boundedness permit a trust-region guarantee via a modified TRPO argument:
$$
\eta(\pi_\text{new}) \geq L_\pi(\pi_\text{new}) - C [ D_{\mathrm{CIM}}(\pi_\text{old}, \pi_\text{new}) ]^2
$$
where $C=O(\epsilon \gamma/(1-\gamma)^2)$, yielding
$$
D_{\mathrm{CIM}}(\pi_\text{old}, \pi_\text{new}) \leq \delta \implies \eta(\pi_\text{new}) \geq L_\pi(\pi_\text{new}) - C\delta^2
$$
This provides a rigorous trust-region bound leveraging the CIM [2110.10522].

## 4. CIM–PPO Surrogate Objective and Policy Update

The CIM–PPO algorithm replaces the KL divergence penalty in the PPO surrogate with the CIM penalty. The CIM–PPO surrogate is:
$$
L^{\mathrm{CIM}}(\pi) = \mathbb{E}_{s,a \sim \pi_\text{old}} \big[ r(\theta) A_\pi(s,a) \big] - \alpha \mathbb{E}_{s \sim d_\pi} \left[ \mathrm{CIM}_\sigma(\pi_\text{old}(\cdot|s), \pi(\cdot|s)) \right]
$$
where $r(\theta) = \pi_\theta(a|s)/\pi_\text{old}(a|s)$ is the probability ratio and $\alpha$ the penalty coefficient. The policy-gradient becomes:
$$
\nabla_\theta L^{\mathrm{CIM}} = \mathbb{E}_{s,a \sim \pi_\text{old}} \left[ \nabla_\theta \log\pi_\theta(a|s) A_\pi(s,a) \right] - \alpha \mathbb{E}_{s \sim d_\pi} [ \nabla_\theta \mathrm{CIM}_\sigma(\pi_\text{old}(\cdot|s), \pi_\theta(\cdot|s)) ]
$$
A clipping variant analogous to PPO-Clip can optionally be used for the advantage term. This objective maintains proximity via a direct metric on policy distributions and eliminates pathology from unbounded, asymmetric KL penalties [2110.10522].

## 5. Computational Complexity Analysis

A comparative assessment of per-sample computational requirements for Gaussian policy parameterizations:

| Method   | Core Operation                | Complexity (per sample) |
|----------|-------------------------------|-------------------------|
| PPO-KL   | $\log\det\Sigma$, $\Sigma^{-1}$ | $O(d^3)$ (full), $O(d^2)$ (diag) |
| PPO-KL   | $\nabla_\theta D_{\mathrm{KL}}$ | $O(d^2)$                |
| CIM–PPO  | Elementwise kernel ops        | $O(d)$                  |

CIM–PPO's kernel-based penalty calculation relies on elementwise operations and a small number of exponentials or powers, in contrast with the matrix inversions and determinants required by KL-based PPO variants. As a result, actor-update cost for CIM–PPO is linear in the action dimension, offering scalability advantages for high-dimensional control [2110.10522].

## 6. Algorithm Structure: CIM–PPO in Practice

The CIM–PPO workflow can be summarized as:

- Initialize policy $\pi_\theta$, value function $V_\varphi$, penalty $\alpha$, and kernel bandwidth $\sigma$.
- For each iteration:
  - Run $\pi_\theta$ for $T$ timesteps to collect trajectories $\mathcal{D}$.
  - Compute rewards-to-go $R_t$ and advantage estimates $\hat{A}_t$ using $V_\varphi$.
  - Update actor $\theta$ via stochastic gradient ascent on
    $$
    L^{\mathrm{CIM}}(\theta) = \frac{1}{|\mathcal{D}|} \sum_{(s,a)\in \mathcal{D}} [ r(\theta)\cdot \hat{A}(s,a) ] - \alpha \cdot \frac{1}{|\mathcal{D}|} \sum_{s\in \mathcal{D}} \mathrm{CIM}_\sigma(\pi_\text{old}(\cdot|s), \pi_\theta(\cdot|s))
    $$
    for $K$ epochs with minibatches, optionally using clipping.
  - Update critic $\varphi$ to minimize
    $$
    \frac{1}{|\mathcal{D}|} \sum_{t\in \mathcal{D}} [ V_\varphi(s_t) - R_t ]^2
    $$
The design mirrors canonical actor–critic frameworks, replacing the trust-region penalty with the CIM [2110.10522].

## 7. Empirical Results on Continuous-Control Benchmarks

Experiments on four OpenAI Gym continuous-control tasks (Pendulum-v0, LunarLanderContinuous-v2, BipedalWalker-v3, BipedalWalkerHardcore-v3) and six Mujoco tasks (Hopper, Walker2d, HalfCheetah, Ant, Humanoid, Swimmer) assess CIM–PPO versus PPO-KL and PPO-Clip under standardized hyperparameters: discount $\gamma=0.9$, actor LR $=1\times10^{-4}$, critic LR $=2\times10^{-4}$, minibatch size 32, and 10 update epochs per iteration. PPO-KL targets $d_\text{targ}=0.1$, PPO-Clip uses $\epsilon=0.2$, CIM–PPO sets $\alpha=1$, $\sigma=1$ (triangular kernel for low-dimension, biweight for higher).

Reported learning curves demonstrate:

- On Pendulum-v0, PPO-KL policy collapse after $\sim$1200 episodes, PPO-Clip plateaus, while CIM–PPO maintains improvement and higher reward.
- On LunarLanderContinuous, CIM–PPO achieves stable, high reward ($\sim +200$) throughout training, surpassing both baselines.
- BipedalWalker: CIM–PPO converges fastest to $\sim 300$ reward; PPO-Clip plateaus at $\sim 260$; PPO-KL lags at $\sim 200$.
- BipedalWalkerHardcore: CIM–PPO reaches $\sim 150$ reward, exceeding PPO-Clip ($\sim 120$) and PPO-KL ($\sim 80$ with high variance).
- Six Mujoco tasks: CIM–PPO matches or exceeds PPO-Clip by $5-15\%$ in final return and attains that return in $20-30\%$ fewer timesteps.

The experiments suggest that the boundedness and symmetry of CIM provide a more stable trust-region mechanism than asymmetric or heuristic penalties [2110.10522].

## 8. Theoretical Guarantee and Significance

CIM–PPO's use of a true metric on distribution space restores formal trust-region guarantees analogous to those in TRPO. The proof requires only that the kernel be symmetric and positive-definite; the advantage function must be uniformly bounded. The result:
$$
D_{\mathrm{CIM}} \leq \delta \implies \eta(\pi_\text{new}) \geq L_\pi(\pi_\text{new}) - C\delta^2
$$
furnishes a principled way to control policy deviation and performance degradation in each update, avoiding pitfalls associated with the asymmetric, unbounded KL penalty or the heuristic nature of clipping [2110.10522].

CIM–PPO constitutes a direct application of distributional metric theory to reinforcement learning, improving both theoretical rigor and empirical robustness in deep continuous control.

Source: https://www.emergentmind.com/topics/correntropy-induced-metric-cim-ppo