CIM-PPO: Correntropy-Induced Metric PPO

• CIM-PPO is a PPO variant that replaces the KL divergence with a symmetric, bounded Liu–Correntropy induced metric, ensuring robust trust-region updates.
• It leverages a kernel-based strategy with linear computational complexity, avoiding costly matrix operations and enhancing scalability.
• Empirical results demonstrate CIM-PPO’s superior stability and faster convergence over traditional PPO methods on various continuous control benchmarks.

Correntropy-Induced Metric Proximal Policy Optimization (CIM-PPO) is a Proximal Policy Optimization (PPO) variant that replaces the standard Kullback-Leibler (KL) divergence penalty with a Liu–Correntropy Induced Metric (CIM), yielding a symmetric, bounded, and computationally efficient trust-region penalty. This approach rectifies the pathological behavior introduced by the asymmetric, unbounded nature of KL divergence in PPO-KL variants, leading to improved update efficiency and policy stability across continuous-control tasks (Guo et al., 2021).

1. Liu–Correntropy Induced Metric: Definition and Properties

The Liu–Correntropy Induced Metric is constructed from the correntropy between two vectors $x, y \in \mathbb{R}^n$ using a (normalized) Gaussian kernel $\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$. The correntropy is $$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$, and the induced metric is:

$$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$

This metric is symmetric, positive-definite, bounded ($\mathrm{CIM}_\sigma(x, y) \leq 1$ when $\kappa_\sigma(0) = 1$), and satisfies the triangle inequality. The feature map $\Phi$ into a Hilbert space $H$ enables an inner-product view:

$$\kappa_\sigma(x - y) = \langle \Phi(x), \Phi(y) \rangle_H$$

such that

$$\mathrm{CIM}_\sigma(x, y) = \| \Phi(x) - \Phi(y) \|_H$$

Verifying that $\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$0 is a bona-fide metric establishes a geometric foundation for trust-region analysis (Guo et al., 2021).

2. KL Divergence as a Special Case and Asymmetry Issue

Local approximations relate the KL divergence penalty to the Fisher information:

$\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$1

However, for general distributions, especially multivariate Gaussians with mismatched covariances, the KL divergence exhibits pronounced asymmetry, given by:

$\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$2

The paper establishes that this asymmetry can be arbitrarily large in high-dimensional settings or with differing variances. Theorems 1 and 2 demonstrate that such asymmetry can misguide the surrogate gradient, causing updates in a direction that actually decreases the advantage, reducing policy improvement efficiency (Guo et al., 2021).

3. Extension to Reproducing Kernel Hilbert Space and Trust-Region Guarantee

CIM is extended to the case of distributions in Reproducing Kernel Hilbert Space (RKHS). For two policies $\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$3 and $\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$4, define:

$\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$5

with

$\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$6

This metric's triangle-inequality and boundedness permit a trust-region guarantee via a modified TRPO argument:

$\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$7

where $\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$8, yielding

$\kappa_\sigma(z) = \exp(-\|z\|^2/(2\sigma^2))$9

This provides a rigorous trust-region bound leveraging the CIM (Guo et al., 2021).

4. CIM–PPO Surrogate Objective and Policy Update

The CIM–PPO algorithm replaces the KL divergence penalty in the PPO surrogate with the CIM penalty. The CIM–PPO surrogate is:

$$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$0

where $$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$1 is the probability ratio and $$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$2 the penalty coefficient. The policy-gradient becomes:

$$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$3

A clipping variant analogous to PPO-Clip can optionally be used for the advantage term. This objective maintains proximity via a direct metric on policy distributions and eliminates pathology from unbounded, asymmetric KL penalties (Guo et al., 2021).

5. Computational Complexity Analysis

A comparative assessment of per-sample computational requirements for Gaussian policy parameterizations:

Method	Core Operation	Complexity (per sample)
PPO-KL	$$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$4, $$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$5	$$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$6 (full), $$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$7 (diag)
PPO-KL	$$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$8	$$V_\sigma(x, y) = \mathbb{E}[ \kappa_\sigma(x - y) ]$$9
CIM–PPO	Elementwise kernel ops	$$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$0

CIM–PPO's kernel-based penalty calculation relies on elementwise operations and a small number of exponentials or powers, in contrast with the matrix inversions and determinants required by KL-based PPO variants. As a result, actor-update cost for CIM–PPO is linear in the action dimension, offering scalability advantages for high-dimensional control (Guo et al., 2021).

6. Algorithm Structure: CIM–PPO in Practice

The CIM–PPO workflow can be summarized as:

• Initialize policy $$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$1, value function $$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$2, penalty $$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$3, and kernel bandwidth $$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$4.
• For each iteration:
  • Run $$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$5 for $$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$6 timesteps to collect trajectories $$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$7.
  • Compute rewards-to-go $$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$8 and advantage estimates $$\mathrm{CIM}_\sigma(x, y) = \sqrt{V_\sigma(0) - V_\sigma(x, y)} = \sqrt{1 - \exp(-\|x - y\|^2/(2\sigma^2))}$$9 using $\mathrm{CIM}_\sigma(x, y) \leq 1$0.
  • Update actor $\mathrm{CIM}_\sigma(x, y) \leq 1$1 via stochastic gradient ascent on

  $\mathrm{CIM}_\sigma(x, y) \leq 1$2

  for $\mathrm{CIM}_\sigma(x, y) \leq 1$3 epochs with minibatches, optionally using clipping. - Update critic $\mathrm{CIM}_\sigma(x, y) \leq 1$4 to minimize

  $\mathrm{CIM}_\sigma(x, y) \leq 1$5

The design mirrors canonical actor–critic frameworks, replacing the trust-region penalty with the CIM (Guo et al., 2021).

7. Empirical Results on Continuous-Control Benchmarks

Experiments on four OpenAI Gym continuous-control tasks (Pendulum-v0, LunarLanderContinuous-v2, BipedalWalker-v3, BipedalWalkerHardcore-v3) and six Mujoco tasks (Hopper, Walker2d, HalfCheetah, Ant, Humanoid, Swimmer) assess CIM–PPO versus PPO-KL and PPO-Clip under standardized hyperparameters: discount $\mathrm{CIM}_\sigma(x, y) \leq 1$6, actor LR $\mathrm{CIM}_\sigma(x, y) \leq 1$7, critic LR $\mathrm{CIM}_\sigma(x, y) \leq 1$8, minibatch size 32, and 10 update epochs per iteration. PPO-KL targets $\mathrm{CIM}_\sigma(x, y) \leq 1$9, PPO-Clip uses $\kappa_\sigma(0) = 1$0, CIM–PPO sets $\kappa_\sigma(0) = 1$1, $\kappa_\sigma(0) = 1$2 (triangular kernel for low-dimension, biweight for higher).

Reported learning curves demonstrate:

• On Pendulum-v0, PPO-KL policy collapse after $\kappa_\sigma(0) = 1$31200 episodes, PPO-Clip plateaus, while CIM–PPO maintains improvement and higher reward.
• On LunarLanderContinuous, CIM–PPO achieves stable, high reward ($\kappa_\sigma(0) = 1$4) throughout training, surpassing both baselines.
• BipedalWalker: CIM–PPO converges fastest to $\kappa_\sigma(0) = 1$5 reward; PPO-Clip plateaus at $\kappa_\sigma(0) = 1$6; PPO-KL lags at $\kappa_\sigma(0) = 1$7.
• BipedalWalkerHardcore: CIM–PPO reaches $\kappa_\sigma(0) = 1$8 reward, exceeding PPO-Clip ($\kappa_\sigma(0) = 1$9) and PPO-KL ($\Phi$0 with high variance).
• Six Mujoco tasks: CIM–PPO matches or exceeds PPO-Clip by $\Phi$1 in final return and attains that return in $\Phi$2 fewer timesteps.

The experiments suggest that the boundedness and symmetry of CIM provide a more stable trust-region mechanism than asymmetric or heuristic penalties (Guo et al., 2021).

8. Theoretical Guarantee and Significance

CIM–PPO's use of a true metric on distribution space restores formal trust-region guarantees analogous to those in TRPO. The proof requires only that the kernel be symmetric and positive-definite; the advantage function must be uniformly bounded. The result:

$\Phi$3

furnishes a principled way to control policy deviation and performance degradation in each update, avoiding pitfalls associated with the asymmetric, unbounded KL penalty or the heuristic nature of clipping (Guo et al., 2021).

CIM–PPO constitutes a direct application of distributional metric theory to reinforcement learning, improving both theoretical rigor and empirical robustness in deep continuous control.