DRM-based PPO Reformulations

• The paper introduces DRM-based PPO variants that replace the conventional trust-region penalty with alternative metrics, divergences, or distributionally robust objectives.
• Empirical results demonstrate improved stability, sample efficiency, and exploration in methods such as CIM-PPO, PPO-RPE, COPG, GePPO, and Ent-PPO.
• These approaches address challenges like asymmetry, unclear minimization targets, and off-policy compatibility, broadening the scope of PPO in reinforcement learning.

“DRM-based Proximal Policy Optimization” can be understood, in the literature surveyed here, as an Editor’s term for a family of PPO variants that replace or reinterpret PPO’s conventional trust-region control through explicit metrics, divergences, or distributionally robust objectives. The relevant works do not define a single canonical algorithm under that label. Instead, they include metric-based reformulations such as "CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metric" (Guo et al., 2021), divergence-based regularization such as "Proximal Policy Optimization with Relative Pearson Divergence" (Kobayashi, 2020), pessimistic clipped-policy-gradient variants connected to distributionally robust policy optimization (Markowitz et al., 2023), off-policy generalizations with generalized clipping and sample reuse (Queeney et al., 2021), optimistic distributionally robust trust-region optimization without policy parameterization (Song et al., 2020), and entropy-regularized PPO adaptations for amortized discrete sampling in GFlowNets (Zykova-Myzina et al., 14 Jun 2026). Taken together, these works address a shared problem: how to preserve PPO’s empirical stability while correcting asymmetry, clarifying the minimization target, improving sample efficiency, or enlarging the admissible policy-update geometry.

1. Baseline PPO and the motivations for DRM-oriented reformulation

According to the penalty mechanism in a surrogate, PPO can be classified into PPO with KL divergence (PPO-KL) and PPO with Clip (PPO-Clip). In the standard account, Trust Region Policy Optimization seeks to produce monotonic improvement through bounded changes in policy outputs, whereas PPO instead uses loss clipping to take multiple safe optimization steps per batch of data, replacing the bound on the single step of TRPO with regularization on multiple steps (Markowitz et al., 2023). This design yields a practical first-order method, but several of the cited papers isolate specific weaknesses in its geometry and objective.

One line of criticism concerns asymmetry. "CIM-PPO" analyzes the impact of asymmetry in KL divergence on PPO-KL and highlights that when this asymmetry is pronounced, it will misguide the improvement of the surrogate (Guo et al., 2021). A second line concerns objective clarity. "PPO-RPE" states that PPO clips density ratio of the latest and baseline policies with a threshold, while its minimization target is unclear, and further argues that the symmetric threshold is given numerically while the density ratio itself is in asymmetric domain, thereby causing unbalanced regularization of the policy (Kobayashi, 2020).

A third line concerns data efficiency. "Generalized Proximal Policy Optimization with Sample Reuse" contrasts the reliable policy improvement of on-policy methods with the sample reuse of off-policy methods, then develops policy improvement guarantees that are suitable for the off-policy setting and connects these bounds to the clipping mechanism used in PPO (Queeney et al., 2021). A fourth line concerns domain transfer. "Proximal Policy Optimization for Amortized Discrete Sampling" derives PPO equivalents for training stochastic policies to sample structured discrete probability distributions under the GFlowNet framework, and shows that the PPO adaptation must incorporate entropy-regularized RL structure rather than naively importing a standard PPO loss (Zykova-Myzina et al., 14 Jun 2026).

These motivations suggest that DRM-oriented PPO research is less about abandoning PPO than about replacing its default local geometry with alternatives that are symmetric, bounded, explicitly regularized, off-policy compatible, or distributionally robust.

2. Metric and divergence replacements: CIM-PPO and PPO-RPE

"CIM-PPO" starts from correntropy. For two arbitrary scalar random variables $x$ and $y$, correntropy is defined as

$V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$

where $\kappa_\sigma$ is a kernel function with bandwidth $\sigma$. Liu et al. define the Correntropy Induced Metric as

$$\mathrm{CIM}(x,y)=\left(V_\sigma(0)-V_\sigma(x,y)\right)^{1/2}.$$

The paper states that PPO-KL can be represented in inner product form, demonstrates that KL divergence is a Correntropy Induced Metric in Euclidean space, extends PPO-KL to the Reproducing Kernel Hilbert Space, redefines the inner products with RKHS, and proposes PPO-CIM (Guo et al., 2021). The corresponding surrogate replaces the KL penalty with

$$\mathcal{L}_\pi^{\mathrm{CIM}}= \mathbb{E}_{(s,a)\sim \pi}\left[ \frac{\pi_{\mathrm{new}}(a|s)}{\pi(a|s)}\hat A_\pi(s,a) -\alpha\,\mathrm{CIM}_\sigma(\pi(\cdot|s),\pi_{\mathrm{new}}(\cdot|s)) \right].$$

The stated properties of CIM are central to the reformulation. For the Gaussian kernel, $0 \le \mathrm{CIM}_\sigma(x,y) < 1$; CIM is symmetric; it satisfies the triangle inequality; and it is described as robust to outliers and non-Gaussian noise. The paper further states that PPO-CIM has a lower computation cost in policy gradient, that the adaptive adjustment mechanism was canceled in CIM-PPO due to enough robustness of CIM, and that PPO-CIM can guarantee the new policy is within the trust region while the kernel satisfies some conditions (Guo et al., 2021). Empirically, the reported conclusion is that PPO-CIM can perform better than both PPO-KL and PPO-Clip in most tasks.

"PPO-RPE" addresses a different defect: the lack of a clear minimization target in clipping-based PPO. It regularizes policy updates by the Relative Pearson divergence. With density ratio

$\rho(a)=\frac{\pi(a|s)}{\pi_b(a|s)},$

mixture policy

$\pi_\beta(a|s)=\beta\pi(a|s)+(1-\beta)\pi_b(a|s),$

and relative ratio

$y$0

the Relative Pearson divergence is defined as

$y$1

The corresponding regularizer is

$y$2

leading to

$y$3

The key stated claim is that this regularization yields the clear minimization target, which constrains the latest policy to the baseline one (Kobayashi, 2020). The paper also derives a symmetric threshold in $y$4 that becomes asymmetric in $y$5, thereby aligning the threshold with the asymmetric domain of the ordinary density ratio. In that sense, PPO-RPE does not merely replace one penalty with another; it redefines the ratio space in which “proximity” is measured.

3. Distributional robustness, pessimism, and non-parametric trust regions

"Clipped-Objective Policy Gradients for Pessimistic Policy Optimization" modifies PPO through a directly clipped policy-gradient objective rather than the clipped importance-sampling surrogate. PPO’s objective is

$y$6

where $y$7. COPG replaces this with

$y$8

The paper states that both PPO and COPG produce biased gradient estimates with respect to the RL objective, but both display significantly reduced variance compared to the unbiased off-policy policy gradient. It further shows that the COPG objective is on average “pessimistic” compared to the PPO objective, that this pessimism promotes enhanced exploration, and that COPG empirically improves learning in single-task, constrained, and multi-task settings without adding significant computational cost or complexity (Markowitz et al., 2023). The supplied technical summary also states that the paper draws analogies between COPG’s pessimism and approaches in distributionally robust policy optimization.

"Optimistic Distributionally Robust Policy Optimization" moves more radically away from parametric PPO-style updates. It formulates policy improvement as

$y$9

where $V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$0 is a statistical distance such as KL divergence or the Wasserstein metric. The paper states that ODRPO solves the trust region constrained optimization problem without parameterizing the policies, improves TRPO and PPO with a higher sample efficiency and a better performance of the final policy while attaining the learning stability, and achieves a globally optimal policy update that is not promised in the prevailing policy based RL algorithms (Song et al., 2020). In the KL case, the optimal policy has the closed form

$V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$1

A common misconception is that robust PPO variants only adjust the penalty coefficient or clip threshold. These papers indicate a broader design space. COPG changes the clipped objective while retaining first-order simplicity; ODRPO changes the feasible set itself by directly optimizing over all admissible policies in a divergence ball. This suggests that “robustness” in PPO research can refer either to a pessimistic surrogate or to a non-parametric trust-region optimization problem.

4. Off-policy sample reuse and generalized clipping

"Generalized Proximal Policy Optimization with Sample Reuse" extends PPO to off-policy data by deriving policy improvement guarantees for samples collected by a mixture of recent policies. The central issue is distribution shift: when data are collected under behavior policies different from the update policy, standard on-policy guarantees no longer apply. GePPO addresses this by introducing a generalized clipping mechanism in which the center of the clip depends on the data-collecting policy:

$V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$2

For data from policy $V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$3, this centers the clip range at $V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$4, so that at the start of the update all samples are at the center of the clip, as in standard PPO (Queeney et al., 2021).

The generalized surrogate objective averages over a convex combination of recent policies and reduces to PPO when $V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$5 or all samples are on-policy. The paper states that GePPO combines the theoretically supported stability benefits of on-policy algorithms with the sample efficiency of off-policy algorithms, and that adaptive learning-rate control is used to stabilize training and control policy changes as dictated by the theoretical bounds (Queeney et al., 2021). The reported empirical comparison is unusually explicit: across a suite of MuJoCo tasks, GePPO consistently matches or exceeds PPO in final performance and learning curve efficiency, and GePPO needs 15–77% fewer samples than PPO to reach PPO’s final performance.

This line is important for DRM-based interpretations of PPO because it preserves PPO’s clipping intuition while redefining what proximity means under sample reuse. The trust region is no longer centered only on the current policy; it becomes a trust region relative to each data policy in the replay mixture.

5. Entropy-regularized PPO in amortized discrete sampling

"Proximal Policy Optimization for Amortized Discrete Sampling" adapts PPO to GFlowNets by using the theoretical equivalence between GFlowNets and entropy-regularized reinforcement learning. The key identification is that, if the intermediate per-step rewards are chosen as

$V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$6

then entropy-regularized RL with discount $V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$7, entropy coefficient $V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$8, and this reward structure recovers the amortized target distribution $V_\sigma(x,y)=\mathbb{E}[\kappa_\sigma(x-y)],$9 over terminal states (Zykova-Myzina et al., 14 Jun 2026). The resulting Ent-PPO is not described as a heuristic transplant of PPO into structured discrete sampling. It is derived from soft policy improvement, and the KL term arises analytically rather than as an auxiliary entropy bonus.

The per-step Ent-PPO loss is summarized as

$\kappa_\sigma$0

where $\kappa_\sigma$1 is the entropy-regularized advantage. The paper emphasizes that correct per-step rewards and explicit entropy regularization are mandatory for matching the amortized sampling task, and reports that removing the KL penalty or the PPO-style clipping causes instability, slow convergence, or outright failure (Zykova-Myzina et al., 14 Jun 2026).

The empirical scope is also broader than classical continuous-control PPO. Benchmarks include Hypergrid, TFBind8, String QM9, sEH, and QM9, with evaluation by TV distance, ELBO, and EUBO. The stated findings are that variance-reduction devices such as reward-to-go, learned value baselines, and GAE substantially improve data efficiency and convergence, and that Ent-PPO outperforms both standard GFlowNet objectives and naive PPO-like baselines, achieving improved or equal final TV, ELBO, and EUBO in fewer environment interactions (Zykova-Myzina et al., 14 Jun 2026).

This branch shows that DRM-like reformulation of PPO is not limited to continuous-control robustness. It can also mean deriving a PPO surrogate whose KL geometry is dictated by the target sampling semantics of an entropy-regularized discrete-generation problem.

6. Empirical synthesis, scope boundaries, and adjacent PPO lines

Across the surveyed methods, the reported empirical pattern is consistent: alternative metrics, divergences, or robust trust-region constructions are introduced not as abstract replacements, but as mechanisms for more stable optimization, better sample efficiency, or more favorable exploration behavior. The following summary compresses the stated empirical characterizations of the principal variants (Guo et al., 2021, Kobayashi, 2020, Markowitz et al., 2023, Queeney et al., 2021, Song et al., 2020, Zykova-Myzina et al., 14 Jun 2026).

Method	Core modification	Stated empirical characterization
CIM-PPO	Replaces KL penalty with Correntropy Induced Metric in RKHS	Better than both PPO-KL and PPO-Clip in most tasks
PPO-RPE	Regularizes by Relative Pearson divergence	Performed as well as or better than the conventional methods
COPG	Replaces clipped importance-sampling surrogate with clipped policy-gradient objective	Improved learning compared to PPO; comparable or superior performance to TRPO
GePPO	Uses off-policy sample reuse with generalized clipping	15–77% fewer samples than PPO to reach PPO’s final performance
ODRPO	Solves trust-region update by optimistic distributionally robust optimization without parameterizing policies	Higher sample efficiency and a better performance of the final policy while attaining learning stability
Ent-PPO	Uses entropy-regularized PPO for GFlowNets with analytic KL and correct per-step rewards	Improved convergence speed and data efficiency compared to standard GFlowNet training objectives

At the same time, not every PPO extension belongs naturally to the DRM-oriented family. "Distributed Proximal Policy Optimization for Contention-Based Spectrum Access" develops a distributed PPO implementation modeled on a two stage MDP, with local actors, centralized training for critics, LSTM layers for partial observability, and a proportional-fairness objective; it reports significantly higher proportional fairness reward than even a genie-aided adaptive energy detection threshold, together with improved sum and maximum user throughputs (Doshi et al., 2021). This suggests a useful scope boundary: some PPO extensions are application-specific implementations of the standard clipped framework, whereas the DRM-based line changes the geometry of proximity itself through metrics, divergences, generalized clipping, or distributionally robust optimization.

A further misconception is that all such variants optimize the same notion of trust region. The cited works indicate otherwise. CIM-PPO uses a symmetric metric with boundedness and triangle inequality; PPO-RPE regularizes a relative density ratio and derives an asymmetric threshold in the original ratio space; GePPO centers a separate clip interval for each behavior policy; Ent-PPO uses an analytic KL induced by soft policy improvement; ODRPO defines the trust region directly as a divergence- or Wasserstein-bounded ambiguity set and computes the globally optimal update within that set. The family resemblance lies in the proximal principle, but the underlying geometry varies substantially.

In that sense, DRM-based PPO is best regarded not as one algorithmic endpoint but as a research program. Its central thesis is that PPO’s practical success can be retained while replacing its default regularization with structures that are more symmetric, more explicit, more off-policy compatible, more entropy-consistent, or more distributionally robust.