Divergence Regularized Policy Optimization

• DRPO is a family of methods that optimize an objective combining expected return with an explicit divergence penalty against a reference policy or distribution.
• It formalizes regularization in MDPs using convex regularizers and Bregman divergences, with extensions to trust-region and mirror descent updates in RL and LLM post-training.
• DRPO’s design space allows for various divergence choices (e.g., KL, f-divergence, Wasserstein) that critically impact update behavior, stability, and efficiency across applications.

Divergence Regularized Policy Optimization (DRPO) denotes a family of policy-optimization methods in which expected return, regularized value, or a surrogate improvement objective is optimized jointly with an explicit divergence penalty to a reference object such as a prior policy, the previous iterate, a behavior policy, or an occupancy measure. In the regularized-MDP literature, this family is formalized through convex regularizers and the induced Bregman divergences; in modern deep RL it appears as trust-region, mirror-descent, occupancy-measure, and offline-regularized policy optimization; and in LLM post-training it has acquired a more specific meaning tied to divergence-based trust regions for autoregressive policies (O'Donoghue, 2022, Yao et al., 8 Jun 2026). The acronym is not fully standardized: in some contemporary reasoning work, “DRPO” instead denotes “Decoupled Reward Policy Optimization,” a distinct construction whose connection to divergence regularization is indirect (Li et al., 6 Oct 2025).

1. Conceptual scope

At its most general, DRPO optimizes an objective of the form

$$\max_\pi \; \text{return}(\pi) - \lambda\, D(\pi,\pi_{\mathrm{ref}}),$$

or an equivalent constrained formulation in which the policy is required to remain within a divergence ball around a reference policy or distribution. The divergence can be KL, a general $f$-divergence, a Bregman divergence induced by a convex mirror map, a relative Pearson divergence, a Wasserstein-style penalty, or an occupancy-measure discrepancy, depending on the state space, policy parameterization, and intended trust-region geometry (Belousov et al., 2019, Wang et al., 25 Jan 2025).

Two structural choices recur across the literature. The first is the reference object: some methods regularize toward a fixed prior or reference policy, some toward the previous policy iterate, some toward an offline behavior policy, and some toward a state-action visitation distribution. The second is the location of the divergence: it may be imposed on per-state action distributions, on state-action occupancy measures, on flow fields in continuous generative models, or on sampled-token probability shifts in LLM RL. This suggests that DRPO is better understood as a design space than as a single algorithmic template.

2. Mathematical basis in regularized MDPs

A canonical formalization appears in regularized discounted MDPs with a continuously differentiable, strictly convex regularizer $\Omega : \Delta(A)\to\mathbb{R}$. For a policy $\pi$, the regularized value functions are

$$Q_\Omega^\pi(s,a) = r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, V_\Omega^\pi(s'),$$

$$V_\Omega^\pi(s) = \sum_a \pi(s,a)\, Q_\Omega^\pi(s,a) - \Omega(\pi(s,\cdot)),$$

and the optimal regularized policy $\pi_\Omega^\star$ is the unique statewise maximizer of the corresponding regularized Bellman objective (O'Donoghue, 2022).

The associated Bregman divergence is

$$D_\Omega(\pi,\pi')=\Omega(\pi)-\Omega(\pi')-\nabla\Omega(\pi')^\top(\pi-\pi').$$

The central theorem establishes a direct value–divergence relation: $$\mathbb{E}_{s \sim \mu^\pi}\Big[ D_\Omega(\pi(s,\cdot), \pi_\Omega^\star(s,\cdot))\Big] \le (1-\gamma)\,\mathbb{E}_{s \sim \rho}\big[ V_\Omega^\star(s) - V_\Omega^\pi(s)\big].$$ If $\pi_\Omega^\star$ lies in the relative interior of the simplex for each state, then equality holds. Under negative entropy regularization, $f$0, the Bregman divergence becomes $f$1 times KL divergence, and the theorem specializes to the exact identity

$f$2

This identity gives DRPO a precise interpretation: divergence to the optimal regularized policy is not merely a heuristic stabilizer but a quantitative certificate of regularized value suboptimality (O'Donoghue, 2022).

3. Mirror descent, trust regions, and divergence choice

The algorithmic core of DRPO is typically a regularized policy-improvement step. In generalized policy mirror descent, for instance, the statewise update is

$f$3

where $f$4 is a convex regularizer and $f$5 is the corresponding Bregman divergence (Zhan et al., 2021). This formulation unifies entropy regularization, KL regularization toward a reference distribution, Tsallis regularization, weighted $f$6 costs, and log-barrier constraints. The same work proves linear convergence over an entire range of learning rates, even when the regularizer lacks strong convexity and smoothness, and shows that this linear rate is stable under inexact policy evaluation and imperfect policy updates (Zhan et al., 2021).

A parallel line replaces KL with general $f$7-divergences. In “f-Divergence constrained policy improvement,” the policy-improvement solution is expressed through the derivative of the convex conjugate of $f$8, with KL recovered as a special case. The paper emphasizes that each policy-improvement geometry comes with a compatible evaluation geometry: the Pearson $f$9-divergence is closely related to mean-squared Bellman error minimization, whereas KL yields the soft-max policy update and a log-sum-exp critic (Belousov et al., 2017). “Entropic Regularization of Markov Decision Processes” develops the same theme in average-reward RL, casting regularized control as optimization over state-action distributions and showing that common least-squares value estimation plus advantage-weighted maximum-likelihood policy improvement correspond to a Pearson $\Omega : \Delta(A)\to\mathbb{R}$0-divergence penalty (Belousov et al., 2019).

The choice of divergence is therefore not cosmetic. PPO-RPE replaces PPO’s raw density-ratio clipping with relative Pearson divergence, introducing a symmetric relative density ratio $\Omega : \Delta(A)\to\mathbb{R}$1 based on the mixture policy $\Omega : \Delta(A)\to\mathbb{R}$2. For $\Omega : \Delta(A)\to\mathbb{R}$3, $\Omega : \Delta(A)\to\mathbb{R}$4 is symmetric around $\Omega : \Delta(A)\to\mathbb{R}$5, and the threshold is defined in this relative-ratio domain rather than directly on the asymmetric raw ratio $\Omega : \Delta(A)\to\mathbb{R}$6. This yields a divergence-derived trust-region parameterization rather than a purely heuristic clip interval (Kobayashi, 2022).

4. Occupancy-measure, offline, and federated extensions

A major extension of DRPO replaces per-state action divergence by discrepancies between discounted visitation distributions. “Stable Policy Optimization via Off-Policy Divergence Regularization” argues that the performance-difference lower bound is controlled by total variation between discounted state visitation distributions, not directly by per-state action KL, and proposes regularizing the divergence between discounted state-action visitation measures of consecutive policies. The resulting PPO-DICE objective augments PPO’s clipped surrogate with a divergence penalty over $\Omega : \Delta(A)\to\mathbb{R}$7 and $\Omega : \Delta(A)\to\mathbb{R}$8, estimated adversarially through a DualDICE-style variational objective (Touati et al., 2020).

“Divergence-Augmented Policy Optimization” develops a closely related occupancy-measure view. Its mirror-descent objective is

$\Omega : \Delta(A)\to\mathbb{R}$9

where the Bregman divergence is applied directly to discounted state-action distributions rather than only to action probabilities. For a conditional-KL choice of $\pi$0, the gradient becomes a multi-step divergence augmentation of the advantage term, and the paper proves that the bias introduced by omitting state-density ratios is bounded by the conditional KL term: $\pi$1 This gives the regularizer a dual role as both trust region and off-policy bias control (Wang et al., 25 Jan 2025).

Offline and federated variants generalize the same idea to more severe distribution shift. “Federated Offline Policy Optimization with Dual Regularization” introduces a local objective with two penalties,

$\pi$2

to address two-tier shift: between the learned policy and the local behavior policy, and between local policies and the global aggregated policy. The theoretical analysis states that, by balancing these two regularizers appropriately, strict policy improvement can be ensured in each federative round (Yue et al., 2024). In robust RL, “Policy Regularized Distributionally Robust Markov Decision Processes with Linear Function Approximation” adds KL regularization toward a reference policy inside robust Bellman equations, yielding Boltzmann policies of the form

$\pi$3

and proving polynomial sample-efficiency guarantees for online robust policy optimization with linear function approximation (Gu et al., 16 Oct 2025).

5. LLM and generative-model post-training

In LLM RL, DRPO has acquired a more specific operational meaning. “Rethinking the Divergence Regularization in LLM RL” observes that ratio clipping is a poor proxy for distributional shift in long-tailed vocabularies and replaces hard-mask trust-region control with a smooth divergence regularizer on sampled-token probability shift. Its DRPO objective is

$\pi$4

which preserves the same Binary-TV trust-region geometry as DPPO while producing bounded, continuous gradient weights

$\pi$5

The claimed effect is smooth attenuation of diverging updates together with corrective rollback beyond the trust-region boundary, rather than simple gradient masking (Yao et al., 8 Jun 2026).

A second contemporary direction broadens the divergence class beyond KL. “Beyond KL Divergence: Policy Optimization with Flexible Bregman Divergences for LLM Reasoning” formulates group-based mirror policy optimization with arbitrary Bregman divergences, including hand-designed $\pi$6 geometry in probability space and learned neural mirror maps. On GSM8K, ProbL2-GRPO reaches $\pi$7 accuracy, improving $\pi$8 points over the Dr. GRPO baseline; on MBPP, neural mirror maps reach $\pi$9–$$Q_\Omega^\pi(s,a) = r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, V_\Omega^\pi(s'),$$0 pass@1, with random initialization already capturing most of the gain (Yuan et al., 4 Feb 2026). This suggests that divergence selection is itself a primary optimization variable in LLM reasoning, not merely a secondary stabilization device.

Adaptive DRPO extends the same principle to sample-wise regularization strength. “Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models” replaces fixed $$Q_\Omega^\pi(s,a) = r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, V_\Omega^\pi(s'),$$1 by $$Q_\Omega^\pi(s,a) = r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, V_\Omega^\pi(s'),$$2, using weaker regularization on high-advantage samples and stronger regularization on poor samples: $$Q_\Omega^\pi(s,a) = r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, V_\Omega^\pi(s'),$$3 The paper instantiates this with Wasserstein-2 regularization for flow-matching text-to-image models and KL regularization for LLMs and multimodal reasoning, arguing that adaptive regularization improves the exploration–exploitation balance across modalities (Fan et al., 20 Oct 2025).

6. Terminology, misconceptions, and open problems

Several misconceptions recur in discussions of DRPO. First, DRPO is not synonymous with KL regularization. The theory around regularized MDPs and mirror descent explicitly covers any continuously differentiable strictly convex regularizer in the Bregman case, and policy-improvement work based on $$Q_\Omega^\pi(s,a) = r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, V_\Omega^\pi(s'),$$4-divergences, relative Pearson divergence, Tsallis entropy, Wasserstein-style surrogates, and learned mirror maps demonstrates that the choice of geometry can materially change update behavior, critic structure, variance, sparsity, and sample efficiency (O'Donoghue, 2022, Belousov et al., 2017, Yuan et al., 4 Feb 2026).

Second, DRPO does not always regularize the same object. Some methods act on policy simplices state by state; some regularize state-action occupancy measures; some regularize flow fields; and some LLM objectives regularize sampled-token probability shift. A separate geometric usage regularizes the divergence of the policy-gradient vector field in a learned Riemannian manifold rather than divergence between policies at all. “Deep Metric Tensor Regularized Policy Gradient” is an example of this latter interpretation, where the regularizer is the squared divergence of the gradient field in parameter space (Chen et al., 2023). This broader usage is conceptually related but not equivalent to policy-distribution regularization.

Third, the acronym itself is overloaded. In the reasoning-model literature, “DRPO” can denote “Decoupled Reward Policy Optimization,” whose core construction is an optimized positive distribution under KL regularization rather than the general divergence-regularized policy-optimization family (Li et al., 6 Oct 2025). This suggests that technical context, not the acronym alone, is necessary for disambiguation.

Current open directions are clear from the literature. One line concerns learned geometries: neural mirror maps for group-based LLM optimization and adaptive divergence coefficients indicate that regularization geometry need not be fixed a priori (Yuan et al., 4 Feb 2026, Fan et al., 20 Oct 2025). A second concerns distribution shift: occupancy-measure trust regions, federated dual regularization, and robust policy regularization all point toward DRPO formulations in which the relevant discrepancy is global and multi-step rather than purely local in action space (Wang et al., 25 Jan 2025, Yue et al., 2024, Gu et al., 16 Oct 2025). A third concerns theory under approximation: recent results give linear convergence for broad classes of convex regularizers and explicit off-policy bias bounds under occupancy-level divergence control, but large-scale deep nonlinear settings remain less settled (Zhan et al., 2021, Wang et al., 25 Jan 2025).