KL-Minimal Policy Projection

• The paper introduces KL-minimal policy projection, a method that minimizes KL divergence to satisfy constraints and ensure monotonic policy improvement.
• Practical implementations use dual optimization and analytic updates in both discrete and Gaussian settings to balance reward maximization with exploration.
• Empirical results show enhanced convergence speed and stability in continuous control, policy evaluation, and constrained reinforcement learning applications.

KL-minimal policy projection refers to the family of operations that map a candidate policy onto a constrained or desired policy set by minimizing the Kullback-Leibler (KL) divergence, with or without additional constraints such as expected reward, entropy regularization, or moment constraints. These projections manifest in trajectory optimization, trust region RL, maximum-entropy RL, policy evaluation, and constrained reinforcement learning. KL-minimality ensures exact constraint satisfaction, serves as a core mechanism for monotonic policy improvement, and yields unique analytic updates in both discrete and Gaussian settings.

1. Core Definition and Mathematical Formulation

The KL-minimal policy projection is defined by the following generic program: given a reference (prior) policy $\pi_0$ and a set of constraints $C$ (typically convex), find

$\pi^* = \arg\min_{\pi \in C} D_{KL}(\pi \| \pi_0)$

where for distributions $p$, $q$ over $\mathcal{A}$,

$$D_{KL}(p \| q) = \int p(a) \log \frac{p(a)}{q(a)} da.$$

This form appears directly in trajectory-based RL, policy evaluation, and resource allocation with constraints on reward, moment matches, or support coverage (Akrour et al., 2016, Weissmann et al., 4 Mar 2025, Qiu, 28 Oct 2025).

When $C$ sets linear equality (moment) constraints $A \pi = t$, the closed-form solution is an exponential family: $$\pi^*(a) = \pi_0(a) \exp(\theta^* \cdot a) / Z(\theta^*)$$ where $Z(\theta) = \int \pi_0(a) \exp(\theta \cdot a) da$ and $\theta^*$ is chosen to satisfy the constraints (Qiu, 28 Oct 2025). Additional constraints, such as entropy lower bounds, are incorporated through Lagrange multipliers and dual optimization (Akrour et al., 2016).

2. Trajectory-Based Policy Optimization and Trust-Region Methods

KL-minimal projections are central to policy optimization with improvement and trust-region guarantees. In the model-free, trajectory-based algorithm of "Model-Free Trajectory-based Policy Optimization with Monotonic Improvement," at each step $t$, the policy update is

$$\begin{aligned} & \max_{\pi} \int_s \int_a \tilde{\rho}_t^i(s) \pi(a|s) \tilde{Q}_t^i(s, a) da ds \ & \text{subject to }\mathbb{E}_{s \sim \tilde{\rho}_t^i} [ KL [\pi(\cdot|s) \| \pi_t^i(\cdot|s)] ] \leq \epsilon, \ & \qquad \mathbb{E}_{s \sim \tilde{\rho}_t^i}[H[\pi(\cdot|s)]] \geq \beta. \end{aligned}$$

The dual minimization yields a new policy of form

$$\pi_{t}^{i+1}(a|s) \propto \pi_t^i(a|s)^{\eta^*/(\eta^* + \omega^*)} \exp\left(\frac{1}{\eta^*+\omega^*} \tilde{Q}_t^i(s, a)\right)$$

with $(\eta^*, \omega^*)$ chosen to enforce the KL and entropy constraints (Akrour et al., 2016). In Gaussian cases with quadratic $Q$, the policy update remains within the Gaussian family and is computable in closed form.

Guaranteed monotonic improvement is established: if every per-step policy update strictly respects the KL bound, then the policy return improves by a computable lower bound: $$J(\pi^{i+1}) - J(\pi^i) \geq \sum_{t=1}^{T} \mathbb{E}_{s \sim \rho_t^i, a \sim \pi_t^{i+1}(\cdot|s)} \left[A_t^{\pi^i}(s, a)\right] - C\sqrt{\epsilon}$$ for some constant $C$ (Akrour et al., 2016).

3. Forward and Reverse KL Projections: Role in Greedification and Maximum Entropy RL

KL-minimal projections underpin both forward-KL ($D_{KL}(q \| \pi)$) and reverse-KL ($D_{KL}(\pi \| q)$) greedification or policy updates in maximum-entropy RL. Let $q(a|s) \propto \exp(Q(s, a)/\alpha)$ define the Boltzmann target.

• Reverse-KL: The update $$\pi^* = \arg\min_\pi D_{KL}(\pi(\cdot|s) \| q(\cdot|s))$$ is mode-seeking, yields classical soft policy improvement guarantees, and is the standard under SAC, TRPO, and related algorithms (Chan et al., 2021, Zhang et al., 2 Jun 2025).
• Forward-KL: The update $$\pi^* = \arg\min_\pi D_{KL}(q(\cdot|s) \| \pi(\cdot|s))$$ matches the moments of $q$ (for Gaussians, means/variances exactly) and is mean-seeking. Forward-KL has analytic projection under Gaussian parameterization and is used for stable initialization of policy in "Bidirectional Soft Actor-Critic" (Zhang et al., 2 Jun 2025).

Comparison:

Direction	Policy Mode	Analytic Solution (Gaussian)	Improvement Guarantee
Reverse-KL	Mode-seeking	No	Stronger
Forward-KL	Mean-seeking	Yes	Weaker unless sufficient FKL reduction

Reverse-KL guarantees monotonic soft-policy improvement if the divergence is reduced, while forward-KL can fail to improve, but can enhance exploration and stability (Chan et al., 2021, Zhang et al., 2 Jun 2025).

4. KL-Minimal Projection in Constrained/Regularized and Multi-Policy Settings

KL-minimal projection is central in constrained RL, policy evaluation, and portfolio allocation with moment/affine constraints.

• Projection-Based Constrained Policy Optimization (PCPO): After an unconstrained update, project the policy onto the constraint-satisfying set via a KL-minimizing update, often formulated in parameter space via

$$\min_\theta \frac{1}{2} (\theta - \theta_0)^T F (\theta - \theta_0) \quad \text{s.t.} \quad g_C^T (\theta - \theta^k) + b \le 0$$

with $F$ the Fisher-information matrix (KL Hessian). This ensures each step remains in a KL-trust region and meets the cost constraint (Yang et al., 2020).

• KL Barycenter for Policy Evaluation: For evaluating $N$ target policies by importance sampling, the KL-minimal (barycenter) behavior policy is the arithmetic mixture: $\pi^*(a) = \sum_{i=1}^{N} w_i \pi_i(a)$ minimizing the average KL to the targets. Clustering policies before mixture, as in CKL-PE, further controls worst-case importance weights and dramatically reduces sample complexity (Weissmann et al., 4 Mar 2025).
• Moment-Constrained KL Projection (EGMU): For moment constraints $A \pi = t$, the unique minimizer is exponential tilt of the reference policy, solved efficiently via dual Newton or Bregman–Dykstra methods. Extensions handle inequalities and support-function (robust) constraints (Qiu, 28 Oct 2025).

5. Practical Algorithms and Implementation

KL-minimal projection admits efficient algorithmic solutions:

• Dual methods: Newton ascent on the dual yields globally and quadratically convergent updates for linear/affine constraints (Qiu, 28 Oct 2025).
• Iterative Proportional Fitting / Bregman-Dykstra: For high-dimensional or mixture constraints, coordinate-wise Bregman projections provably converge to the KL projection, even under inequality or convex-set constraints (Qiu, 28 Oct 2025).
• Closed-form analytic updates: For discrete action spaces, simplex optimization with KL and entropy constraints yields closed-form Boltzmann updates, as in cautious policy programming (Zhu et al., 2021).
• Hybrid (Bidirectional) learning: Combining forward and reverse KL projections—initializing with the analytic FKL match and refining via reverse-KL gradients—improves convergence stability and efficiency in continuous control (Zhang et al., 2 Jun 2025).

Numerical stability (LogSumExp, covariance regularization), sensitivity analysis, and robustification against infeasible or uncertain constraints are critical for scalability and accuracy (Qiu, 28 Oct 2025, Yang et al., 2020).

6. Theoretical Guarantees, Limitations, and Policy Improvement

• Monotonic improvement: If each update enforces an expected-KL trust region, then the total policy return is provably non-decreasing up to a computable error (Akrour et al., 2016, Zhu et al., 2021).
• Constraint satisfaction: For convex, feasible constraint sets, the KL projection is unique and strictly positive, naturally preserving support and avoiding categorical collapse (Qiu, 28 Oct 2025).
• Directionality: Reverse-KL projections ensure policy improvement in the soft-return objective; forward-KL lacks this guarantee unless FKL is sufficiently reduced or additional assumptions hold (Chan et al., 2021).

Insufficiently constrained forward-KL updates can degrade return or explore undesirable regions; thus, practical algorithms often combine both directions or enforce additional modulation (e.g., entropy, trust-region scaling) (Chan et al., 2021, Zhang et al., 2 Jun 2025).

7. Empirical Performance and Application Domains

• Continuous control: KL-minimal projection-based updates match or exceed standard policy gradient and TRPO-based methods in convergence speed and sample efficiency, especially when exploiting analytic Gaussian forward-KL solutions (Zhang et al., 2 Jun 2025, Akrour et al., 2016).
• Policy evaluation: CKL-PE and KL-barycenter policies provably minimize variance and regret for importance-sampling-based model selection in bandits (Weissmann et al., 4 Mar 2025).
• Safe and constrained RL: Projection-based methods such as PCPO attain both lower cost violation and higher reward than baselines by explicit KL projection in parameter or policy space (Yang et al., 2020).
• Exploration and model-based RL: PO-MPC and closely related algorithms use KL-minimal projections to balance planner guidance, reward maximization, and exploration, demonstrating state-of-the-art performance in high-dimensional benchmarks (Serra-Gomez et al., 5 Oct 2025).

In summary, KL-minimal policy projection constitutes a mathematically principled and practically versatile mechanism for performing safe, stable, and monotonic policy optimization under a wide array of constraints and regularization regimes. Its analytic tractability in finite and Gaussian settings, coupled with robust dual/primal optimization algorithms, underpins its prominence across modern RL, bandit, and constrained optimization literature (Akrour et al., 2016, Zhang et al., 2 Jun 2025, Qiu, 28 Oct 2025, Yang et al., 2020, Chan et al., 2021, Weissmann et al., 4 Mar 2025, Serra-Gomez et al., 5 Oct 2025).