Reverse-KL Regularization

• Reverse-KL regularization is a technique that aligns a learned model with a reference distribution using a mode-seeking KL divergence.
• It enhances sample efficiency and strong convexity, yielding improved convergence in RL, RLHF, and LLM alignment tasks.
• Careful estimator design and tuning of the regularization strength are essential to balance mode fidelity with diversity.

Reverse-KL regularization is a widely studied technique in modern machine learning and reinforcement learning that couples optimization objectives to prior (reference) distributions through the reverse Kullback–Leibler divergence. Intuitively, this regularization encourages the learned distribution or policy to remain close to a reference—such as a pretrained model, a behavior policy, or a teacher model—while seeking solutions (modes) with higher reward or likelihood under observed data. It is distinct from forward-KL regularization in its mode-seeking and zero-forcing inductive bias and underpins numerous theoretical results, algorithmic frameworks, and practical benchmarks across offline RL, RLHF, LLM alignment, and generative modeling.

1. Mathematical Definition and Mode-Seeking Properties

Reverse-KL divergence between two distributions $p(x)$ (the learned model or policy) and $q(x)$ (the reference) is defined as

$$D_{\mathrm{KL}}(p \| q) = \int p(x) \log\left( \frac{p(x)}{q(x)} \right) dx.$$

Minimizing this divergence penalizes the learned distribution $p$ for assigning probability mass where $q$ is small (zero-forcing), but not for ignoring regions where $q$ has mass but $p$ does not. This is in contrast to forward-KL, which is mean-seeking and penalizes mass-miss more heavily. The consequence is that reverse-KL regularization leads policies or distributions to "commit" to high-density modes of the reference or target and strongly avoid out-of-distribution regions, a property usually referred to as mode-seeking behavior (Cai et al., 2022, Yao et al., 16 Feb 2025, Shi et al., 2024).

For policy learning, the foundational optimization objective reads: $$J(\pi) = \mathbb{E}_{y \sim \pi}[r(y)] - \lambda D_{\mathrm{KL}}(\pi \| \pi_{\mathrm{ref}})$$ where $\pi$ is the policy to be learned, $\pi_{\mathrm{ref}}$ is the reference policy (e.g., an earlier model, pretrained model, or behavior policy), $q(x)$0 is the reward, and $q(x)$1 is a regularization strength (GX-Chen et al., 23 Oct 2025, Aminian et al., 3 Feb 2025, Zhang et al., 23 May 2025, Shah et al., 26 Dec 2025).

The closed-form optimizer subject to a simplex constraint is a Boltzmann (Gibbs) policy: $q(x)$2 showing that reverse-KL regularization interpolates between the reference model (large $q(x)$3) and mode-seeking on the reward (small $q(x)$4), with the regularization coefficient $q(x)$5 controlling the trade-off (GX-Chen et al., 23 Oct 2025, Aminian et al., 3 Feb 2025, Wang et al., 2023).

2. Theoretical Guarantees and Sample Efficiency

Reverse-KL regularization profoundly impacts statistical efficiency, learning guarantees, and sample complexity. In RL and RLHF, it renders the loss strongly convex around the reference, allowing algorithms to achieve suboptimality gaps and regret bounds that scale linearly in the inverse accuracy, improving over the quadratic dependence typical of unregularized or forward-KL settings (Zhao et al., 2024, Nayak et al., 15 Oct 2025, Aminian et al., 3 Feb 2025).

Key theoretical results established include:

• In contextual bandits and RLHF, O(1/ε) sample complexity for suboptimality ε, under sufficient data coverage and fixed regularization, as opposed to the O(1/ε²) scaling without KL regularization (Zhao et al., 2024).
• In KL-regularized zero-sum Markov games, logarithmic regret O((1/β) log²(T)) can be achieved with strong reference anchoring, where β is the reverse-KL penalty and T is episode count (Nayak et al., 15 Oct 2025).
• In multi-reference RLHF, the optimal solution is a softmax over a geometric mixture of references, yielding convergence rates O(1/n) for the suboptimality gap with n samples (Aminian et al., 3 Feb 2025).

The table below summarizes the sample complexity across prominent settings: | Setting | Reverse-KL Sample Complexity | Forward-KL Sample Complexity | |---------------------------|-----------------------------------|------------------------------| | Contextual Bandits/RLHF | O(η/ε), O(log²T) (log regret) | O(1/ε²), O(√T) | | Multi-reference RLHF | O(1/n) for suboptimality gap | O(1/√n) | | KL-regularized Markov SGs | O((1/β) log²T) | N/A |

3. Algorithmic Implementations and Surrogate Losses

Reverse-KL regularization arises in actor-critic, policy gradient, trust region, and generative modeling frameworks. Proper implementation of the regularizer in deep RL and RLHF is nuanced, as estimator design affects unbiasedness, stability, and downstream performance (Shah et al., 26 Dec 2025, Liu et al., 2 Oct 2025, Zhang et al., 23 May 2025).

In RLHF and LLM alignment, estimator configurations for reverse-KL regularization can be categorized as:

• K1-in-reward: Subtracts the log-ratio estimator from the RL reward with the coefficient detached, yielding an unbiased policy gradient (score-function estimator) and supporting stable, high-performance training (Shah et al., 26 Dec 2025, Liu et al., 2 Oct 2025).
• K2-as-loss: Fully differentiable squared log-ratio penalty, gradient-equivalent to K1-in-reward for on-policy updates but not off-policy (Liu et al., 2 Oct 2025).
• K3 ("low variance") estimators: Biased first-order approximations, unstable in practice and discouraged (Shah et al., 26 Dec 2025, Liu et al., 2 Oct 2025, Zhang et al., 23 May 2025).

Correct application in off-policy or asynchronous scenarios requires explicit importance weighting of the KL term (Liu et al., 2 Oct 2025, Zhang et al., 23 May 2025); many practical implementations omit these weights, causing drift and bias.

In reinforcement learning, reverse-KL is used in greedification/operators for soft actor-critic (SAC) (Zhang et al., 2 Jun 2025, Chan et al., 2021), where the actor minimizes

$q(x)$6

which is equivalent to minimizing the reverse-KL divergence to a Boltzmann policy. Bidirectional algorithms initialize with a forward-KL step and refine with reverse-KL for stability and improvement guarantees (Zhang et al., 2 Jun 2025).

4. Mode-Seeking, Collapse, and Diversity

The mode-seeking nature of reverse-KL induces specific behaviors in model fitting:

• Policy or model assigns probability mass predominantly to regions where the reference has high support and the reward is high.
• For multimodal reference or behavior distributions, reverse-KL avoids out-of-distribution interpolations, "committing" to a single mode (Cai et al., 2022, Shi et al., 2024, Yao et al., 16 Feb 2025).
• In the limit of low regularization (λ→0), the solution collapses to the highest-reward mode, potentially inducing diversity loss ("mode collapse") (GX-Chen et al., 23 Oct 2025).

Extending reverse-KL regularization with mechanisms such as diffusive smoothing (He et al., 2024) or mode-anchored reward augmentation (MARA) (GX-Chen et al., 23 Oct 2025) can restore mode coverage, explicitly flattening the objective over high-reward regions to prevent collapse. Table below highlights typical behaviors:

Regime	λ Low (Strong RKL)	λ High (Weak RKL)	With Diffusive/MARA
Mode Seeking	Yes (collapse)	No (spread)	Yes (multi-mode)
Diversity	Low	Higher	High (if enforced)

5. Applications in RL, RLHF, and LLM Alignment

Reverse-KL regularization is integral in:

• Offline RL: Per-state reverse-KL penalties improve behavior cloning in mixed-policy datasets, strongly outperformed mean-seeking regularizers by avoiding OOD actions (Cai et al., 2022).
• RLHF: Principal regularizer for alignment by controlling policy deviation from the reference, with theoretical equivalence to Direct Preference Optimization (DPO) via the dual Boltzmann optimality (Wang et al., 2023, Aminian et al., 3 Feb 2025). Reverse-KL also provides sharp theoretical calibration error bounds and sample efficiency (Aminian et al., 3 Feb 2025, Zhao et al., 2024, Yao et al., 16 Feb 2025).
• LLM distillation and supervised alignment: Reverse-KL as a distillation or knowledge-transfer loss transfers modes directly and outperforms forward-KL in settings with high reference confidence or label corruption (Shi et al., 2024, Yao et al., 16 Feb 2025). It improves generalization by focusing on high-confidence teacher predictions.
• Zero-sum Games and Self-Play: Reverse-KL-anchored solutions enable alignment and regret-efficient training with pretrained LLMs as reference in game-theoretic learning (Nayak et al., 15 Oct 2025).

6. Extensions, Advanced Objectives, and Controversies

Generalizations of reverse-KL regularization and hybrid divergences further refine its properties:

• Surprisal-Rényi Free Energy (SRFE): Extends reverse-KL by incorporating mean-variance trade-offs, controlling not only expectation but tail deviations, and interpolates between mass-covering and mode-seeking regimes (Matsumoto et al., 3 Mar 2026).
• Diffusive KL: Aggregates reverse-KL over convolved, noise-blurred densities to regularize training and enforce coverage over all modes in multimodal settings (He et al., 2024).
• Multiple Reference Models: The optimal solution under reverse-KL with mixture references is a softmax over a geometric mixture, with improved adaptability and convergence guarantees (Aminian et al., 3 Feb 2025).
• Mode Collapse Debate: Recent work demonstrates that mode-seeking or collapse under reverse-KL is not universal; it depends on relative scales of reward, reference, and regularization strength. Explicit reward shaping or augmentation is necessary for ensuring full mode coverage (GX-Chen et al., 23 Oct 2025).

7. Practical Recommendations and Empirical Insights

Empirically, reverse-KL regularization outperforms mean-seeking methods in alignment, RLHF, and offline RL tasks when the objective is mode fidelity, safety, or reward maximization under high-confidence priors (Cai et al., 2022, Aminian et al., 3 Feb 2025, Shah et al., 26 Dec 2025). However, proper estimator configuration and handling of bias/variance trade-offs are critical for stability and out-of-distribution generalization (Shah et al., 26 Dec 2025, Liu et al., 2 Oct 2025). Careful annealing of regularization strength or advance in algorithm design is warranted to balance mode-fidelity and diversity for downstream applications (GX-Chen et al., 23 Oct 2025, He et al., 2024).

A summary of guidelines:

Scenario	Recommended RKL configuration	References
RLHF/LLM fine-tuning	K1-in-reward, stop-gradient	(Shah et al., 26 Dec 2025, Liu et al., 2 Oct 2025)
Off-policy RL	Importance-weighted K1/K2 in loss	(Liu et al., 2 Oct 2025, Zhang et al., 23 May 2025)
Multimodal generative tasks	Diffusive or MARA-augmented RKL	(GX-Chen et al., 23 Oct 2025, He et al., 2024)

Reverse-KL regularization thus remains a fundamental component in state-of-the-art policy optimization, characterized by its mode-seeking behavior, strong convexity for statistical learning, and adaptability across advanced objectives in modern machine learning frameworks.