Divergence-Regularized Guidance

• Divergence-regularized guidance is a framework that uses f-divergence measures to control the alignment between learned and target distributions across various modeling tasks.
• It enhances diffusion models by integrating discriminator and f-divergence sampling to improve sample diversity and mitigate mode collapse with measurable FID improvements.
• The approach extends to optimal transport, reinforcement learning, and function estimation by providing theoretical guarantees, stability, and effective bias-variance trade-offs.

Divergence-regularized guidance encompasses a suite of techniques that employ statistical divergence measures—typically $f$-divergences—as explicit objectives or regularizers to shape the behavior of learning systems. These methods provide a principled framework for aligning generative models, training discriminators, controlling sample distributions in optimal transport and reinforcement learning, and tuning regularization in function estimation. This entry reviews the principal formulations, theoretical guarantees, and practical implementations of divergence-regularized guidance, with an emphasis on recent advancements in diffusion models, optimal transport, reinforcement learning, and classical $L_2$-regularized estimators.

1. Core Principles of Divergence-Regularized Guidance

The unifying principle of divergence-regularized guidance is the explicit penalization—or direct control—of the divergence between a "learned" distribution and a reference or target distribution. The most common divergences are $f$-divergences, encompassing Kullback-Leibler (KL), Jensen-Shannon, $L^p$, and Hellinger distances. The general form of a divergence-regularized objective is:

$$\max_{p \in \mathcal{P}}\,\mathbb{E}_{x \sim p}[r(x)] - \lambda D_f(p \| q)$$

where $r(x)$ is a reward or utility, $q$ is a baseline or prior, and $D_f$ is an $f$-divergence. This paradigm constrains the learned $p$ to remain close to $q$ in the divergence sense, providing stability and bias-variance trade-offs absent in unconstrained optimization.

In modern diffusion models, divergence-regularized guidance is used to refine sample quality by matching not only outcome distributions but also score (gradient) information, addressing issues such as overfitting or mode collapse. In optimal transport, divergence regularization imbues empirical transport estimators with dimension-independent convergence guarantees. In reinforcement learning, divergence regularization directs the policy induced occupancy towards that of desirable behaviors or datasets, yielding robust data selection and stable policy improvement.

2. Divergence-Regularized Guidance in Diffusion Models

2.1. Discriminator and Classifier Guidance

In score-based diffusion models, "discriminator guidance" trains a time-conditioned discriminator $d_\varphi(x, t)$ to distinguish between real noised data and generated samples. Standard approaches use the cross-entropy loss

$$L_{\mathrm{CE}}^d(\varphi) = \int_0^T \lambda(t) \left[ \mathbb{E}_{x \sim P_t}[-\log \sigma(d_\varphi(x, t))] + \mathbb{E}_{x \sim \hat{P}_t} [-\log(1 - \sigma(d_\varphi(x, t)))] \right] dt$$

where $\sigma$ is the sigmoid nonlinearity. At inference, the discriminator's gradient is added to the score network:

$$s_\theta^\text{refined}(x, t) = s_\theta(x, t) + \nabla_x d_\varphi(x, t).$$

However, cross-entropy alone may drive the model further from the data distribution if the discriminator overfits, as it does not control score gradients. To address this, (Verine et al., 20 Mar 2025) proposes a divergence-regularized objective that directly targets KL minimization by matching score gradients:

$$L_{\mathrm{MSE}}^d(\varphi) = \int_0^T \lambda(t) \, \mathbb{E}_{x_0 \sim P_0, x_t \sim P_{t | x_0}}\left[ \|\nabla_x \log p_t(x_t|x_0) - s_\theta(x_t, t) - \nabla_x d_\varphi(x_t, t) \|^2 \right] dt$$

The overall training loss is

$$L_{\text{train}}^d = L_{\mathrm{MSE}}^d + \gamma L_{\mathrm{CE}}^d$$

with $\gamma$ trading off stability and strict KL control.

2.2. f-Divergence Regularized Sampling

In classifier-guided diffusion, overconfident classifiers cause guidance gradients to vanish. (Javid et al., 8 Nov 2025) introduces $f$-divergence-based sampling gradients:

$$\nabla_x S_D(x, y) = \nabla_x \log p(y | x) - \alpha \nabla_x D_f(q_y \| p(\cdot|x))$$

with explicit formulations for reverse-KL (mode covering), forward-KL (mode seeking), and Jensen–Shannon (balanced) divergences. This regularization maintains diversity (mode coverage) and prevents mode collapse, yielding new state-of-the-art FID scores with negligible overhead.

Guidance Method	FID (ResNet-101)	Precision	Recall
Baseline	2.19	0.79	0.58
FKL guided	2.17	0.80	0.59
RKL guided	2.14	0.79	0.59
JS guided (div.-reg.)	2.13	0.79	0.60

3. Divergence-Regularized Optimal Transport

Divergence-regularized optimal transport (DOT) augments the classical Kantorovich OT problem with an $f$-divergence regularizer:

$$S(\mu, \nu) = \inf_{\pi \in \Pi(\mu, \nu)} \int c(x, y)\, d\pi(x, y) + \int \phi\left(\frac{d\pi}{d(\mu \otimes \nu)}(x, y)\right) d(\mu \otimes \nu)(x, y)$$

where $\phi$ is a convex superlinear function and $\psi$ the convex conjugate.

Yang & Zhang (Yang et al., 2 Oct 2025) prove that under bounded cost and smoothness, the empirical DOT estimator achieves dimension-free parametric rate $\mathcal{O}(n^{-1/2})$ and admits central limit theorems for hypothesis testing and confidence intervals. Practical implementations use Sinkhorn-type algorithms and cross-validation to choose the strength of regularization.

Key advantages:

• Bypasses curse of dimensionality present in unregularized OT.
• Flexible regularizer choice: entropic, quadratic, $L^p$, etc., enabling bias-variance trade-off.
• Valid enables for high-dimensional inference: confidence intervals, sample-splitting, and plug-in variance estimation.

4. Divergence-Regularized Guidance in Reinforcement Learning

Regularized optimal experience replay (ROER) leverages $f$-divergence regularization to relate prioritized experience replay (PER) to occupancy-based reweighting. (Li et al., 4 Jul 2024) frames the off-policy optimization problem as

$$\max_{d^*}\; \mathbb{E}_{(s, a) \sim d^*}[r(s, a)] - \beta D_f(d^* \| d^\mathcal{D})$$

with $d^\mathcal{D}$ the buffer occupancy. The associated dual yields the optimal sampling weights as

$w^*(s, a) = f_*'(\delta_Q(s, a) / \beta)$

where $f_*$ is the convex conjugate and $\delta_Q$ the TD-error. For the KL regularizer, this reduces to

$p(i) \propto \exp(\delta_i / \beta)$

Directly connecting buffer prioritization to divergence minimization yields principled, robust sample selection and improved empirical performance over heuristic PER in MuJoCo, DM Control, and offline-to-online RL.

Task	ROER	PER	UER
Ant-v2	2275 ± 599	1654 ± 343	1153 ± 336
HalfCheetah-v2	10695 ± 183	9240 ± 277	9017 ± 172
Hopper-v2	3010 ± 299	2938 ± 334	2813 ± 481

5. Divergence-Regularized Guidance in Function Estimation

L$_2$-regularized estimators such as smoothing splines, penalized splines, ridge regression, and functional linear regression use explicit divergence (trace of the smoothing matrix, i.e., "degrees of freedom") to guide model complexity selection (Fang et al., 2012). The key result is that

$$\operatorname{div}(\lambda) = \operatorname{tr}(S(\lambda))$$

where $S(\lambda)$ is the hat matrix. Minimizing GCV or SURE then corresponds to balancing bias (residual sum of squares) and divergence (complexity) to select regularization. This approach extends universally to a range of settings and is algorithmically efficient via eigendecomposition or Demmler–Reinsch diagonalization.

6. Implementation Considerations and Empirical Performance

Practical implementation of divergence-regularized guidance requires:

• Efficient computation of divergence terms (autodiff for gradients, matrix traces for splines, log-ratios for buffer priorities).
• Careful tuning of regularization strength (e.g., $\gamma$ for divergence vs. CE in diffusion, $\lambda$ in DOT, $\beta$ in ROER).
• Retaining stabilizing cross-entropy or auxiliary losses in diffusion to avoid pathological overfitting.
• For high-dimensional or overparameterized settings, sufficient regularization to prevent dual potential ill-conditioning or non-Lipschitz potentials (DOT), or gradient explosion (DG in diffusion).

Empirical improvements are consistently observed:

• Diffusion: Lower FID and improved precision/recall (e.g., FID improvement $\sim$0.03–0.06 over baselines in (Verine et al., 20 Mar 2025); FID=2.13 on ImageNet (Javid et al., 8 Nov 2025)).
• Optimal transport: Statistically valid confidence intervals for empirical OT costs in high dimensions (Yang et al., 2 Oct 2025).
• RL: Data efficiency and robust Q-value estimation in MuJoCo/DM Control (Li et al., 4 Jul 2024).
• Function estimation: Unbiased complexity control and automated regularization selection (Fang et al., 2012).

7. Theoretical Guarantees and Limitations

Divergence-regularized guidance methods enjoy strong theoretical guarantees:

• Under mild smoothness conditions, minimizing MSE on gradient scores in diffusion guarantees monotonic KL reduction and first-order convergence of the guided sampler (Verine et al., 20 Mar 2025).
• For DOT, parametric $\mathcal{O}(n^{-1/2})$ rates and central limit theorems guarantee valid inference (Yang et al., 2 Oct 2025).
• In RL, ROER's derivation provides a formal link between TD-error prioritization and occupancy reweighting via convex duality, justifying sampling schemes and bias corrections (Li et al., 4 Jul 2024).
• Classical function estimation benefits from provably unbiased estimators of effective degrees of freedom and principled risk-minimization (Fang et al., 2012).

In practice, limitations include:

• Instability if divergence regularization is too weak (mode collapse, overfitting).
• Computational cost in evaluating higher-order derivatives (autodiff through gradients).
• Potential loss of diversity in overaggressive guidance, necessitating balance via hyperparameters.

Divergence-regularized guidance thus provides a mathematically rigorous, versatile, and empirically validated framework for model fitting, generative modeling, and learning from complex data distributions across statistical and machine learning domains.