Kernel Kullback–Leibler Divergence (KKL)

• Kernel Kullback–Leibler Divergence (KKL) is a family of methods that generalizes classical KL divergence by mapping probability measures into reproducing kernel Hilbert spaces using covariance operators.
• It employs operator-based, regularized, and variational formulations to ensure finite and consistent divergence estimation even when data supports do not perfectly overlap.
• KKL facilitates robust divergence estimation in high-dimensional applications by integrating gradient-based optimization and low-rank approximations for computational efficiency.

Kernel Kullback–Leibler Divergence (KKL) encompasses a family of divergences that generalize or extend the classical Kullback–Leibler (KL) divergence to the field of reproducing kernel Hilbert spaces (RKHS), operator algebra, and kernel density frameworks. These methods provide theoretically rigorous and computationally practical approaches to comparing probability measures beyond conventional density-ratio-based KL, leveraging the structure of kernels and RKHS for consistent estimation, improved stability, and applicability to infinite-dimensional spaces.

1. Key Definitions and Formulations

The Kernel Kullback–Leibler Divergence is not a single object but includes several concrete constructions:

Operator-based KKL (Quantum Kullback–Leibler)

Given a positive-definite kernel $k$ with corresponding RKHS $\mathcal{H}$ and feature map $\phi$, each probability measure $P$ on $\mathbb{R}^d$ admits a covariance operator: $\Sigma_P = \int \phi(x) \otimes \phi(x) \, dP(x)$ The KKL is then defined using the quantum KL (or quantum relative entropy) of operators: $$D_{\mathrm{KKL}}(P \| Q) = \operatorname{Tr}\left[ \Sigma_P (\log\Sigma_P - \log\Sigma_Q) \right]$$ provided $$\mathrm{supp}(\Sigma_P) \subseteq \mathrm{supp}(\Sigma_Q)$$, and is $+\infty$ otherwise. This formulation extends the notion of KL from measures to covariance operators in a potentially infinite-dimensional (RKHS) setting (Chazal et al., 29 Aug 2024).

Regularized KKL (Skewed/Smoothed Variant)

To ensure finiteness and applicability to measures with mismatched or disjoint supports, a regularization scheme is introduced: $$\Sigma_{Q,\alpha} = (1-\alpha)\Sigma_Q + \alpha \Sigma_P$$ and

$$D_{\mathrm{KKL}}^{\alpha}(P \| Q) = \operatorname{Tr}\left[ \Sigma_P \log\Sigma_P \right] - \operatorname{Tr}\left[ \Sigma_P \log\Sigma_{Q,\alpha} \right]$$

for $\alpha \in (0,1)$, which enjoys $D_{\mathrm{KKL}}^{\alpha}(P \| Q) < \infty$ always and recovers the original KKL as $\alpha \to 0$ (Chazal et al., 29 Aug 2024).

RKHS Variational KKL via Donsker-Varadhan

Through the Donsker–Varadhan variational representation,

$$D_{\mathrm{KL}}(P\|Q) = \sup_{f\in\mathcal{F}} \left\{ \mathbb{E}_P[f] - \log\mathbb{E}_Q[e^{f}] \right\}$$

and restricting $\mathcal{F}$ to functions in an RKHS $\mathcal{H}_K$ (with a norm constraint or regularization), a kernelized empirical KL estimator is realized as a convex finite-dimensional program over the kernel coefficients of $f$ (Ahuja, 2019, Ghimire et al., 2021).

2. Statistical and Consistency Properties

Kernel Kullback–Leibler divergences have several crucial theoretical properties:

• Monotonicity and regularization: $D_{\mathrm{KKL}}^{\alpha}(P \| Q)$ is nonincreasing in $\alpha$ and vanishes if and only if $P = Q$ (Chazal et al., 29 Aug 2024).
• Strong consistency: For empirical covariance operators from i.i.d. samples, concentration inequalities guarantee that $$|\widehat{D}_{\mathrm{KKL}}^{\alpha} - D_{\mathrm{KKL}}^{\alpha}|$$ contracts at $O(1/\sqrt{n \wedge m})$, with explicit, dimension-independent rates (Chazal et al., 29 Aug 2024, Quang, 2022).
• Operator continuity: The KKL functional is continuous in the Hilbert–Schmidt norm for covariances, enabling its estimation via empirical Gram matrices (Quang, 2022).
• Convexity: The variational RKHS KL estimator's objective is convex in coefficients, ensuring efficient and globally optimal optimization (Ahuja, 2019).

3. Algorithmic Approaches

Finite-sample Operator KKL

Given samples $X = \{x_i\}_{i=1}^n \sim P$, $Y = \{y_j\}_{j=1}^m \sim Q$:

• Construct Gram matrices for data and, if needed, cross-Gram for $P$ and $Q$:

$K_{XX}, \quad K_{YY}, \quad K_{XY}$

• For the regularized KKL, assemble the block matrix:

$$K = \begin{pmatrix} \frac{\alpha}{n} K_{XX} & \sqrt{\frac{\alpha(1-\alpha)}{nm}} K_{XY} \ \sqrt{\frac{\alpha(1-\alpha)}{nm}} K_{XY}^\top & \frac{1-\alpha}{m} K_{YY} \end{pmatrix}$$

• Compute the divergence:

$$D_{\mathrm{KKL}}^{\alpha}(\widehat{P}_n \| \widehat{Q}_m) = \frac{1}{n}\log\frac{1}{n} - \operatorname{Tr}[I_\alpha K \log K]$$

where $I_\alpha$ selects the $\alpha$-component for $P$ (Chazal et al., 29 Aug 2024).

Computational complexity is dominated by $O((n+m)^3)$ for eigendecomposition of the block matrix. For large datasets, low-rank approximations (e.g., Nyström, random Fourier features) reduce this cost (Quang, 2022).

Variational RKHS KKL (DV-based)

• Collect $n$ samples from $P$, $m$ from $Q$, pool into $Z = \{z_1, ..., z_{n+m}\}$.
• Let $K$ denote the full Gram matrix.
• Solve

$$\widehat{D}_{\mathrm{KL}} = \sup_{\alpha} \left\{ \frac{1}{n} \sum_{i=1}^n f_\alpha(x_i) - \log \left( \frac{1}{m} \sum_{j=1}^m e^{f_\alpha(y_j)} \right) - \lambda \|f_\alpha\|_{H_k}^2 \right\}$$

with $f_\alpha(z) = \alpha^\top K_{:,z}$ and $\|f_\alpha\|^2_{H_k} = \alpha^\top K \alpha$.

• Optimize via (projected) gradient ascent on $\alpha$ (Ahuja, 2019, Ghimire et al., 2021).

Wasserstein Gradient Descent with KKL

• Evaluate the first variation of $D_{\mathrm{KKL}}^{\alpha}(P \| Q)$ with respect to the support points of $P$ using closed-form expressions involving the kernel Gram matrix and its eigendecomposition.
• Update the support points along the negative gradient to minimize KKL, yielding a Wasserstein flow (Chazal et al., 29 Aug 2024).

4. Relationships to Other Kernel and Information-Theoretic Quantities

• The KKL divergence employs second-order (covariance) summaries rather than first-order (mean) or pointwise density ratios, distinguishing it from methods like Maximum Mean Discrepancy (MMD).
• Under certain circumstances, the KKL vanishes only if $P=Q$ (with characteristic kernels), similar to MMD, yet provides a quantum information-theoretic analogue via operator means.
• Variational (RKHS) KKL generalizes MINE (Mutual Information Neural Estimator) by substituting neural network discriminators with RKHS functions, yielding improved consistency and stability, especially in small-sample regimes (Ahuja, 2019, Ghimire et al., 2021).

5. Applications and Empirical Properties

• Stable KL Estimation: RKHS-based methods demonstrate a reduction in estimator variance and improved training stability across tasks such as estimating KL between Gaussians, mutual information, and adversarial variational Bayes—where neural network discriminators often suffer from high variance or exploding estimates (Ghimire et al., 2021).
• Transport and Generative Modeling: KKL supports Wasserstein gradient flows that transport empirical measures to targets in a geometrically meaningful way. In synthetic examples (e.g., mapping point clouds onto rings or spirals), KKL-driven flows exhibit sharper support preservation and faster convergence compared to MMD flows (Chazal et al., 29 Aug 2024).
• Goodness-of-Fit Testing: Bias-reduced kernel density estimators yield strongly consistent KLD estimates suitable for model selection and hypothesis testing (Ngom et al., 2018).
• Thermodynamic Formalism and Symbolic Dynamics: In variant forms, the kernel KL constructs are foundational for the analysis of entropy production via involution kernels in dynamical systems (Lopes et al., 2020).

6. Theoretical and Practical Limitations

• The original unregularized KKL divergence is undefined (infinite) when the supports of covariance operators do not nest; this necessitates regularization ($\alpha > 0$) for empirical or disjointly supported data.
• Both operator-based and variational KKL exhibit $O((n+m)^3)$ computational complexity in their naive implementation, motivating practical acceleration via low-rank approximations (Chazal et al., 29 Aug 2024, Quang, 2022).
• While RKHS methods afford strong consistency and theoretical guarantees, their performance degenerates if the RKHS is too restrictive; empirical bias-variance trade-offs depend on kernel choice, regularization strength, and feature richness.

7. Connections to Broader Theory and Future Directions

• Regularized KKL forms a smooth interpolation between classical KL and kernel-based metrics such as MMD, enabling flexible optimization in Wasserstein space and the exploration of new information-theoretic structures not tied to densities (Chazal et al., 29 Aug 2024).
• The empirical operator framework developed for KKL is foundational for dimension-independent estimation of information-theoretic quantities, exploiting concentration results for Hilbert–Schmidt operators (Quang, 2022).
• Open directions include extensions to other $f$-divergences, treatment of higher-dimensional or dependent data, and the systematic selection or learning of kernels for improved estimator adaptivity (Ngom et al., 2018).

---

Variant/Formulation	Key Definition/Formula	Core Application
Operator-based KKL	$$D_{\mathrm{KKL}}(P \| Q) = \mathrm{Tr}[\Sigma_P (\log\Sigma_P - \log\Sigma_Q)]$$	Covariance operator embedding; quantum info theory
Regularized (Skewed) KKL	$$D_{\mathrm{KKL}}^{\alpha}(P \| Q) = \mathrm{Tr}[\Sigma_P \log\Sigma_P] - \mathrm{Tr}[\Sigma_P \log \Sigma_{Q,\alpha}]$$	Disjoint supports; practical estimation
Variational RKHS KKL (DV)	$$\sup_{f \in H_k,\,\|f\| \leq M} \mathbb{E}_P[f] - \log\mathbb{E}_Q[e^{f}]$$	KL/MI estimation; stable training

These approaches constitute a nonparametric, theoretically grounded, and practically implementable methodology for divergence estimation, with rigorous statistical guarantees, broad applicability in machine learning, and deep ties to both information theory and operator algebra.