---
title: Kullback–Leibler Divergence (Relative Entropy)
url: https://www.emergentmind.com/topics/kullback-leibler-entropy
type: topic
---

# Kullback–Leibler Divergence (Relative Entropy)

Kullback–Leibler entropy is an informal name for the **Kullback–Leibler (KL) divergence**, also called **relative entropy** or information divergence. For probability measures $P$ and $Q$ on a common measurable space, with $P\ll Q$, it is defined by
$$
D_{\mathrm{KL}}(P\|Q)
=
\int \log\!\left(\frac{dP}{dQ}\right)\,dP.
$$
For densities $p$ and $q$ relative to a common dominating measure,
$$
D_{\mathrm{KL}}(p\|q)
=
\int p(x)\log\frac{p(x)}{q(x)}\,dx.
$$
The first argument is the distribution being averaged over, conventionally interpreted as the empirical, true, or observed distribution; the second is the reference, theoretical, or model distribution. KL divergence is nonnegative, vanishes exactly when the two distributions coincide almost everywhere, and is generally asymmetric. Strictly, Shannon entropy is a one-distribution functional, whereas KL divergence compares two distributions. This distinction is central to statistical inference, information theory, kinetic theory, Bayesian analysis, stochastic dynamics, machine learning, and multiscale modeling.

## 1. Definition, terminology, and fundamental properties

For a finite distribution $p=(p_1,\ldots,p_m)$ and reference distribution $q=(q_1,\ldots,q_m)$,
$$
D_{\mathrm{KL}}(p\|q)
=
\sum_{j=1}^m p_j\log\frac{p_j}{q_j}.
$$
The usual conventions are
$$
0\log\frac{0}{x}=0,
\qquad
x\log\frac{x}{0}=+\infty
\quad (x>0).
$$
Consequently,
$$
D_{\mathrm{KL}}(p\|q)<\infty
\quad\Longleftrightarrow\quad
p\ll q,
$$
meaning that every event assigned zero probability by $q$ is also assigned zero probability by $p$. If $p_j>0$ and $q_j=0$, the divergence is infinite. This is the mathematical basis for interpreting a zero-probability model prediction contradicted by an observation as strict falsification [0808.4111].

KL divergence satisfies Gibbs’ inequality:
$$
D_{\mathrm{KL}}(p\|q)\geq 0,
$$
with equality if and only if $p=q$. It is not a metric: generally,
$$
D_{\mathrm{KL}}(p\|q)\neq D_{\mathrm{KL}}(q\|p),
$$
and it does not satisfy the triangle inequality. The asymmetry is not a defect of notation but expresses the distinct inferential roles of data and model. Reversing the arguments changes both the weighting distribution and the support condition.

Using natural logarithms gives units of nats; logarithms to base $2$ give bits. KL divergence is invariant under a common invertible change of variables because the Jacobian factors cancel in the density ratio. This contrasts with differential Shannon entropy, which changes under rescaling.

The divergence decomposes into cross-entropy and Shannon entropy:
$$
D_{\mathrm{KL}}(p\|q)
=
-H(p)-\int p(x)\log q(x)\,dx
=
H(p,q)-H(p),
$$
where
$$
H(p)=-\int p(x)\log p(x)\,dx,
\qquad
H(p,q)=-\int p(x)\log q(x)\,dx.
$$
Thus KL divergence is the excess expected logarithmic loss or coding cost incurred when data generated by $p$ are encoded using $q$. Shannon entropy measures uncertainty within one distribution; cross-entropy measures the average code length under another; KL divergence measures their difference.

The divergence is jointly convex:
$$
D_{\mathrm{KL}}\!\left(
\lambda p_1+(1-\lambda)p_2
\middle\|
\lambda q_1+(1-\lambda)q_2
\right)
\leq
\lambda D_{\mathrm{KL}}(p_1\|q_1)
+
(1-\lambda)D_{\mathrm{KL}}(p_2\|q_2).
$$
It is also convex in either argument separately under the usual domain conditions. For a measurable transformation or Markov kernel $T$, the data-processing inequality states
$$
D_{\mathrm{KL}}(P\|Q)
\geq
D_{\mathrm{KL}}(TP\|TQ).
$$
Coarse-graining, marginalization, or stochastic processing therefore cannot increase distinguishability. In finite spaces, aggregating categories into super-categories $F_J=\sum_{j\in J}p_j$ and $G_J=\sum_{j\in J}q_j$ gives
$$
D_{\mathrm{KL}}(F\|G)\leq D_{\mathrm{KL}}(p\|q).
$$

## 2. Statistical inference, likelihood, and information geometry

Let $D=(x_1,\ldots,x_n)$ be independent observations with $m$ possible outcomes. If $n_j$ observations fall in category $j$, the empirical distribution or **type** is
$$
f_j^D=\frac{n_j}{n},
\qquad
n=\sum_{j=1}^m n_j.
$$
For a model distribution $f^M$, the likelihood of the ordered sample is
$$
P(D\mid f^M)
=
\prod_{j=1}^m (f_j^M)^{n_j}
=
\exp\!\left[
-nD_{\mathrm{KL}}(f^D\|f^M)-nH(f^D)
\right].
$$
Since $H(f^D)$ is fixed for observed data, maximizing likelihood is equivalent to minimizing
$$
D_{\mathrm{KL}}(f^D\|f^M).
$$
Thus maximum likelihood is a forward-KL projection of the empirical distribution onto the model family.

The probability of observing a type includes the multiplicity of ordered sequences having that type:
$$
\frac{n!}{n_1!\cdots n_m!}.
$$
Stirling’s approximation yields the large-deviation relation
$$
P(f^D\mid f^M)
\cong
\exp\!\left[-nD_{\mathrm{KL}}(f^D\|f^M)\right],
$$
where $a_n\cong b_n$ means
$$
\lim_{n\to\infty}\frac1n\log\frac{a_n}{b_n}=0.
$$
KL divergence is therefore the exponential rate at which an empirical distribution becomes unlikely under a model.

For a single hypothesis $H_0:f^M=f^0$, the statistic
$$
2nD_{\mathrm{KL}}(f^D\|f^0)
$$
is asymptotically chi-square distributed with $m-1$ degrees of freedom. The model is rejected at significance level $\alpha$ when
$$
2nD_{\mathrm{KL}}(f^D\|f^0)
\geq
\chi^2_{1-\alpha}[m-1].
$$
For a model family $\mathcal M$, the maximum-likelihood projection is
$$
\widehat f^{\mathcal M}
=
\arg\min_{f\in\mathcal M}
D_{\mathrm{KL}}(f^D\|f),
$$
and the residual statistic is
$$
2nD_{\mathrm{KL}}(f^D\|\widehat f^{\mathcal M}).
$$
Its asymptotic degrees of freedom are
$$
\dim(S_m)-\dim(\mathcal M).
$$

For nested families $\mathcal M_0\subset\mathcal M_1$, the likelihood-ratio statistic becomes
$$
2n\left[
D_{\mathrm{KL}}(f^D\|\widehat f^{\mathcal M_0})
-
D_{\mathrm{KL}}(f^D\|\widehat f^{\mathcal M_1})
\right].
$$
In exponential families, the difference admits the Pythagorean form
$$
D_{\mathrm{KL}}(f^D\|\widehat f^{\mathcal M_0})
-
D_{\mathrm{KL}}(f^D\|\widehat f^{\mathcal M_1})
=
D_{\mathrm{KL}}(\widehat f^{\mathcal M_1}\|\widehat f^{\mathcal M_0}).
$$

For two simple hypotheses $f^0$ and $f^1$, the Neyman–Pearson rejection region can be written asymptotically in terms of a difference of KL divergences:
$$
D_{\mathrm{KL}}(f\|f^0)
-
D_{\mathrm{KL}}(f\|f^1)
\geq
\frac1n\log T.
$$
The optimal equal-prior error exponent is the Chernoff information,
$$
C(f^0,f^1)
=
-\min_{0\leq\mu\leq1}
\log\left[
\sum_k (f_k^0)^\mu(f_k^1)^{1-\mu}
\right].
$$

Near equality of $f$ and $g$, KL divergence has a quadratic expansion related to Pearson’s chi-square discrepancy:
$$
2D_{\mathrm{KL}}(f\|g)
\approx
\sum_j\frac{(f_j-g_j)^2}{g_j}.
$$
This local quadratic structure is the information-geometric origin of the Fisher information metric. For discrete normal distributions, KL divergence is a reverse Bregman divergence in natural coordinates:
$$
D_{\mathrm{KL}}(p_\xi\|p_{\xi'})
=
B_F(\xi':\xi),
$$
where $F$ is the log-normalizing function associated with the Riemann theta function [2109.14920].

Unrestricted maximum likelihood gives
$$
\widehat f^M=f^D.
$$
Although this minimizes empirical forward KL exactly, it can overfit sampling noise. Restricted model families, likelihood-ratio tests, complexity penalties, Bayesian priors, maximum-entropy reconstruction, and Markov-order restrictions address this limitation.

## 3. Relation to entropy families and variational principles

KL divergence is the order-one member of the Rényi-divergence family. For $\alpha>0$, $\alpha\neq1$,
$$
D_\alpha(P\|Q)
=
\frac{1}{\alpha-1}
\log\int p^\alpha q^{1-\alpha}\,d\mu.
$$
The limit $\alpha\to1$ is
$$
\lim_{\alpha\to1}D_\alpha(P\|Q)
=
D_{\mathrm{KL}}(P\|Q).
$$
Rényi divergence is nonnegative, obeys data processing, is monotone in $\alpha$ under the stated finiteness conditions, and is additive for product measures. In particular,
$$
D_{1/2}(P\|Q)
\leq
D_{\mathrm{KL}}(P\|Q)
\leq
D_2(P\|Q).
$$
The order-$1/2$ and order-$2$ cases connect KL divergence to Hellinger and $\chi^2$ distances:
$$
D_{1/2}(P\|Q)
=
-2\log\left(1-\frac{H^2(P,Q)}{2}\right),
$$
and
$$
D_2(P\|Q)=\log(1+\chi^2(P,Q)).
$$
Consequently,
$$
H^2(P,Q)
\leq
D_{1/2}(P\|Q)
\leq
D_{\mathrm{KL}}(P\|Q)
\leq
D_2(P\|Q)
\leq
\chi^2(P,Q)
$$
[1001.4448].

Shannon entropy can be recovered by selecting a reference measure. For a discrete probability mass function $p$, taking $Q$ to be counting measure gives
$$
D_{\mathrm{KL}}(P\|Q)
=
\sum_xp(x)\log p(x),
$$
so
$$
H(P)=-D_{\mathrm{KL}}(P\|Q).
$$
Taking $Q$ to be the uniform probability on $m$ points gives
$$
D_{\mathrm{KL}}(P\|Q)=\log m-H(P).
$$
Thus entropy is not itself a two-distribution divergence; it is obtained from relative entropy after specifying a reference measure.

A variational representation is
$$
D_{\mathrm{KL}}(P\|Q)
=
\sup_\varphi
\left\{
\mathbb E_P[\varphi]
-
\log\mathbb E_Q[e^\varphi]
\right\},
$$
under suitable integrability conditions. A categorical characterization shows that, on finite statistical systems, KL divergence is uniquely determined up to a nonnegative scalar by functorial additivity under composition, convex linearity, lower semicontinuity, and vanishing for optimal Bayesian inverses [1402.3067]. In that framework, a deterministic map is a measurement, a stochastic right inverse is a hypothesis about the preimage state, and relative entropy quantifies the information gained when the induced prior is updated to the true distribution.

If a probability kernel $\hat\nu=\{\hat\nu_y:y\in Y\}$ supplies a reference distribution on each fiber of $X\times Y$, then the generalized entropy
$$
H^{\hat\nu}(\pi)
=
-D_{\mathrm{KL}}(\pi\|\hat\nu\,dQ)
$$
is the negative KL divergence between the actual joint law $\pi$ and the law obtained by retaining its $Y$-marginal $Q$ while replacing its conditional distribution on $X$ by $\hat\nu_y$. Choosing $\hat\nu_y=P_X$ yields mutual information:
$$
I(X;Y)
=
D_{\mathrm{KL}}(P_{XY}\|P_X\otimes P_Y).
$$
This conditional-kernel formulation extends entropy and information gain to general measurable spaces [2003.02030].

## 4. Maximum entropy, Bayesian updating, and reconstruction

For incomplete observations, KL projection provides a principled reconstruction mechanism. If the compatible empirical distributions form a set $\mathcal D$, the most probable reconstruction under a model $f^M$ is
$$
\widetilde f^{\mathcal D}
=
\arg\min_{f\in\mathcal D}
D_{\mathrm{KL}}(f\|f^M).
$$
Sanov’s theorem gives
$$
P(f^D\in\mathcal D\mid f^M)
\cong
\exp\!\left[
-nD_{\mathrm{KL}}(\widetilde f^{\mathcal D}\|f^M)
\right].
$$
If $f^M$ is uniform, then
$$
D_{\mathrm{KL}}(f^D\|f^M)
=
\log m-H(f^D),
$$
so minimizing relative entropy is equivalent to maximizing Shannon entropy. Maximum entropy is therefore a prior-dependent information projection, not a reconstruction generated without assumptions.

For a linear constraint
$$
\sum_j f_ja_j=\bar a,
$$
the KL projection has exponential-family form
$$
\widetilde f_j^{\mathcal D}
=
\frac{f_j^M e^{\theta a_j}}{Z(\theta)},
\qquad
Z(\theta)=\sum_k f_k^M e^{\theta a_k}.
$$
For energy constraints and a uniform prior, this becomes the Boltzmann–Gibbs distribution
$$
\widetilde f_j^{\mathcal D}
=
\frac{e^{-\beta E_j}}{Z(\beta)}.
$$
The same construction produces gravity-type models under constraints on distances, costs, origin profiles, destination profiles, stayers, or balanced flows [0808.4111].

For a Dirichlet prior $g\sim\mathcal D(\underline\alpha)$ with $\pi_j=\alpha_j/\alpha$, the prior has the asymptotic form
$$
\rho(g\mid\underline\alpha)
\cong
\exp[-\alpha D_{\mathrm{KL}}(\pi\|g)].
$$
After observing counts, the posterior remains Dirichlet with parameters $\underline\alpha+\underline n$, and
$$
\rho(g\mid\underline\alpha+\underline n)
\cong
\exp\!\left[
-\alpha D_{\mathrm{KL}}(\pi\|g)
-nD_{\mathrm{KL}}(f^D\|g)
\right].
$$
The prior strength $\alpha$ consequently acts in effective sample-size units.

Pólya Tree priors exploit a multiplicative binary partition structure. The logarithm of a Pólya Tree density becomes an additive series of independent beta-branching contributions, yielding a representation of the form
$$
D_{\mathrm{KL}}(f\|\theta)
=
\int f\log f
-
\sum_{\epsilon}
\left[
F(B_{\epsilon0})\log(2Y_{\epsilon0})
+
F(B_{\epsilon1})\log\bigl(2(1-Y_{\epsilon0})\bigr)
\right].
$$
Under summability conditions on the beta parameters, posterior expected KL divergence converges to zero, and posterior mass also concentrates in total variation. A related posterior estimator of differential entropy is consistent in probability for compactly supported vectors under the stated regularity assumptions. In one dimension, truncation at depth approximately
$$
3\log_2 n
$$
is theoretically sufficient for retaining the data-relevant part of the tree [2504.02950].

## 5. Dynamical, thermodynamic, and physical interpretations

KL divergence frequently acts as a Lyapunov functional when one distribution evolves toward a reference equilibrium. For finite-state Markov dynamics,
$$
\dot p(t)=Hp(t),
$$
where $H$ is infinitesimal stochastic. If $p(t)$ and $q(t)$ evolve under the same Markov process, then
$$
\frac{d}{dt}D_{\mathrm{KL}}(p(t)\|q(t))\leq0.
$$
If $q$ is stationary, $Hq=0$, then both
$$
D_{\mathrm{KL}}(p(t)\|q)
\quad\text{and}\quad
D_{\mathrm{KL}}(q\|p(t))
$$
have corresponding monotonicity results under the conditions considered. Detailed balance is not required for the general contraction theorem [1512.02742].

When $q$ is a Boltzmann distribution,
$$
q_i=\frac{e^{-\beta E_i}}{Z(\beta)},
$$
the free-energy difference satisfies
$$
F(p)-F(q)=T D_{\mathrm{KL}}(p\|q)
$$
under the paper’s convention for $T$. KL divergence is therefore a nonequilibrium free-energy excess, and its decrease corresponds to relaxation toward equilibrium.

For reaction networks, populations need not be normalized. The appropriate generalized divergence is
$$
I(P,Q)
=
\sum_i
\left[
P_i\log\frac{P_i}{Q_i}
-(P_i-Q_i)
\right].
$$
For a complex-balanced equilibrium $Q$, mass-action dynamics satisfies
$$
\frac{d}{dt}I(P(t),Q)\leq0.
$$
This extends the Lyapunov interpretation from normalized Markov distributions to nonlinear chemical reaction networks.

In kinetic theory, if $f(v,t)$ evolves under the Boltzmann equation and $M^f$ is the Maxwellian with the same mass, mean velocity, and temperature, then
$$
D_{\mathrm{KL}}(f(t)\|M^f)
=
H(M^f)-H(f(t)).
$$
The Boltzmann $H$-theorem implies that physical entropy increases and the relative entropy to the Maxwellian decreases. KL divergence thus quantifies convergence toward equilibrium [2507.11387].

For stationary trajectories, temporal irreversibility is quantified by comparing a forward block probability with the probability of its reverse:
$$
D_m^X
=
D_{\mathrm{KL}}\bigl[p(x_1^m)\|p(x_m^1)\bigr].
$$
The asymptotic KLD rate is
$$
d^X=\lim_{m\to\infty}\frac{D_m^X}{m}.
$$
The observable rate provides a lower bound on physical entropy production:
$$
\langle\dot S\rangle\geq k\,d^X.
$$
Equality requires complete microscopic information or conditions under which the observed variables determine entropy production. Hidden variables and discarded transition-time information reduce the observable bound [1201.5613].

For a stationary first-order Markov process,
$$
d^X
=
\sum_{x_1,x_2}
p(x_1,x_2)
\log
\frac{p(x_2\mid x_1)}{p(x_1\mid x_2)}.
$$
Two-point statistics then suffice. Non-Markovian observations can remain time-asymmetric even when all one-step probability currents vanish, so higher-order blocks may be necessary.

In cosmology, a domain-averaged KL functional compares an inhomogeneous matter density $\rho$ with its homogeneous domain average:
$$
\frac{S_D}{V_D}
=
\left\langle
\rho\log\frac{\rho}{\langle\rho\rangle_D}
\right\rangle_D.
$$
It vanishes for homogeneous density and measures information lost when the spatial density field is replaced by its average. In the Lemaître–Tolman–Bondi dust model, the leading perturbative contribution is
$$
\frac{S_D}{V_D}
=
\frac{
\langle(\rho^{(1)})^2\rangle
-
\langle\rho^{(1)}\rangle^2
}{
2\rho^{(0)}
}
+\cdots.
$$
The first nonzero term is therefore second order and proportional to the variance of the first-order density perturbation. For spherical LTB domains, the perturbative backreaction vanishes and the domain-averaged KL entropy is proportional to the averaged Weyl curvature squared through second order:
$$
\frac{S_D}{V_D}
=
\frac{9t^2}{256\pi}
\left\langle
C_{\mu\nu\lambda\rho}
C^{\mu\nu\lambda\rho}
\right\rangle_D.
$$
This is an averaged perturbative relation, not a pointwise or fully nonlinear identity [1411.4368].

## 6. Estimation, mixtures, spatial models, and modern applications

KL divergence is often analytically tractable for individual members of a common exponential family but generally lacks a finite closed form for mixture distributions. For
$$
m(x)=\sum_iw_ip_i(x),
\qquad
m'(x)=\sum_jw'_jp'_j(x),
$$
the problematic term is
$$
\int m(x)\log\left(\sum_jw'_jp'_j(x)\right)\,dx.
$$
The logarithm prevents separation of mixture components, and even univariate Gaussian-mixture KL can be mathematically nonanalytic as a function of mixture parameters.

Deterministic mixture bounds can be constructed from piecewise log-sum-exp inequalities. Defining the weighted upper envelope
$$
E(x)=\max_j\{w'_jp'_j(x)\},
$$
one has
$$
E(x)\leq m'(x)\leq k'E(x).
$$
Partitioning the real line into intervals on which one weighted component dominates gives computable lower and upper bounds on cross-entropy, entropy, and KL divergence. The basic additive width is bounded by
$$
\log k+\log k'.
$$
Adaptive bounds replace worst-case $\log k$ terms by local residuals measuring the separation of nondominant components. They are tighter but may require pairwise intersection calculations. These methods are especially effective for univariate Gaussian mixtures, where component intersections reduce to quadratic equations [1606.05850].

For multivariate continuous distributions, nearest-neighbor estimators avoid full density estimation. The Kozachenko–Leonenko entropy estimator is
$$
\widehat H_{N,k}(f)
=
\psi(N)-\psi(k)+\log V_d
+
\frac dN\sum_{i=1}^N\log\rho_{i,k,N}.
$$
Given independent samples from $f$ and $g$, a kNN estimator of forward KL is
$$
\widehat D_{\mathrm{KL}}(f\|g)
=
\frac dN\sum_{i=1}^N
\log\frac{\nu_{i,k}}{\rho_{i,k,N}}
+
\psi(M)-\psi(N-1).
$$
Under continuity, tail, moment, and support conditions, these estimators have consistency and asymptotic convergence properties. A related goodness-of-fit statistic compares the entropy of the moment-matched Gaussian with a kNN estimate of the unknown entropy:
$$
T_{N,k}^{\mathrm{KL}}
=
\frac12\log\!\left((2\pi e)^d\det S_N\right)
-
\widehat H_{N,k}(f).
$$
Because the Gaussian uniquely maximizes differential entropy among distributions with fixed mean and covariance,
$$
D_{\mathrm{KL}}(f\|\phi_{\mu,\Sigma})
=
H\!\left(N(\mu,\Sigma)\right)-H(f).
$$
The statistic converges to zero under multivariate normality and to a positive limit under fixed non-Gaussian alternatives. Parametric bootstrap calibration is used for finite samples [2603.06759].

For pairwise isotropic Gaussian-Markov random fields, local conditional KL expressions reduce to determinants, traces, quadratic forms, and spatial covariance summaries. In the multivariate case, the divergence includes covariance matrices, precision matrices, cross-covariances, and a coupling-modulated Mahalanobis mean term. When the spatial dependence parameters vanish, the expression reduces to the standard multivariate Gaussian KL divergence. The resulting symmetrized divergence can be used for image denoising, texture analysis, multichannel image comparison, and unsupervised metric learning [2203.13164].

In multiscale molecular simulation, KL divergence compares a reference canonical ensemble with an approximate ensemble:
$$
S_{r_s}
=
D_{\mathrm{KL}}(P_\infty\|P_{r_s}).
$$
For potential-energy discrepancy
$$
\Delta U_{r_s}=U_{r_s}-U_\infty,
$$
the exact canonical expression is
$$
S_{r_s}
=
\beta\langle\Delta U_{r_s}\rangle_\infty
+
\log\left\langle e^{-\beta\Delta U_{r_s}}\right\rangle_\infty.
$$
For small discrepancies, the leading approximation is
$$
\breve S_{r_s}
\approx
\frac{\beta^2}{2}
\operatorname{Var}_\infty(\Delta U_{r_s}).
$$
This provides an intensive measure of multiscale error and a workflow for selecting a switching distance in Relative Resolution simulations [2403.09061].

KL divergence also supports information-theoretic causation measures. An intervention that shifts $x_0$ changes the conditional distribution of a future variable $y_\tau$. The local response divergence compares the perturbed and unperturbed conditional laws, while the perturbation divergence measures the information cost of the intervention. Their small-perturbation ratio is an information response:
$$
\Gamma^{x\to y}_\tau
=
\lim_{\epsilon\to0}
\frac{
\langle d^{x\to y}_\tau\rangle
}{
c_x(\epsilon)
}.
$$
The expansion is governed by a ratio of Fisher informations. In linear Gaussian systems, information response is nonlinearly related to transfer entropy:
$$
\Gamma^{x\to y}_\tau
=
e^{2T^{x\to y}_\tau}-1.
$$
This equivalence does not generally persist for nonlinear systems [2102.06839].

Recent diffusion-model analyses use marginal KL entropy production to decompose terminal generation error into initialization, score-approximation, and time-discretization contributions. For Euler–Maruyama sampling, the marginal KL discretization contribution is bounded at order
$$
O(h^2),
$$
under smoothness, ellipticity, and finite-Fisher-information assumptions. This is distinct from path-space KL estimates obtained through Girsanov arguments, which commonly yield order $O(h)$. The decomposition is schematically
$$
\text{terminal KL}
\lesssim
\text{initialization error}
+
\text{score error}
+
\text{discretization error}.
$$
The framework also compares score-based SDEs, probability-flow ODEs, flow matching, and stochastic interpolants through their current velocities and diffusion coefficients [2608.02406].

## 7. Interpretive limits and related divergences

The phrase “Kullback–Leibler entropy” can obscure three distinctions.

First, KL divergence is not Shannon entropy. $H(p)$ measures uncertainty in one law, whereas $D_{\mathrm{KL}}(p\|q)$ measures the discrepancy between an ordered pair. Entropy can be negative in the differential case; KL divergence remains nonnegative when defined.

Second, a signed difference of KL divergences is not itself a divergence:
$$
\Delta_{\mathrm{KL}}(p,q)
=
D_{\mathrm{KL}}(p\|q)
-
D_{\mathrm{KL}}(q\|p).
$$
It can be positive, zero, or negative. Expanding gives
$$
\Delta_{\mathrm{KL}}(p,q)
=
\sum_k(p_k+q_k)\log\frac{p_k}{q_k}.
$$
Its sign reflects a weighted category-wise log-ratio contrast, not support inclusion, coverage, breadth, or generality. Positive and negative category contributions can cancel. After smoothing discrete empirical distributions, all categories may have full support, so literal support comparisons are no longer available [2606.04852].

Third, KL divergence is not universally the most appropriate discrepancy. Wasserstein distances are symmetric metrics under suitable moment assumptions and remain meaningful for singular or disjointly supported distributions. Fourier-based metrics compare characteristic functions but require moment conditions. Energy distances are symmetric, sample-computable, and metric-like under their stated conditions. These alternatives can be preferable in high-dimensional settings where density-ratio estimation is unstable, support mismatch causes infinite KL, or reliable density estimation is difficult [2507.11387].

For unbalanced optimal transport, KL penalties on marginals must be distinguished from entropy regularization of the coupling. For finite measures $\rho$ and $\eta$,
$$
\operatorname{KL}(\rho\mid\eta)
=
\int
\left(
r\log r-r+1
\right)d\eta,
\qquad
r=\frac{d\rho}{d\eta}.
$$
This generalized KL includes mass discrepancy. In Gaussian unbalanced transport with no coupling-entropy term, the optimizer is a mass-scaled quadratic Wasserstein coupling between adjusted Gaussian marginals and can be supported on an affine graph. Adding a coupling-entropy penalty produces a nondegenerate joint plan instead [2605.02497].

KL divergence is therefore best understood as a directional, reference-dependent information functional. Its core properties—nonnegativity, support sensitivity, convexity, data processing, large-deviation rates, variational representations, and local Fisher geometry—explain its central role in inference. The same functional appears as likelihood loss, free-energy excess, entropy-production rate, Bayesian information gain, maximum-entropy projection, model discrepancy, and predictive regret. Its interpretation always depends on the ordered distributions, the reference measure or kernel, the support conditions, and the level of coarse-graining used to represent the underlying system.

Source: https://www.emergentmind.com/topics/kullback-leibler-entropy