Kullback–Leibler Divergence (Relative Entropy)

• Kullback–Leibler divergence is a nonnegative, asymmetric measure of discrepancy between two probability distributions that becomes infinite when the reference assigns zero probability to an observed event.
• It supports maximum-likelihood estimation, hypothesis testing, Bayesian updating, maximum-entropy reconstruction, and machine-learning objectives by quantifying excess logarithmic loss or information gain.
• Its data-processing inequality makes it decrease under coarse-graining, while applications in thermodynamics, kinetic theory, diffusion models, and molecular simulation interpret it as free-energy excess, irreversibility, or model error.

Kullback–Leibler entropy is an informal name for the Kullback–Leibler (KL) divergence, also called relative entropy or information divergence. For probability measures $P$ and $Q$ on a common measurable space, with $P\ll Q$, it is defined by

$$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$

For densities $p$ and $q$ relative to a common dominating measure,

$$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$

The first argument is the distribution being averaged over, conventionally interpreted as the empirical, true, or observed distribution; the second is the reference, theoretical, or model distribution. KL divergence is nonnegative, vanishes exactly when the two distributions coincide almost everywhere, and is generally asymmetric. Strictly, Shannon entropy is a one-distribution functional, whereas KL divergence compares two distributions. This distinction is central to statistical inference, information theory, kinetic theory, Bayesian analysis, stochastic dynamics, machine learning, and multiscale modeling.

1. Definition, terminology, and fundamental properties

For a finite distribution $p=(p_1,\ldots,p_m)$ and reference distribution $q=(q_1,\ldots,q_m)$,

$$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$

The usual conventions are

$Q$0

Consequently,

$Q$1

meaning that every event assigned zero probability by $Q$2 is also assigned zero probability by $Q$3. If $Q$4 and $Q$5, the divergence is infinite. This is the mathematical basis for interpreting a zero-probability model prediction contradicted by an observation as strict falsification (0808.4111).

KL divergence satisfies Gibbs’ inequality:

$Q$6

with equality if and only if $Q$7. It is not a metric: generally,

$Q$8

and it does not satisfy the triangle inequality. The asymmetry is not a defect of notation but expresses the distinct inferential roles of data and model. Reversing the arguments changes both the weighting distribution and the support condition.

Using natural logarithms gives units of nats; logarithms to base $Q$9 give bits. KL divergence is invariant under a common invertible change of variables because the Jacobian factors cancel in the density ratio. This contrasts with differential Shannon entropy, which changes under rescaling.

The divergence decomposes into cross-entropy and Shannon entropy:

$P\ll Q$0

where

$P\ll Q$1

Thus KL divergence is the excess expected logarithmic loss or coding cost incurred when data generated by $P\ll Q$2 are encoded using $P\ll Q$3. Shannon entropy measures uncertainty within one distribution; cross-entropy measures the average code length under another; KL divergence measures their difference.

The divergence is jointly convex:

$P\ll Q$4

It is also convex in either argument separately under the usual domain conditions. For a measurable transformation or Markov kernel $P\ll Q$5, the data-processing inequality states

$P\ll Q$6

Coarse-graining, marginalization, or stochastic processing therefore cannot increase distinguishability. In finite spaces, aggregating categories into super-categories $P\ll Q$7 and $P\ll Q$8 gives

$P\ll Q$9

2. Statistical inference, likelihood, and information geometry

Let $$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$0 be independent observations with $$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$1 possible outcomes. If $$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$2 observations fall in category $$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$3, the empirical distribution or type is

$$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$4

For a model distribution $$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$5, the likelihood of the ordered sample is

$$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$6

Since $$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$7 is fixed for observed data, maximizing likelihood is equivalent to minimizing

$$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$8

Thus maximum likelihood is a forward-KL projection of the empirical distribution onto the model family.

The probability of observing a type includes the multiplicity of ordered sequences having that type:

$$D_{\mathrm{KL}}(P\|Q) = \int \log\!\left(\frac{dP}{dQ}\right)\,dP.$$9

Stirling’s approximation yields the large-deviation relation

$p$0

where $p$1 means

$p$2

KL divergence is therefore the exponential rate at which an empirical distribution becomes unlikely under a model.

For a single hypothesis $p$3, the statistic

$p$4

is asymptotically chi-square distributed with $p$5 degrees of freedom. The model is rejected at significance level $p$6 when

$p$7

For a model family $p$8, the maximum-likelihood projection is

$p$9

and the residual statistic is

$q$0

Its asymptotic degrees of freedom are

$q$1

For nested families $q$2, the likelihood-ratio statistic becomes

$q$3

In exponential families, the difference admits the Pythagorean form

$q$4

For two simple hypotheses $q$5 and $q$6, the Neyman–Pearson rejection region can be written asymptotically in terms of a difference of KL divergences:

$q$7

The optimal equal-prior error exponent is the Chernoff information,

$q$8

Near equality of $q$9 and $$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$0, KL divergence has a quadratic expansion related to Pearson’s chi-square discrepancy:

$$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$1

This local quadratic structure is the information-geometric origin of the Fisher information metric. For discrete normal distributions, KL divergence is a reverse Bregman divergence in natural coordinates:

$$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$2

where $$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$3 is the log-normalizing function associated with the Riemann theta function (Nielsen, 2021).

Unrestricted maximum likelihood gives

$$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$4

Although this minimizes empirical forward KL exactly, it can overfit sampling noise. Restricted model families, likelihood-ratio tests, complexity penalties, Bayesian priors, maximum-entropy reconstruction, and Markov-order restrictions address this limitation.

3. Relation to entropy families and variational principles

KL divergence is the order-one member of the Rényi-divergence family. For $$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$5, $$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$6,

$$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$7

The limit $$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$8 is

$$D_{\mathrm{KL}}(p\|q) = \int p(x)\log\frac{p(x)}{q(x)}\,dx.$$9

Rényi divergence is nonnegative, obeys data processing, is monotone in $p=(p_1,\ldots,p_m)$0 under the stated finiteness conditions, and is additive for product measures. In particular,

$p=(p_1,\ldots,p_m)$1

The order-$p=(p_1,\ldots,p_m)$2 and order-$p=(p_1,\ldots,p_m)$3 cases connect KL divergence to Hellinger and $p=(p_1,\ldots,p_m)$4 distances:

$p=(p_1,\ldots,p_m)$5

and

$p=(p_1,\ldots,p_m)$6

Consequently,

$p=(p_1,\ldots,p_m)$7

(Erven et al., 2010).

Shannon entropy can be recovered by selecting a reference measure. For a discrete probability mass function $p=(p_1,\ldots,p_m)$8, taking $p=(p_1,\ldots,p_m)$9 to be counting measure gives

$q=(q_1,\ldots,q_m)$0

so

$q=(q_1,\ldots,q_m)$1

Taking $q=(q_1,\ldots,q_m)$2 to be the uniform probability on $q=(q_1,\ldots,q_m)$3 points gives

$q=(q_1,\ldots,q_m)$4

Thus entropy is not itself a two-distribution divergence; it is obtained from relative entropy after specifying a reference measure.

A variational representation is

$q=(q_1,\ldots,q_m)$5

under suitable integrability conditions. A categorical characterization shows that, on finite statistical systems, KL divergence is uniquely determined up to a nonnegative scalar by functorial additivity under composition, convex linearity, lower semicontinuity, and vanishing for optimal Bayesian inverses (Baez et al., 2014). In that framework, a deterministic map is a measurement, a stochastic right inverse is a hypothesis about the preimage state, and relative entropy quantifies the information gained when the induced prior is updated to the true distribution.

If a probability kernel $q=(q_1,\ldots,q_m)$6 supplies a reference distribution on each fiber of $q=(q_1,\ldots,q_m)$7, then the generalized entropy

$q=(q_1,\ldots,q_m)$8

is the negative KL divergence between the actual joint law $q=(q_1,\ldots,q_m)$9 and the law obtained by retaining its $$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$0-marginal $$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$1 while replacing its conditional distribution on $$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$2 by $$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$3. Choosing $$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$4 yields mutual information:

$$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$5

This conditional-kernel formulation extends entropy and information gain to general measurable spaces (Lopes et al., 2020).

4. Maximum entropy, Bayesian updating, and reconstruction

For incomplete observations, KL projection provides a principled reconstruction mechanism. If the compatible empirical distributions form a set $$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$6, the most probable reconstruction under a model $$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$7 is

$$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$8

Sanov’s theorem gives

$$D_{\mathrm{KL}}(p\|q) = \sum_{j=1}^m p_j\log\frac{p_j}{q_j}.$$9

If $Q$00 is uniform, then

$Q$01

so minimizing relative entropy is equivalent to maximizing Shannon entropy. Maximum entropy is therefore a prior-dependent information projection, not a reconstruction generated without assumptions.

For a linear constraint

$Q$02

the KL projection has exponential-family form

$Q$03

For energy constraints and a uniform prior, this becomes the Boltzmann–Gibbs distribution

$Q$04

The same construction produces gravity-type models under constraints on distances, costs, origin profiles, destination profiles, stayers, or balanced flows (0808.4111).

For a Dirichlet prior $Q$05 with $Q$06, the prior has the asymptotic form

$Q$07

After observing counts, the posterior remains Dirichlet with parameters $Q$08, and

$Q$09

The prior strength $Q$10 consequently acts in effective sample-size units.

Pólya Tree priors exploit a multiplicative binary partition structure. The logarithm of a Pólya Tree density becomes an additive series of independent beta-branching contributions, yielding a representation of the form

$Q$11

Under summability conditions on the beta parameters, posterior expected KL divergence converges to zero, and posterior mass also concentrates in total variation. A related posterior estimator of differential entropy is consistent in probability for compactly supported vectors under the stated regularity assumptions. In one dimension, truncation at depth approximately

$Q$12

is theoretically sufficient for retaining the data-relevant part of the tree (Corrêa et al., 3 Apr 2025).

5. Dynamical, thermodynamic, and physical interpretations

KL divergence frequently acts as a Lyapunov functional when one distribution evolves toward a reference equilibrium. For finite-state Markov dynamics,

$Q$13

where $Q$14 is infinitesimal stochastic. If $Q$15 and $Q$16 evolve under the same Markov process, then

$Q$17

If $Q$18 is stationary, $Q$19, then both

$Q$20

have corresponding monotonicity results under the conditions considered. Detailed balance is not required for the general contraction theorem (Baez et al., 2015).

When $Q$21 is a Boltzmann distribution,

$Q$22

the free-energy difference satisfies

$Q$23

under the paper’s convention for $Q$24. KL divergence is therefore a nonequilibrium free-energy excess, and its decrease corresponds to relaxation toward equilibrium.

For reaction networks, populations need not be normalized. The appropriate generalized divergence is

$Q$25

For a complex-balanced equilibrium $Q$26, mass-action dynamics satisfies

$Q$27

This extends the Lyapunov interpretation from normalized Markov distributions to nonlinear chemical reaction networks.

In kinetic theory, if $Q$28 evolves under the Boltzmann equation and $Q$29 is the Maxwellian with the same mass, mean velocity, and temperature, then

$Q$30

The Boltzmann $Q$31-theorem implies that physical entropy increases and the relative entropy to the Maxwellian decreases. KL divergence thus quantifies convergence toward equilibrium (Auricchio et al., 15 Jul 2025).

For stationary trajectories, temporal irreversibility is quantified by comparing a forward block probability with the probability of its reverse:

$Q$32

The asymptotic KLD rate is

$Q$33

The observable rate provides a lower bound on physical entropy production:

$Q$34

Equality requires complete microscopic information or conditions under which the observed variables determine entropy production. Hidden variables and discarded transition-time information reduce the observable bound (Roldán et al., 2012).

For a stationary first-order Markov process,

$Q$35

Two-point statistics then suffice. Non-Markovian observations can remain time-asymmetric even when all one-step probability currents vanish, so higher-order blocks may be necessary.

In cosmology, a domain-averaged KL functional compares an inhomogeneous matter density $Q$36 with its homogeneous domain average:

$Q$37

It vanishes for homogeneous density and measures information lost when the spatial density field is replaced by its average. In the Lemaître–Tolman–Bondi dust model, the leading perturbative contribution is

$Q$38

The first nonzero term is therefore second order and proportional to the variance of the first-order density perturbation. For spherical LTB domains, the perturbative backreaction vanishes and the domain-averaged KL entropy is proportional to the averaged Weyl curvature squared through second order:

$Q$39

This is an averaged perturbative relation, not a pointwise or fully nonlinear identity (Li et al., 2014).

6. Estimation, mixtures, spatial models, and modern applications

KL divergence is often analytically tractable for individual members of a common exponential family but generally lacks a finite closed form for mixture distributions. For

$Q$40

the problematic term is

$Q$41

The logarithm prevents separation of mixture components, and even univariate Gaussian-mixture KL can be mathematically nonanalytic as a function of mixture parameters.

Deterministic mixture bounds can be constructed from piecewise log-sum-exp inequalities. Defining the weighted upper envelope

$Q$42

one has

$Q$43

Partitioning the real line into intervals on which one weighted component dominates gives computable lower and upper bounds on cross-entropy, entropy, and KL divergence. The basic additive width is bounded by

$Q$44

Adaptive bounds replace worst-case $Q$45 terms by local residuals measuring the separation of nondominant components. They are tighter but may require pairwise intersection calculations. These methods are especially effective for univariate Gaussian mixtures, where component intersections reduce to quadratic equations (Nielsen et al., 2016).

For multivariate continuous distributions, nearest-neighbor estimators avoid full density estimation. The Kozachenko–Leonenko entropy estimator is

$Q$46

Given independent samples from $Q$47 and $Q$48, a kNN estimator of forward KL is

$Q$49

Under continuity, tail, moment, and support conditions, these estimators have consistency and asymptotic convergence properties. A related goodness-of-fit statistic compares the entropy of the moment-matched Gaussian with a kNN estimate of the unknown entropy:

$Q$50

Because the Gaussian uniquely maximizes differential entropy among distributions with fixed mean and covariance,

$Q$51

The statistic converges to zero under multivariate normality and to a positive limit under fixed non-Gaussian alternatives. Parametric bootstrap calibration is used for finite samples (Cadirci et al., 6 Mar 2026).

For pairwise isotropic Gaussian-Markov random fields, local conditional KL expressions reduce to determinants, traces, quadratic forms, and spatial covariance summaries. In the multivariate case, the divergence includes covariance matrices, precision matrices, cross-covariances, and a coupling-modulated Mahalanobis mean term. When the spatial dependence parameters vanish, the expression reduces to the standard multivariate Gaussian KL divergence. The resulting symmetrized divergence can be used for image denoising, texture analysis, multichannel image comparison, and unsupervised metric learning (Levada, 2022).

In multiscale molecular simulation, KL divergence compares a reference canonical ensemble with an approximate ensemble:

$Q$52

For potential-energy discrepancy

$Q$53

the exact canonical expression is

$Q$54

For small discrepancies, the leading approximation is

$Q$55

This provides an intensive measure of multiscale error and a workflow for selecting a switching distance in Relative Resolution simulations (Chaimovich et al., 2024).

KL divergence also supports information-theoretic causation measures. An intervention that shifts $Q$56 changes the conditional distribution of a future variable $Q$57. The local response divergence compares the perturbed and unperturbed conditional laws, while the perturbation divergence measures the information cost of the intervention. Their small-perturbation ratio is an information response:

$Q$58

The expansion is governed by a ratio of Fisher informations. In linear Gaussian systems, information response is nonlinearly related to transfer entropy:

$Q$59

This equivalence does not generally persist for nonlinear systems (Auconi et al., 2021).

Recent diffusion-model analyses use marginal KL entropy production to decompose terminal generation error into initialization, score-approximation, and time-discretization contributions. For Euler–Maruyama sampling, the marginal KL discretization contribution is bounded at order

$Q$60

under smoothness, ellipticity, and finite-Fisher-information assumptions. This is distinct from path-space KL estimates obtained through Girsanov arguments, which commonly yield order $Q$61. The decomposition is schematically

$Q$62

The framework also compares score-based SDEs, probability-flow ODEs, flow matching, and stochastic interpolants through their current velocities and diffusion coefficients (Wu et al., 3 Aug 2026).

7. Interpretive limits and related divergences

The phrase “Kullback–Leibler entropy” can obscure three distinctions.

First, KL divergence is not Shannon entropy. $Q$63 measures uncertainty in one law, whereas $Q$64 measures the discrepancy between an ordered pair. Entropy can be negative in the differential case; KL divergence remains nonnegative when defined.

Second, a signed difference of KL divergences is not itself a divergence:

$Q$65

It can be positive, zero, or negative. Expanding gives

$Q$66

Its sign reflects a weighted category-wise log-ratio contrast, not support inclusion, coverage, breadth, or generality. Positive and negative category contributions can cancel. After smoothing discrete empirical distributions, all categories may have full support, so literal support comparisons are no longer available (Osaki, 3 Jun 2026).

Third, KL divergence is not universally the most appropriate discrepancy. Wasserstein distances are symmetric metrics under suitable moment assumptions and remain meaningful for singular or disjointly supported distributions. Fourier-based metrics compare characteristic functions but require moment conditions. Energy distances are symmetric, sample-computable, and metric-like under their stated conditions. These alternatives can be preferable in high-dimensional settings where density-ratio estimation is unstable, support mismatch causes infinite KL, or reliable density estimation is difficult (Auricchio et al., 15 Jul 2025).

For unbalanced optimal transport, KL penalties on marginals must be distinguished from entropy regularization of the coupling. For finite measures $Q$67 and $Q$68,

$Q$69

This generalized KL includes mass discrepancy. In Gaussian unbalanced transport with no coupling-entropy term, the optimizer is a mass-scaled quadratic Wasserstein coupling between adjusted Gaussian marginals and can be supported on an affine graph. Adding a coupling-entropy penalty produces a nondegenerate joint plan instead (Yang et al., 4 May 2026).

KL divergence is therefore best understood as a directional, reference-dependent information functional. Its core properties—nonnegativity, support sensitivity, convexity, data processing, large-deviation rates, variational representations, and local Fisher geometry—explain its central role in inference. The same functional appears as likelihood loss, free-energy excess, entropy-production rate, Bayesian information gain, maximum-entropy projection, model discrepancy, and predictive regret. Its interpretation always depends on the ordered distributions, the reference measure or kernel, the support conditions, and the level of coarse-graining used to represent the underlying system.