Logarithmic Divergence in Theory & Geometry

• Logarithmic divergence is defined as the use of logarithmic forms, notably the Kullback–Leibler divergence, to quantify differences between probability distributions and assess scoring rule regret.
• It underlies the universality of proper loss functions where minimizing expected log-loss guarantees an upper bound on regret across a broad class of convex losses.
• The concept extends to logarithmic L^(α)-divergence in information geometry, robust statistical inference, and scaling laws in optimization, highlighting its practical versatility.

Searching arXiv for the primary and related papers on “logarithmic divergence.” arxiv.search({"query":"ti:\"Bregman Divergence Bounds and Universality Properties of the Logarithmic Loss\" OR (Painsky et al., 2018)","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}) arxiv.search({"query":"\"logarithmic divergences\" information geometry L^alpha divergence Rényi","max_results":10,"sort_by":"relevance","sort_order":"descending"}) Logarithmic divergence is not a single universally fixed object. In the literature surveyed here, the phrase most commonly denotes the Kullback–Leibler divergence—also called relative entropy, information divergence, or the regret of the logarithmic score—but it also denotes a distinct family of logarithmic $L^{(\alpha)}$-divergences in information geometry, and in several areas of physics and harmonic analysis it refers instead to a logarithmic growth law such as $\log(L/\epsilon)$ or divergence of logarithmic means rather than to a divergence functional between probability measures (Harremoës, 2017, Painsky et al., 2018, Wong, 2017, Goginava et al., 2024).

1. Logarithmic divergence as information divergence

In its classical information-theoretic sense, logarithmic divergence is the Kullback–Leibler divergence

$$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$

with the continuous analogue

$D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$

and $D(P\Vert Q)=+\infty$ if $p$ is not absolutely continuous with respect to $q$ (Harremoës, 2017). The same paper identifies information divergence, Kullback–Leibler divergence, relative entropy, and logarithmic divergence as synonymous terminology in this setting, and notes that the logarithmic score $S(Q,x)=-\log Q(x)$ is a proper scoring rule whose Bayes regret is exactly KL divergence (Harremoës, 2017).

For binary prediction, the logarithmic loss is

$$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$

with expected loss

$$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$

and Bayes risk

$\log(L/\epsilon)$0

Its regret is exactly

$\log(L/\epsilon)$1

so minimizing expected log-loss is equivalent to minimizing KL divergence between $\log(L/\epsilon)$2 and $\log(L/\epsilon)$3 (Painsky et al., 2018).

The same binary framework places logarithmic divergence inside the general theory of proper losses and Bregman divergences. For a smooth proper loss with generalized entropy $\log(L/\epsilon)$4, Savage’s representation gives

$\log(L/\epsilon)$5

and the regret is the Bregman divergence generated by $\log(L/\epsilon)$6,

$\log(L/\epsilon)$7

For log-loss, the weight function is $\log(L/\epsilon)$8, and the associated Bregman divergence is KL itself (Painsky et al., 2018).

2. Universality and structural uniqueness

A central universality result states that, for binary classification, every admissible loss $\log(L/\epsilon)$9 that is strictly proper, fair, regular, convex in $$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$0, and $$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$1 satisfies

$$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$2

for all $$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$3 (Painsky et al., 2018). This means that minimizing log-loss minimizes an upper bound on the regret for any smooth proper convex loss in that admissible class. The paper develops the same theme on arbitrary finite alphabets through Hessian-dominance conditions for general Bregman generators and through separable generators $$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$4, yielding inequalities of the form

$$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$5

and, in the separable case,

$$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$6

under the stated regularity assumptions (Painsky et al., 2018).

The local version of this universality is expressed through Fisher information. For $$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$7,

$$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$8

and every admissible loss-induced divergence is bounded above, up to the constant $$D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)},$$9, by the same Fisher-information curvature $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$0 (Painsky et al., 2018). This identifies logarithmic divergence as the second-order control quantity for a large class of proper losses.

A stronger structural characterization is obtained from convex optimization and information geometry. For regret functions $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$1 on the state space of a finite-dimensional $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$2-algebra with at least three orthogonal states, the following are equivalent: $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$3 is entropy times a negative constant plus an affine term; $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$4 is proportional to information divergence; $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$5 is monotone; $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$6 satisfies sufficiency; and $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$7 is local (Harremoës, 2017). In this sense, logarithmic divergence is uniquely selected, up to a positive scalar, by the joint requirements of sufficiency, locality, and monotonicity.

This equivalence also clarifies when KL does not arise. The same source explicitly notes that piecewise linear coding objectives, quadratic scores such as the Brier score, and the Itakura–Saito distance on $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$8 lead to other divergences because the sufficiency/locality/monotonicity hypotheses fail or the domain changes (Harremoës, 2017). A common misconception is therefore that every optimization-based regret must be logarithmic divergence; the cited results make the dependence on structural assumptions explicit.

3. The logarithmic $D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx,$9-divergence in information geometry

A distinct usage of the term appears in information geometry and optimal transport. For $D(P\Vert Q)=+\infty$0 and a differentiable $D(P\Vert Q)=+\infty$1-exponentially concave potential $D(P\Vert Q)=+\infty$2, the logarithmic $D(P\Vert Q)=+\infty$3-divergence is

$D(P\Vert Q)=+\infty$4

while the logarithmic transport cost is

$D(P\Vert Q)=+\infty$5

(Wong, 2017). As $D(P\Vert Q)=+\infty$6, $D(P\Vert Q)=+\infty$7 converges to the Bregman divergence, since $D(P\Vert Q)=+\infty$8 (Wong, 2017).

This divergence admits an $D(P\Vert Q)=+\infty$9-gradient

$p$0

an $p$1-conjugate potential $p$2, and a self-dual representation

$p$3

(Wong, 2017). On the corresponding generalized exponential families $p$4 and $p$5, the same divergence coincides exactly with Rényi divergences of orders $p$6 and $p$7, respectively (Wong, 2017).

The induced geometry is not dually flat. Its metric in primal coordinates is

$p$8

and the manifold is dually projectively flat with constant sectional curvature $p$9 for $q$0, and $q$1 for $q$2 (Wong, 2017). A companion geometric treatment shows that the same object is monotone-equivalent to a conformal Bregman divergence, is equivalent via an affine immersion to Kurose’s geometric divergence, and serves as the canonical divergence of a statistical manifold with constant sectional curvature $q$3 (Wong et al., 2019).

The projection theory of this logarithmic divergence extends Amari’s dually flat construction. A dual foliation theorem gives orthogonal decompositions into primal and dual autoparallel submanifolds, and an $q$4-PCA problem is formulated by minimizing

$q$5

over $q$6-dimensional affine subspaces in primal coordinates (Tao et al., 2021). In continuous-time optimization, the same geometry yields conformal mirror descent, with logarithmic cost

$q$7

metric

$q$8

and dynamics that are a time change of a Hessian gradient flow (Kainth et al., 2022).

4. Logarithmic divergence families in robust statistical inference

Another branch of the literature uses logarithmic transforms of divergence functionals to construct robust inference procedures. The Logarithmic Super Divergence is defined, for densities $q$9, by

$S(Q,x)=-\log Q(x)$0

where $S(Q,x)=-\log Q(x)$1 and $S(Q,x)=-\log Q(x)$2 (Maji et al., 2014). It contains the Logarithmic Power Divergence at $S(Q,x)=-\log Q(x)$3, the Logarithmic Density Power Divergence at $S(Q,x)=-\log Q(x)$4, and reduces to the likelihood disparity, a KL form, at $S(Q,x)=-\log Q(x)$5 (Maji et al., 2014).

For minimum-LSD estimation, the model influence function at $S(Q,x)=-\log Q(x)$6 is independent of $S(Q,x)=-\log Q(x)$7, is unbounded at $S(Q,x)=-\log Q(x)$8, and is bounded and redescending for $S(Q,x)=-\log Q(x)$9 (Maji et al., 2014). The discrete asymptotic theory shows consistency and asymptotic normality of the minimum-LSD estimator, with asymptotic variance depending on $$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$0 but not on $$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$1, while empirical robustness away from the model is still strongly affected by $l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$2 (Maji et al., 2014).

A broader characterization is given by the Functional Density Power Divergence class

$$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$3

For fixed $$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$4, this is a valid divergence if and only if $$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$5 is convex and strictly increasing on its domain (Ray et al., 2021). This recovers the DPD with $$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$6 and the LDPD with $$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$7, and it explains why the logarithmic transform is compatible with divergence structure despite the concavity of $$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$8 in its original variable (Ray et al., 2021).

Within the Bregman framework, the logarithmic construction is sharply limited. The characterization of logarithmic Bregman functions shows that, up to affine terms, the only strictly convex Bregman generators that yield a meaningful logarithmic piecewise transform are power generators $$l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q},$$9, so the only resulting logarithmic Bregman divergences are positive multiples of the LDPD family (Ray et al., 2021). This circumscribes the search for new “logarithmic” Bregman-type divergences.

5. Optimization, scaling laws, and generalized logarithms

Logarithmic divergence also appears as an operational quantity in coding and optimization. For fixed-length distribution matching over a finite alphabet with target $$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$0, the optimal informational divergence obeys

$$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$1

and the coding rate satisfies

$$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$2

(Kramer, 2021). This shows that logarithmic growth is an unavoidable asymptotic penalty for optimal fixed-length invertible distribution matching, and that threshold-type codebooks outperform constant-composition schemes when the alphabet size exceeds two (Kramer, 2021).

For model geometry, maximizing information divergence from linear and toric statistical models is studied through logarithmic Voronoi polytopes. In linear models, the global maximum $$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$3 is attained at the boundary of the simplex, more precisely at a vertex of a logarithmic Voronoi polytope attached to a vertex of the model (Alexandr et al., 2023). In toric models, local maximizers are characterized as projection points, and the paper combines chamber complexes with numerical algebraic geometry to enumerate complementary vertex–face pairs and compute global maxima (Alexandr et al., 2023).

A further generalization replaces $$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$4 by a two-parameter deformed logarithm

$$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$5

with $$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$6 or $$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$7 (Lantéri, 2023). This form contains the natural logarithm, Tsallis $$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$8-logarithm, Kaniadakis $$L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q},$$9-logarithm, and Abe logarithm as special cases. The same paper applies $\log(L/\epsilon)$00 blockwise to classical divergences, derives gradients and $\log(L/\epsilon)$01 decompositions, and constructs additive, pseudo-multiplicative, and multiplicative algorithms for linear inverse problems under nonnegativity and scale-invariance constraints (Lantéri, 2023).

6. Other technical meanings of the phrase

The phrase “logarithmic divergence” is also used for logarithmic behavior rather than for a divergence functional. In cosmological large-scale structure, a logarithmic density variable

$\log(L/\epsilon)$02

is used to straighten the relation between overdensity and displacement divergence. Starting from the continuity equation, one obtains

$\log(L/\epsilon)$03

or equivalently $\log(L/\epsilon)$04 with $\log(L/\epsilon)$05 (Falck et al., 2011). The same source explicitly states that this usage is unrelated to “logarithmic divergence” in quantum field theory or renormalization (Falck et al., 2011).

In conformal field theory and entanglement theory, logarithmic divergence often denotes a universal $\log(L/\epsilon)$06-scaling term. For OPE blocks in two-dimensional CFT, the connected correlators obey

$\log(L/\epsilon)$07

with cutoff-independent coefficients related to conformal-block amplitudes by a UV/IR relation (Long, 2020). For entanglement entropy of a scalar field in BTZ spacetime, the logarithmic term is universal and tied to the conformal anomaly; in the dimensionally reduced setup reviewed there, the coefficient $\log(L/\epsilon)$08 yields

$\log(L/\epsilon)$09

for the universal logarithmic contribution (Singh et al., 2014). By contrast, in a CTMRG study of the square-lattice hard-rod model, no logarithmic divergence of von Neumann or Rényi entropies is observed at the reported transitions, in contrast with Potts benchmarks in the same geometry (Chatelain et al., 2020).

Harmonic analysis uses the phrase yet differently. For Fourier series, logarithmic means are the Nörlund means

$\log(L/\epsilon)$10

and the paper on general orthonormal systems proves a transfer principle

$\log(L/\epsilon)$11

which yields everywhere or positive-measure divergence results for trigonometric, Walsh–Paley, and bounded orthonormal systems (Goginava et al., 2024). Here again, “logarithmic divergence” refers to divergence of logarithmic means, not to KL-type information geometry.

7. Conceptual synthesis

Across these literatures, two principal meanings dominate. The first is the classical information-theoretic one: logarithmic divergence as KL divergence, the regret of logarithmic scoring, the Bregman divergence of negative Shannon entropy, and the universal upper bound for broad classes of proper convex losses (Harremoës, 2017, Painsky et al., 2018). The second is geometric: logarithmic $\log(L/\epsilon)$12-divergence as a logarithmic deformation of Bregman divergence associated with the cost $\log(L/\epsilon)$13, constant-curvature statistical manifolds, and generalized exponential families tied to Rényi geometry (Wong, 2017, Wong et al., 2019).

A recurring structural theme is that logarithmic constructions are unusually rigid. Sufficiency, locality, and monotonicity force regret to be proportional to information divergence in one framework (Harremoës, 2017); only power generators survive as logarithmic Bregman functions in another (Ray et al., 2021); and logarithmic loss dominates admissible proper convex losses up to an explicit normalization constant in yet another (Painsky et al., 2018). This suggests a broad unifying principle, though it remains an interpretation rather than a theorem across all cited domains: logarithmic divergence repeatedly appears where optimization, duality, and invariance impose unusually strong structural constraints.

At the same time, the surveyed literature makes it essential to distinguish divergence functionals from logarithmic singularities or growth laws. In CFT, black-hole entropy, phase transitions, Fourier analysis, and cosmological reconstruction, the same phrase often names a $\log(L/\epsilon)$14-scaling phenomenon rather than a statistical discrepancy measure (Long, 2020, Singh et al., 2014, Goginava et al., 2024, Falck et al., 2011). Any technical use of the term therefore depends decisively on context.