--- title: Logarithmic Divergence in Theory & Geometry url: https://www.emergentmind.com/topics/logarithmic-divergence type: topic --- # Logarithmic Divergence in Theory & Geometry Searching arXiv for the primary and related papers on “logarithmic divergence.” arxiv.search({"query":"ti:\"Bregman Divergence Bounds and Universality Properties of the Logarithmic Loss\" OR 1810.07014","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}) arxiv.search({"query":"\"logarithmic divergences\" information geometry L^(alpha) divergence Rényi","max_results":10,"sort_by":"relevance","sort_order":"descending"}) Logarithmic divergence is not a single universally fixed object. In the literature surveyed here, the phrase most commonly denotes the Kullback–Leibler divergence—also called relative entropy, information divergence, or the regret of the logarithmic score—but it also denotes a distinct family of logarithmic \(L^{(\alpha)}\)-divergences in information geometry, and in several areas of physics and harmonic analysis it refers instead to a logarithmic growth law such as \(\log(L/\epsilon)\) or divergence of logarithmic means rather than to a divergence functional between probability measures [1701.01010] [1810.07014] [1712.03610] [2401.11003]. ## 1. Logarithmic divergence as information divergence In its classical information-theoretic sense, logarithmic divergence is the Kullback–Leibler divergence \[ D_{\mathrm{KL}}(P\Vert Q)=\sum_{x} P(x)\log\frac{P(x)}{Q(x)}, \] with the continuous analogue \[ D(P\Vert Q)=\int p(x)\log\frac{p(x)}{q(x)}\,dx, \] and \(D(P\Vert Q)=+\infty\) if \(p\) is not absolutely continuous with respect to \(q\) [1701.01010]. The same paper identifies information divergence, Kullback–Leibler divergence, relative entropy, and logarithmic divergence as synonymous terminology in this setting, and notes that the logarithmic score \(S(Q,x)=-\log Q(x)\) is a proper scoring rule whose Bayes regret is exactly KL divergence [1701.01010]. For binary prediction, the logarithmic loss is \[ l_{\log}(y,q)= y\log\frac{1}{q}+(1-y)\log\frac{1}{1-q}, \] with expected loss \[ L_{\log}(p,q)=(1-p)\log\frac{1}{1-q}+p\log\frac{1}{q}, \] and Bayes risk \[ G_{\log}(p)=p\log\frac{1}{p}+(1-p)\log\frac{1}{1-p}. \] Its regret is exactly \[ \Delta L_{\log}(p,q)=D_{\mathrm{KL}}(p\Vert q) = p\log\frac{p}{q}+(1-p)\log\frac{1-p}{1-q}, \] so minimizing expected log-loss is equivalent to minimizing KL divergence between \(\mathrm{Bern}(p)\) and \(\mathrm{Bern}(q)\) [1810.07014]. The same binary framework places logarithmic divergence inside the general theory of proper losses and Bregman divergences. For a smooth proper loss with generalized entropy \(G(p)=L(p,p)\), Savage’s representation gives \[ L(p,q)=G(q)+(p-q)G'(q), \] and the regret is the Bregman divergence generated by \(-G\), \[ \Delta L(p,q)=D_{-G}(p\Vert q). \] For log-loss, the weight function is \(w_{\mathrm{KL}}(q)=1/[q(1-q)]\), and the associated Bregman divergence is KL itself [1810.07014]. ## 2. Universality and structural uniqueness A central universality result states that, for binary classification, every admissible loss \(l(y,q)\) that is strictly proper, fair, regular, convex in \(q\), and \(C^3\) satisfies \[ D_{\mathrm{KL}}(p\Vert q)\ge \frac{1}{C(G)}\,D_{-G}(p\Vert q), \qquad C(G)>-\frac{1}{2}G''\!\left(\tfrac{1}{2}\right), \] for all \(p,q\in[0,1]\) [1810.07014]. This means that minimizing log-loss minimizes an upper bound on the regret for any smooth proper convex loss in that admissible class. The paper develops the same theme on arbitrary finite alphabets through Hessian-dominance conditions for general Bregman generators and through separable generators \(g\), yielding inequalities of the form \[ \tilde D_{\mathrm{KL}}(p^m\Vert q^m)\ge \frac{1}{C(f)}D_f(p^m\Vert q^m), \] and, in the separable case, \[ \tilde D_{\mathrm{KL}}(p^m\Vert q^m)\ge \frac{1}{C(g)}D_g(p^m\Vert q^m) \] under the stated regularity assumptions [1810.07014]. The local version of this universality is expressed through Fisher information. For \(q=p+dp\), \[ D_{\mathrm{KL}}(p\Vert p+dp)=\frac{dp^2}{2}\frac{1}{p(1-p)}+o(dp^2), \] and every admissible loss-induced divergence is bounded above, up to the constant \(C(G)\), by the same Fisher-information curvature \(J(p)=1/[p(1-p)]\) [1810.07014]. This identifies logarithmic divergence as the second-order control quantity for a large class of proper losses. A stronger structural characterization is obtained from convex optimization and information geometry. For regret functions \(D_F\) on the state space of a finite-dimensional \(C^*\)-algebra with at least three orthogonal states, the following are equivalent: \(F\) is entropy times a negative constant plus an affine term; \(D_F\) is proportional to information divergence; \(D_F\) is monotone; \(D_F\) satisfies sufficiency; and \(D_F\) is local [1701.01010]. In this sense, logarithmic divergence is uniquely selected, up to a positive scalar, by the joint requirements of sufficiency, locality, and monotonicity. This equivalence also clarifies when KL does *not* arise. The same source explicitly notes that piecewise linear coding objectives, quadratic scores such as the Brier score, and the Itakura–Saito distance on \([0,\infty)\) lead to other divergences because the sufficiency/locality/monotonicity hypotheses fail or the domain changes [1701.01010]. A common misconception is therefore that every optimization-based regret must be logarithmic divergence; the cited results make the dependence on structural assumptions explicit. ## 3. The logarithmic \(L^{(\alpha)}\)-divergence in information geometry A distinct usage of the term appears in information geometry and optimal transport. For \(\alpha>0\) and a differentiable \(\alpha\)-exponentially concave potential \(\phi\), the logarithmic \(L^{(\alpha)}\)-divergence is \[ D^{(\alpha)}[\xi:\xi'] = \frac{1}{\alpha}\log\bigl(1+\alpha \nabla\phi(\xi')\cdot(\xi-\xi')\bigr) - \bigl(\phi(\xi)-\phi(\xi')\bigr), \] while the logarithmic transport cost is \[ c^{(\alpha)}(x,y)=\frac{1}{\alpha}\log(1+\alpha x\cdot y) \] [1712.03610]. As \(\alpha\to 0^+\), \(D^{(\alpha)}\) converges to the Bregman divergence, since \((1/\alpha)\log(1+\alpha u)\to u\) [1712.03610]. This divergence admits an \(\alpha\)-gradient \[ D^{(\alpha)}\phi(\xi)=\frac{\nabla\phi(\xi)}{1-\alpha \nabla\phi(\xi)\cdot \xi}, \] an \(\alpha\)-conjugate potential \(\psi\), and a self-dual representation \[ D^{(\alpha)}[\xi:\xi'] = c^{(\alpha)}(\xi,\eta')-\phi(\xi)-\psi(\eta'), \qquad \eta'=D^{(\alpha)}\phi(\xi') \] [1712.03610]. On the corresponding generalized exponential families \(\mathcal F^{(\alpha)}\) and \(\mathcal F^{(-\alpha)}\), the same divergence coincides exactly with Rényi divergences of orders \(1+\alpha\) and \(1-\alpha\), respectively [1712.03610]. The induced geometry is not dually flat. Its metric in primal coordinates is \[ g_{ij}(\xi) = -\frac{\partial^2\phi}{\partial \xi^i\partial \xi^j} - \alpha \frac{\partial\phi}{\partial \xi^i}\frac{\partial\phi}{\partial \xi^j}, \] and the manifold is dually projectively flat with constant sectional curvature \(K=-\alpha\) for \(D^{(\alpha)}\), and \(K=+\alpha\) for \(D^{(-\alpha)}\) [1712.03610]. A companion geometric treatment shows that the same object is monotone-equivalent to a conformal Bregman divergence, is equivalent via an affine immersion to Kurose’s geometric divergence, and serves as the canonical divergence of a statistical manifold with constant sectional curvature \(-\alpha\) [1906.09103]. The projection theory of this logarithmic divergence extends Amari’s dually flat construction. A dual foliation theorem gives orthogonal decompositions into primal and dual autoparallel submanifolds, and an \(L^{(\alpha)}\)-PCA problem is formulated by minimizing \[ \sum_{i=1}^N {\bf L}_{\psi}^{(\alpha)}[y(i):\eta(i)] \] over \(k\)-dimensional affine subspaces in primal coordinates [2105.07767]. In continuous-time optimization, the same geometry yields conformal mirror descent, with logarithmic cost \[ c_\lambda(x,y)=\frac{-1}{\lambda}\log(1+\lambda\langle x,y\rangle), \] metric \[ G_\lambda(\theta)=\nabla_\theta^2\varphi(\theta)+\lambda \nabla_\theta\varphi(\theta)\nabla_\theta\varphi(\theta)^\top, \] and dynamics that are a time change of a Hessian gradient flow [2209.02938]. ## 4. Logarithmic divergence families in robust statistical inference Another branch of the literature uses logarithmic transforms of divergence functionals to construct robust inference procedures. The Logarithmic Super Divergence is defined, for densities \(g,f\), by \[ {\rm LSD}_{\beta,\gamma}(g,f) = \frac{1}{A}\log \int f^{1+\beta} - \frac{1+\beta}{AB}\log \int f^B g^A + \frac{1}{B}\log \int g^{1+\beta}, \] where \(A=1+\gamma(1-\beta)\) and \(B=\beta-\gamma(1-\beta)\) [1407.3961]. It contains the Logarithmic Power Divergence at \(\beta=0\), the Logarithmic Density Power Divergence at \(\gamma=0\), and reduces to the likelihood disparity, a KL form, at \((\beta,\gamma)=(0,0)\) [1407.3961]. For minimum-LSD estimation, the model influence function at \(g=f_\theta\) is independent of \(\gamma\), is unbounded at \(\beta=0\), and is bounded and redescending for \(\beta>0\) [1407.3961]. The discrete asymptotic theory shows consistency and asymptotic normality of the minimum-LSD estimator, with asymptotic variance depending on \(\beta\) but not on \(\gamma\), while empirical robustness away from the model is still strongly affected by \(\gamma\) [1406.2112]. A broader characterization is given by the Functional Density Power Divergence class \[ \mathrm{FDPD}_{\varphi,\alpha}(g,f) = \varphi\!\left(\int f^{\alpha+1}\right) - \left(1+\frac{1}{\alpha}\right)\varphi\!\left(\int f^\alpha g\right) + \frac{1}{\alpha}\varphi\!\left(\int g^{\alpha+1}\right). \] For fixed \(\alpha>0\), this is a valid divergence if and only if \(\psi(x)=\varphi(e^x)\) is convex and strictly increasing on its domain [2105.06094]. This recovers the DPD with \(\varphi(x)=x\) and the LDPD with \(\varphi(x)=\log x\), and it explains why the logarithmic transform is compatible with divergence structure despite the concavity of \(\log\) in its original variable [2105.06094]. Within the Bregman framework, the logarithmic construction is sharply limited. The characterization of logarithmic Bregman functions shows that, up to affine terms, the only strictly convex Bregman generators that yield a meaningful logarithmic piecewise transform are power generators \(B(x)=Kx^{1+\alpha}\), so the only resulting logarithmic Bregman divergences are positive multiples of the LDPD family [2105.05963]. This circumscribes the search for new “logarithmic” Bregman-type divergences. ## 5. Optimization, scaling laws, and generalized logarithms Logarithmic divergence also appears as an operational quantity in coding and optimization. For fixed-length distribution matching over a finite alphabet with target \(T\), the optimal informational divergence obeys \[ D(P_{A^n}\Vert T^n)=\frac{1}{2}\log_2 n + O(1), \] and the coding rate satisfies \[ \frac{1}{n}\log_2 |S| \to H(T) \] [2105.00406]. This shows that logarithmic growth is an unavoidable asymptotic penalty for optimal fixed-length invertible distribution matching, and that threshold-type codebooks outperform constant-composition schemes when the alphabet size exceeds two [2105.00406]. For model geometry, maximizing information divergence from linear and toric statistical models is studied through logarithmic Voronoi polytopes. In linear models, the global maximum \(D(\mathcal M)\) is attained at the boundary of the simplex, more precisely at a vertex of a logarithmic Voronoi polytope attached to a vertex of the model [2308.15598]. In toric models, local maximizers are characterized as projection points, and the paper combines chamber complexes with numerical algebraic geometry to enumerate complementary vertex–face pairs and compute global maxima [2308.15598]. A further generalization replaces \(\log\) by a two-parameter deformed logarithm \[ \mathrm{Log}_{ab}(x)=\frac{x^{a-1}-x^{b-1}}{a-b}, \] with \(0