Smoothed Classical Divergences

• Smoothed classical divergences are defined by regularizing classical discrepancy measures through techniques like total-variation smoothing, infimal convolution, or simultaneous smoothing of both densities.
• They facilitate robust statistical inference and optimization by modifying the geometry and admissible perturbations while preserving key discriminative properties.
• Applications include clipping mechanisms for divergence-independent optimizers, kernel-based smoothing in continuous models, and establishing sharp Pinsker-type bounds.

In the literature considered here, smoothed classical divergences comprise several distinct regularization mechanisms for classical discrepancy functionals. One line of work identifies a canonical divergence on flat statistical manifolds and shows that, on the manifold of positive measures, it coincides exactly with the classical $\alpha$-divergence (Felice et al., 2019). Another defines smoothing by minimization over a total-variation ball, $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$, and proves that the optimizer is a divergence-independent clipped probability vector (Gour, 10 Mar 2026). A further line regularizes Rényi divergence by infimal convolution with an integral probability metric, thereby interpolating between Rényi divergences and IPMs (Birrell et al., 2022). In statistical inference for continuous models, smoothing can also mean smoothing both the data density and the model density before divergence minimization, as in the Basu–Lindsay approach for minimum $S$-divergence estimation (Ghosh et al., 2014).

1. Formal setting and recurrent constructions

A recurring framework starts from a classical divergence

$$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$

assumed to satisfy the data processing inequality

$$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$

In the total-variation formulation, smoothing is defined through the ball

$$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$

and the smoothed divergence is

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$

The same work emphasizes that $D^\varepsilon$ is again a classical divergence, because both the feasible set $B^\varepsilon(p)$ and $D$ are compatible with stochastic maps (Gour, 10 Mar 2026).

A different regularization pattern is infimal convolution. For a test-function space $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$0, the infimal-convolution $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$1-Rényi divergence is defined by

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$2

where

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$3

Here smoothing acts on the second argument through an auxiliary measure $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$4 and a geometric penalty determined by $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$5 (Birrell et al., 2022).

In continuous-model inference, smoothing can instead be applied to the compared objects themselves. The Basu–Lindsay method replaces the empirical distribution by a kernel density estimator

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$6

and compares it not with the raw model density $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$7, but with the smoothed model

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$8

The resulting estimator minimizes $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$9 rather than the unsmoothed divergence (Ghosh et al., 2014).

Taken together, these constructions indicate that smoothing need not refer to a single operation. It may act on the feasible set, on one argument of the divergence, on both arguments, or on the underlying geometric structure.

2. Information-geometric smoothing and the classical $S$0-divergence

The information-geometric formulation begins with a smooth manifold $S$1 equipped with a dualistic structure

$S$2

where $S$3 is a Riemannian metric and $S$4 are affine connections dual with respect to $S$5 in the sense that

$S$6

If both connections are torsion-free, the manifold is statistical; if both have zero curvature,

$S$7

the structure is flat (Felice et al., 2019).

On the manifold of positive measures, the $S$8-connections interpolate between the mixture and exponential connections: $S$9 These $$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$0-connections are dual with respect to the Fisher metric, and the pair $$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$1 is dually flat (Felice et al., 2019).

The canonical divergence introduced by Ay and Amari is defined by integrating the squared norm of the $$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$2-geodesic velocity: $$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$3 where $$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$4 is the $$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$5-geodesic from $$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$6 to $$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$7. In the general construction, the inverse exponential map is used to define the initial velocity and to transport it along the geodesic. On dually flat manifolds, this canonical divergence specializes to a Bregman-type canonical divergence; in the self-dual Levi-Civita case, it becomes $$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$8 (Felice et al., 2019).

For the manifold

$$D:\bigcup_{d\in\mathbb N}\big(\Delta_d\times \Delta_d\big)\to \mathbb R\cup\{\infty\},$$9

equipped with the Fisher metric

$$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$0

the paper computes the canonical divergence explicitly and proves

$$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$1

That is, the canonical divergence coincides exactly with the classical $$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$2-divergence on positive measures (Felice et al., 2019).

The same analysis records the limiting behavior of the classical family. As $$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$3, the divergence converges to the Kullback–Leibler divergence,

$$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$4

while $$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$5 yields the reverse KL form. The paper also notes that for $$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$6 the divergence is closely related to the Hellinger-type geometry, and that the parameter change $$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$7 connects the family to Tsallis/$$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$8-divergence (Felice et al., 2019). This suggests that, in flat $$D(Ep\|Eq)\le D(p\|q) \qquad\text{for every column-stochastic map }E.$$9-geometry, smoothing is realized intrinsically rather than by external mollification.

3. Total-variation smoothing and the clipping principle

For classical divergences satisfying data processing, the central structural result is that smoothing over a total-variation ball has a divergence-independent optimizer. Let

$$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$0

with the likelihood ratios ordered as $$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$1. Define the clipping thresholds

$$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$2

The clipped vector $$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$3 is then

$$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$4

Equivalently,

$$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$5

This is the unique vector obtained by clipping the likelihood ratio $$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$6 to the interval $$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$7 (Gour, 10 Mar 2026).

The main theorem states that for every classical divergence $$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$8,

$$B^\varepsilon(p):=\left\{p'\in\Delta_d:\frac12\|p-p'\|_1\le \varepsilon\right\},$$9

Hence the minimizer in the smoothing problem is always the same clipped vector $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$0, independent of the specific divergence. The divergence dependence enters only through the final evaluation at the clipped point (Gour, 10 Mar 2026).

The geometric mechanism is majorization. The total-variation ball $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$1 has extremal elements under majorization, and for minimization the relevant one is the flattest approximation. Operationally, smoothing allows redistribution of at most $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$2 total mass; the optimizer decreases overly large coordinates, increases overly small coordinates, and leaves intermediate coordinates unchanged. In relative-majorization language, the clipped construction is extremal in the $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$3-ball, which is the order-theoretic reason for divergence-independence (Gour, 10 Mar 2026).

A significant technical reduction uses relative majorization and uniform reference. For rational

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$4

the pair $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$5 can be converted into a pair $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$6 with $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$7 uniform, and for every classical divergence $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$8,

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q).$$9

The uniform-reference case is then lifted back to general $D^\varepsilon$0 (Gour, 10 Mar 2026).

4. Function-space regularized Rényi divergences

The function-space regularized Rényi divergences place Rényi divergence into an explicit smoothed-divergence framework. For $D^\varepsilon$1, the classical Rényi divergence between $D^\varepsilon$2 and $D^\varepsilon$3 is paired with an IPM penalty through the infimal convolution

$D^\varepsilon$4

The function space $D^\varepsilon$5 specifies the regularization geometry: choosing $D^\varepsilon$6 gives Wasserstein regularization, bounded functions give total variation, $D^\varepsilon$7 gives Dudley-type metrics, and RKHS unit balls give MMD-type regularization (Birrell et al., 2022).

The computational core is the dual representation obtained by Fenchel–Rockafellar duality: $D^\varepsilon$8 The paper emphasizes that this formula avoids the risk-sensitive exponential terms appearing in the Donsker–Varadhan representation and therefore exhibits lower variance, making it well-behaved when $D^\varepsilon$9 (Birrell et al., 2022).

Theorem 1 establishes the basic interpolation property: $B^\varepsilon(p)$0 together with nonnegativity and equality at $B^\varepsilon(p)$1. If $B^\varepsilon(p)$2 is strictly admissible, the divergence property also holds. The same theorem proves convexity in $B^\varepsilon(p)$3, joint convexity in $B^\varepsilon(p)$4 when $B^\varepsilon(p)$5, and lower semicontinuity (Birrell et al., 2022).

The limiting regimes make the interpolation precise. For admissible $B^\varepsilon(p)$6,

$B^\varepsilon(p)$7

while for strictly admissible $B^\varepsilon(p)$8,

$B^\varepsilon(p)$9

At $D$0, the construction yields the expected reverse-KL-type regularization; at $D$1, the rescaled family converges to a regularized worst-case-regret divergence (Birrell et al., 2022).

A major consequence is the removal of the absolute-continuity obstruction. Classical $D$2 is infinite for $D$3 unless $D$4, whereas the infimal-convolution regularized versions can compare measures that are not mutually absolutely continuous, including empirical distributions and low-dimensional supports (Birrell et al., 2022). The paper also records a data-processing inequality for the regularized divergences. By contrast, naive regularizations based on simply restricting the Donsker–Varadhan test-function space can fail to satisfy $D$5 for $D$6 and can be numerically unstable (Birrell et al., 2022).

5. Binary reduction and Pinsker-type bounds for smoothed divergences

For a broad family of data-processing divergences, optimal lower bounds in terms of trace or variational distance reduce to binary classical states. Writing

$D$7

for binary distributions, the fundamental linear lower-bound problem is

$D$8

and Theorem 1 gives the binary reduction

$D$9

The optimal convex bound

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$00

is also attained on two-dimensional classical states (Wienecke et al., 15 Jan 2026).

For smoothed divergences,

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$01

the paper proves a general shift-and-cutoff theorem. Assuming data processing, non-negativity, convexity in the first argument, and faithfulness, the convex lower bound becomes

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$02

Thus the unsmoothed bound is shifted to the right by $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$03 and set to zero on $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$04 (Wienecke et al., 15 Jan 2026).

The paper provides explicit bounds for several classical and quantum divergences whose binary optimization is classical in form.

Divergence	Lower bound in terms of $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$05	Smoothed form
Umegaki divergence	$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$06 as the simple Pinsker form	Shift rule applies
Fidelity divergence	$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$07	Shift rule applies
Neyman/Pearson $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$08	$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$09 for $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$10, then $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$11	Shift rule applies
Max divergence	$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$12	For $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$13, zero on $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$14, then $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$15

The same work notes several structural features: the bounds are often piecewise and meet continuously with matching derivatives; standard Pinsker-type estimates are loose for large $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$16; for Rényi divergences with $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$17, the large-$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$18 branch $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$19 is universal; and smoothing can remove the divergence of unsmoothed bounds as $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$20 (Wienecke et al., 15 Jan 2026).

6. Statistical inference, smooth surrogates, and constrained optimization

In continuous parametric models, the minimum $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$21-divergence estimator requires smoothing because the empirical distribution is discrete while the model density is continuous. The $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$22-divergence family is indexed by $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$23 and $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$24,

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$25

with

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$26

and includes the Cressie–Read power divergence at $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$27, the density power divergence at $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$28, and the $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$29 divergence at $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$30 (Ghosh et al., 2014).

Using the Basu–Lindsay approach, one smooths both the data and the model with the same kernel and minimizes

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$31

The resulting minimum $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$32-divergence estimator satisfies consistency and asymptotic normality under identifiability, common-support, smoothness, domination, and positive-definiteness conditions. At the model, both the influence function and the asymptotic distribution are independent of $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$33, while second-order influence analysis shows explicitly that $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$34 enters at second order (Ghosh et al., 2014). This corrects the common first-order impression that the choice of $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$35 is asymptotically irrelevant.

A different smoothing goal is the replacement of nonsmooth $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$36 discrepancies by smooth divergence surrogates. The smooth generator $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$37 introduced for generalized $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$38-divergences is strictly convex and $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$39, satisfies

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$40

and converges pointwise as

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$41

Consequently,

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$42

and the scaled shift divergence similarly converges to a weighted $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$43-distance. The paper also states the upper bound

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$44

with equality iff $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$45 (Bertrand et al., 31 Oct 2025). Here smoothing is a smooth approximation of a nonsmooth classical target.

In inverse problems, smoothing also appears through regularization and scale invariance. The generic objective is

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$46

with $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$47 interpreted as a Tikhonov-type regularizer and $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$48 as a default or smoothed solution. To handle nonnegative vectors with arbitrary total mass, the paper introduces scale-invariant divergences satisfying

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$49

often through an invariance factor

$$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$50

A central warning is that simplified probability-density forms of classical divergences can be unsuitable for inverse-problem optimization because the gradient with respect to the model argument may fail to vanish at $$D^\varepsilon(p\|q):=\min_{p'\in B^\varepsilon(p)}D(p'\|q)$$51; the KL example is used to illustrate this point (Lantéri, 2020). This is a distinct but related sense in which smoothing modifies classical divergences so that they remain compatible with optimization, constraints, and regularization.

These developments collectively show that smoothed classical divergences are not a single family but a class of constructions unified by a common purpose: preserving the discriminative content of classical divergences while modifying geometry, admissible perturbations, arguments, or generators so that the resulting objects remain tractable in information geometry, sharp inequalities, robust inference, and constrained optimization.