KL-Sphere in Divergence Geometry

• KL-Sphere is a framework where KL divergence defines level sets and neighborhoods, guiding distributions toward a uniform or nominal reference in various embedding spaces.
• It employs methods like kernel-density estimators and full-dimensional deterministic regularizers to reduce projection variance and improve instance separation in self-supervised learning.
• Empirical techniques such as k-nearest neighbor estimation and norm-based bounds enable accurate estimation and integration of KL-defined geometries in high-dimensional statistical models.

Kullback-Leibler Divergence Sphere, or KL-Sphere, is not a single universally standardized object in the literature. In the narrow and explicit sense introduced for self-supervised learning on hyperspherical latent spaces, it denotes a full-dimensional, deterministic regularizer on the unit hypersphere that drives the embedding distribution toward the uniform distribution on $\mathbb S^{d-1}$ by a kernel-density-estimated KL objective (Nicollier et al., 16 Jun 2026). In broader and mostly informal usage, the term is compatible with several KL-level-set or KL-neighborhood constructions: the independent Gaussian point that minimizes KL under fixed diagonal variances, the robust divergence from a distribution to a Lévy ball around a nominal law, and approximate equal-KL neighborhoods in Euclidean embeddings of model space (Fang et al., 2020, Yang et al., 2017, Kishino et al., 21 May 2025). Across these settings, the common idea is that KL divergence induces a geometry in which a reference distribution, plan, or model acts as a center, while departures in covariance, dependence, density, or behavior move outward.

1. Terminological scope and status

The term KL-Sphere is used explicitly in the hyperspherical representation-learning setting, but several adjacent papers emphasize that they do not define a literal KL-sphere. In entropy-relaxed optimal transport, the closest related object is a KL ball around the independent coupling ${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$, used as a constraint interpretation of entropy regularization rather than as a formally named sphere (Amari et al., 2017). In robust universal hypothesis testing for continuous distributions, the neighborhood is a Lévy ball around a nominal distribution, and KL is minimized over that set; the construction is described as a KL-neighborhood style construction, not as a classical KL-sphere (Yang et al., 2017). In reverse-Pinsker analysis, the relevant object is the KL depth of the complement of an $\ell_1$ ball, again a sphere-like boundary condition rather than a named KL sphere (Berend et al., 2012).

This terminological dispersion matters because KL divergence is asymmetric and does not generate Euclidean spheres in the usual metric sense. A plausible implication is that “KL-Sphere” functions best as an umbrella term for KL-defined level sets, constraint sets, and outward-from-center geometries, with the precise meaning determined by the ambient structure: simplex, Gaussian family, transportation-plan manifold, hypersphere, or empirical model map.

2. Hyperspherical KL-Sphere in self-supervised learning

In the explicit usage of the term, KL-Sphere is a regularizer for self-supervised learning with $\ell_2$-normalized embeddings on the sphere. The training objective is written as

$$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$

with

$$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$

The target prior is explicitly

$q = \mathrm{Unif}(\mathbb S^{d-1}),$

and the manifold KL divergence is

$$\mathrm{KL}(p\|q) = \int_{\mathbb S^{d-1}} p(x)\log\frac{p(x)}{q(x)}\,\mu(dx) = \mathbb E_{x\sim p}\!\left[\log p(x)-\log q(x)\right].$$

Because $q(x)=1/|\mathbb S^{d-1}|$, minimizing KL to the uniform target is equivalent to maximizing entropy on the sphere (Nicollier et al., 16 Jun 2026).

The empirical minibatch distribution is discrete, so the construction replaces it with a continuous surrogate via KDE. In the main text,

$\tilde p(x)=\mathbb E_{x'\sim \hat p}[k(x,x')],$

and in the appendix the paper uses the leave-one-out form

${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$0

The explicit normalized KL regularizer is

${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$1

with

${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$2

The kernels are rotationally invariant,

${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$3

with spectral expansion

${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$4

and the paper studies Heat and Bandlimited filters. For KL experiments it uses the heat kernel with ${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$5.

Relative to sliced regularizers such as SIGReg and SUSReg, KL-Sphere is full-dimensional and deterministic. The paper’s stated contrast is that sliced methods approximate continuous objectives through Monte Carlo projections and therefore inject projection variance into the training gradients, whereas the exact full-dimensional objectives remove that source of stochasticity. Empirically, the paper reports that the continuous KDE-based KL divergence promotes fine-grained instance separation, underperforms MMD and KSD on ImageNet-100 and Galaxy10, but yields the strongest result on procedural texture retrieval, with average Recall@1 of 95.3% versus 88.7% for SUSReg.

3. Central points, equality cases, and Gaussian KL geometry

A second important sense of KL-Sphere arises from KL-level-set geometry around an independent Gaussian reference. For ${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$6 and ${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$7 as ${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$8-dimensional random vectors, with ${\mathbf P}_D={\mathbf p}\otimes{\mathbf q}$9 Gaussian and

$\ell_1$0

Proposition 1 gives

$\ell_1$1

with equality when $\ell_1$2 is Gaussian and $\ell_1$3 is also diagonal (Fang et al., 2020).

The paper builds this through three components: a Gaussianization property, an independence-additivity property, and a diagonal-covariance decomposition. The Gaussian KL formula used is

$\ell_1$4

Its geometric interpretation is explicit in the paper’s discussion: for fixed marginal variances, correlations only increase the KL cost relative to the diagonal or independent case, so the independent Gaussian is the lowest KL level set point over all distributions sharing those diagonal variances. This is the clearest “center” interpretation of a KL-Sphere in the Gaussian family.

A complementary boundary-based picture appears in the reverse-Pinsker problem

$\ell_1$5

which studies the smallest KL value compatible with a fixed total-variation radius (Berend et al., 2012). The optimizer has a two-level form, and for balanced distributions the small-$\ell_1$6 expansion gives

$\ell_1$7

This suggests a distinct but related notion of KL-Sphere: the KL “height” attained on the boundary of an $\ell_1$8 sphere around a reference law.

4. KL-neighborhoods and robust hypothesis testing

In continuous universal hypothesis testing, the central construction is not a KL sphere but a Lévy ball

$\ell_1$9

where

$\ell_2$0

The robust KLD is defined as

$\ell_2$1

The paper stresses that this is best viewed as a KL-neighborhood style construction, because the geometric neighborhood is defined in Lévy metric and KL is minimized over that neighborhood rather than defining the neighborhood itself (Yang et al., 2017).

The main theorem states that if $\ell_2$2 is continuous in $\ell_2$3, then for any $\ell_2$4,

$\ell_2$5

is continuous in $\ell_2$6 with respect to weak convergence. This continuity repairs a basic deficiency of classical KL in continuous settings, where only lower semicontinuity generally holds. The resulting detector is

$\ell_2$7

for the null

$\ell_2$8

The paper’s asymptotic minimax Neyman-Pearson result shows that this robust threshold test is optimal under the stated open-set condition.

In this setting, a KL-Sphere interpretation is necessarily informal. The stable geometric object is the Lévy ball, while KL supplies a set-valued distance to that ball. The significance is operational: robustifying the nominal model from a point $\ell_2$9 to a neighborhood $$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$0 restores continuity and makes optimal universal testing possible for continuous observations.

5. Transport-plan manifolds and model-map embeddings

Entropy-relaxed optimal transport provides another KL-induced geometry. On the discrete simplex

$$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$1

the transportation plan $$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$2 from $$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$3 to $$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$4 is relaxed by entropy through

$$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$5

This is equivalently a restriction to a KL ball around the independent coupling $$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$6 because

$$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$7

The paper then constructs a canonical divergence between optimal transportation plans,

$$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$8

and a Bregman-like divergence on the simplex. It explicitly notes, however, that the framework uses simplices, flat manifolds, and level-set-like subsets, but no explicit “sphere” construction (Amari et al., 2017).

A different KL-induced geometry appears in language-model comparison. For a LLM $$\mathcal L = (1-\lambda)\mathcal L_{\mathrm{inv}} + \lambda \mathcal L_{\mathrm{reg}},$$9 on a fixed text set $$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$0, the log-likelihood vector is

$$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$1

and after double-centering the matrix of such vectors one obtains coordinates $$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$2 satisfying

$$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$3

In this representation, equal-KL neighborhoods can be viewed approximately as spheres centered at a model point, but the observed dynamics are not spherical: pretraining trajectories show a spiral structure, layerwise trajectories are thread-like, and diffusion in log-likelihood space is strongly subdiffusive, with $$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$4 compared with $$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$5 in weight space (Kishino et al., 21 May 2025).

Taken together, these two literatures show that KL-Sphere language is most faithful when it refers to approximate equal-KL neighborhoods rather than literal spherical manifolds. The transport case emphasizes KL balls and dually flat structure; the model-map case emphasizes Euclidean embeddings in which squared distance approximates KL.

6. Estimation, norm control, and bounded substitutes

A KL-Sphere becomes empirically meaningful only when its radius can be estimated or bounded. For continuous distributions with densities on $$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$6, one k-nearest-neighbor estimator is

$$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$7

built from two independent samples and the $$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$8-nearest-neighbor radii within and across samples (Bulinski et al., 2019). Under broad assumptions, the estimator is asymptotically unbiased and $$\tilde z_{n,v} = \frac{z_{n,v}}{\|z_{n,v}\|} \in \mathbb S^{d-1}, \qquad \mathcal L_{\mathrm{inv}} = \frac{1}{V_a} \sum_{v=1}^{V_a} \|\tilde z_{n,v} - \mu_n\|_2^2, \qquad \mu_n = \frac{1}{V_g} \sum_{v'=1}^{V_g} \tilde z_{n,v'}.$$9-consistent, including for Gaussian measures with nondegenerate covariance matrices. A related fixed-$q = \mathrm{Unif}(\mathbb S^{d-1}),$0 kNN estimator,

$q = \mathrm{Unif}(\mathbb S^{d-1}),$1

has bias and variance bounds together with minimax lower bounds, and the paper concludes that the kNN method is asymptotically rate optimal up to logarithmic factors (Zhao et al., 2020).

Norm-based bounds further constrain KL neighborhoods. For densities $q = \mathrm{Unif}(\mathbb S^{d-1}),$2 on $q = \mathrm{Unif}(\mathbb S^{d-1}),$3, the paper derives upper bounds showing that sufficiently strong $q = \mathrm{Unif}(\mathbb S^{d-1}),$4 convergence of densities implies KL convergence, while Pinsker’s inequality gives the converse implication from small KL to small $q = \mathrm{Unif}(\mathbb S^{d-1}),$5 distance. Under additional assumptions, $q = \mathrm{Unif}(\mathbb S^{d-1}),$6, $q = \mathrm{Unif}(\mathbb S^{d-1}),$7, and KL convergence become equivalent. The paper summarizes this by stating that convergence in KL-divergence sense sandwiches between the convergence of density functions in terms of $q = \mathrm{Unif}(\mathbb S^{d-1}),$8 and $q = \mathrm{Unif}(\mathbb S^{d-1}),$9 norms (Yao et al., 2024). This suggests that KL-spheres in density space are trapped between norm neighborhoods rather than behaving like Euclidean spheres.

A separate line of work argues that in some applications KL should ideally be bounded. For finite alphabets, one paper derives an upper bound of $$\mathrm{KL}(p\|q) = \int_{\mathbb S^{d-1}} p(x)\log\frac{p(x)}{q(x)}\,\mu(dx) = \mathbb E_{x\sim p}\!\left[\log p(x)-\log q(x)\right].$$0 for cross entropy and therefore for KL, and proposes bounded alternatives such as

$$\mathrm{KL}(p\|q) = \int_{\mathbb S^{d-1}} p(x)\log\frac{p(x)}{q(x)}\,\mu(dx) = \mathbb E_{x\sim p}\!\left[\log p(x)-\log q(x)\right].$$1

whose range is $$\mathrm{KL}(p\|q) = \int_{\mathbb S^{d-1}} p(x)\log\frac{p(x)}{q(x)}\,\mu(dx) = \mathbb E_{x\sim p}\!\left[\log p(x)-\log q(x)\right].$$2 (Chen et al., 2019). That work does not define a KL-sphere, but it does motivate a bounded divergence region when an application requires interpretability of the distortion scale.

In aggregate, the estimation and bounding literature does not standardize the term KL-Sphere, yet it provides the technical infrastructure needed to use KL-defined neighborhoods as statistical objects: sample-based radius estimation, asymptotic guarantees, and analytic containment relations between KL level sets and norm balls.