Simplex-Space Smoothing Overview

• Simplex-space smoothing is a unifying framework that replaces hard or high-dimensional operations with simplex-aware averaging across probability simplices, geometric simplices, and simplicial lattices.
• It encompasses methods like Gibbs smoothing for vector quantization, barycentric interpolation on permutohedral lattices for scalable Gaussian processes, and Dirichlet-based diffusion for categorical modeling.
• Applications span improved quantizer utilization, efficient kernel matrix-vector multiplies, nonparametric regression on simplices, and enhanced higher-order attention in neural networks.

Across recent literature, the expression simplex-space smoothing appears in several technically distinct settings in which smoothing is performed either on a probability simplex, on the standard geometric simplex, or on a simplicial lattice (Morita, 26 Sep 2025, Kapoor et al., 2021, Richemond et al., 2022, Ma et al., 2021, Genest et al., 2024, Dussolle et al., 17 Dec 2025). In smoothed vector quantization, continuous features are mapped to simplex-valued soft assignments and regularized toward simplex vertices to prevent code collapse (Morita, 26 Sep 2025). In scalable Gaussian-process inference, kernel matrix-vector multiplies are approximated by splat–blur–slice operations on the permutohedral lattice (Kapoor et al., 2021). In categorical diffusion, independent Cox–Ingersoll–Ross processes on the positive orthant are normalized to induce Dirichlet-attracted dynamics on $\Delta^{n-1}$ (Richemond et al., 2022). In approximation theory and nonparametric regression, smoothing is performed directly on $\Delta^d$ via Kantorovich-type operators or Dirichlet kernels (Ma et al., 2021, Genest et al., 2024). In higher-order attention, the phrase denotes the contraction of feature differences under $N$-simplicial aggregation (Dussolle et al., 17 Dec 2025). This suggests a unifying geometric theme rather than a single algorithm: hard or high-dimensional operations are replaced by simplex-aware averaging.

1. Smoothed vector quantization and simplex vertices

In smoothed vector quantization, a continuous feature $z\in\mathbb R^d$ is quantized with respect to a codebook $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$. Classical hard quantization selects

$$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$

which is non-differentiable. The smoothed formulation relaxes the one-hot index to a density on the probability simplex

$$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$

typically via Gibbs smoothing

$$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$

or, under $L_2$ normalization of both $z$ and the codebook, $\Delta^d$0. The differentiable quantized output is then

$\Delta^d$1

The vertices of $\Delta^d$2 are the canonical one-hot vectors $\Delta^d$3, and the limit $\Delta^d$4 recovers hard assignment. The two stated desiderata are that smoothed assignments remain close to a one-hot vector and that all codebook entries be utilized; the second requirement addresses code collapse, where only a small subset of vertices is ever approached and unused codewords receive little or no gradient (Morita, 26 Sep 2025).

Morita proposes a KNN vertex regularizer that minimizes the average distance between each simplex vertex and its $\Delta^d$5 nearest smoothed assignments in the current mini-batch or a small buffer:

$\Delta^d$6

The distance $\Delta^d$7 may be squared $\Delta^d$8 distance or cross-entropy. The regularizer is added to the task loss, for example

$\Delta^d$9

or, in Wav2Vec 2.0 style pretraining, to the contrastive objective on internal smoothed codes. Because $N$0 is fully differentiable, gradients of $N$1 flow to both the codebook and upstream parameters, and no additional annealing schedule for $N$2 is required.

The reported empirical pattern is explicit. In discrete image autoencoding on ImageNet, STE or Gumbel-softmax plus perplexity-maximization often collapse without the KNN regularizer: only $N$3–$N$4 of codes are used when $N$5, with similarly poor results for $N$6. With KNN-$N$7 or KNN-CE, code-use goes to $N$8 across all three feature-map $N$9 code-book configurations, and reconstruction metrics (rMSE, FID, IS) improve or match the best prior method that required carefully annealed Gumbel-softmax and perplexity penalties. In contrastive speech representation learning with Wav2Vec 2.0 pretraining on LibriSpeech, single-codebook $z\in\mathbb R^d$0 and dual-codebook $z\in\mathbb R^d$1 settings show near $z\in\mathbb R^d$2–$z\in\mathbb R^d$3 usage without the KNN loss; KNN-CE yields $z\in\mathbb R^d$4 usage of every code in all settings, and downstream word-error-rate improvements track the improved quantizer utilization. The ablation reported in the same study states that cross-entropy is more robust to code collapse under Gumbel sampling, whereas $z\in\mathbb R^d$5 suffices for pure softmax smoothing.

2. Permutohedral-lattice smoothing for scalable kernel MVMs

A different use of simplex-space smoothing arises in scalable Gaussian processes through the permutohedral lattice. The construction works in the $z\in\mathbb R^d$6-dimensional hyperplane

$z\in\mathbb R^d$7

whose simplicial cells are permutations of a canonical simplex. Kapoor et al. use a fast triangular embedding matrix $z\in\mathbb R^d$8 to map $z\in\mathbb R^d$9 to $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$0. A Conway–Sloane style algorithm then locates the enclosing simplicial cell in $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$1 time, identifies its $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$2 neighboring lattice vertices, and computes barycentric weights $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$3 such that

$C=\{c_1,\dots,c_K\}\subset\mathbb R^d$4

These barycentric coordinates are the simplex-valued interpolation coefficients that drive the subsequent sparse MVM approximation (Kapoor et al., 2021).

The core computation is the three-stage splat–blur–slice pipeline. Given inputs $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$5 with values $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$6, splat scatters each $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$7 to the $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$8 lattice vertices of the enclosing simplex using the barycentric weights; blur performs 1D convolutions along each of the $C=\{c_1,\dots,c_K\}\subset\mathbb R^d$9 lattice axes using a small stencil; slice interpolates the blurred lattice values back to the original points. In SKI notation, these three stages are

$$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$0

so that

$$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$1

The reported complexity is $$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$2 time and $$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$3 memory, where $$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$4 is the number of occupied lattice nodes and typically satisfies $$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$5. The stated practical significance is that $$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$6, so the method avoids the curse of dimensionality associated with dense rectangular grids.

For the RBF kernel $$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$7, a 1D convolution along each lattice axis with the binomial stencil $$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$8 yields a separable approximation to the multi-dimensional Gaussian in $$i^*(z)=\arg\min_{i=1..K}\|z-c_i\|^2,\qquad q(z)=c_{i^*(z)},$$9. More generally, any stationary kernel can be approximated by choosing a small 1D stencil of radius $$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$0 on each axis. The paper also gives a fused CUDA implementation: each input point is processed in its own GPU thread during splat and slice; the sparse hash table of active lattice nodes is stored in shared or global memory with open addressing; and blur along each axis is launched as a separate kernel. On a Titan RTX GPU, for $$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$1 and $$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$2, a single MVM is reported as approximately $$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$3 faster than an exact KeOps implementation of the RBF MVM. Even with order-1 stencils $$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$4, the cosine-error of the approximate MVM relative to the exact MVM is below $$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$5.

3. Simplex diffusion and Dirichlet-attracted dynamics

In categorical diffusion, simplex-space smoothing is realized through stochastic dynamics on the positive orthant followed by normalization. Fix $$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$6 and concentration parameters $$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$7. The forward process consists of $$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$8 independent Cox–Ingersoll–Ross processes

$$\Delta^{K-1}=\{p\in\mathbb R_+^K:\sum_{i=1}^K p_i=1\},$$9

with $$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$0 and independent Brownian motions $$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$1. Since each coordinate remains nonnegative almost surely, one normalizes

$$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$2

As $$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$3, each $$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$4 converges in law to $$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$5, hence $$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$6 converges to $$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$7. The paper presents this as the simplex analogue of progressive Gaussian smoothing in Euclidean diffusion models (Richemond et al., 2022).

The scalar CIR transition law is available in closed form. If

$$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$8

then $$s_i(z)=\frac{\exp(-\|z-c_i\|^2/\tau)}{\sum_{j=1}^K \exp(-\|z-c_j\|^2/\tau)},$$9 can be sampled exactly through a noncentral $L_2$0 random variable:

$L_2$1

By sampling the $L_2$2 independently in parallel and normalizing, one obtains exact samples on the simplex with no time-discretization error. The associated Fokker–Planck equation,

$L_2$3

plays the role that the Ornstein–Uhlenbeck heat/Fokker–Planck equation plays in Gaussian diffusion: the law is progressively driven from a delta-like initial condition toward the stationary Gamma law, and after normalization toward the Dirichlet law on the simplex.

The paper is equally explicit about limitations. Euler–Maruyama discretization requires clipping or reflection to maintain nonnegativity and a small stabilizer $L_2$4 inside the square root. Exact noncentral-$L_2$5 sampling avoids time-discretization bias but can be more expensive for large $L_2$6 or very small $L_2$7. In high-dimensional simplices, the normalized vector $L_2$8 can exhibit extremely heavy-tailed behavior in its components, making score estimation difficult. There is also an interpretational gap: while the Dirichlet is presented as a natural Gaussian analogue on $L_2$9, the posterior $z$0 after normalization is a nonlinear function of $z$1, so the Dirichlet view is described as a helpful but inexact intuition for the final categorical likelihood.

4. Approximation operators on the standard simplex

A classical smoothing perspective appears in approximation theory on the standard simplex

$z$2

The paper on convergence in Orlicz space studies multivariate operators on $z$3 within the Orlicz class $z$4 associated with a Young function $z$5 satisfying the $z$6 condition. Two positive linear operators are central: the Meyer–König–Zeller–Kantorovich operator $z$7 and the Stancu–Kantorovich operator $z$8, both defined by weighted averages of local integrals over small simplex-aligned cells. They reproduce constants exactly and satisfy

$z$9

with first moments vanishing like $\Delta^d$00. Their operator norms obey

$\Delta^d$01

(Ma et al., 2021).

The relevant smoothness functional is the second-order modulus of smoothness

$\Delta^d$02

The two main approximation theorems have the same rate:

$\Delta^d$03

and

$\Delta^d$04

The approximation scale is therefore $\Delta^d$05.

The interpretation given in the paper is directly smoothing-theoretic: these operators average $\Delta^d$06 over cells of diameter $\Delta^d$07 and thus smooth out fine oscillations. If $\Delta^d$08, then

$\Delta^d$09

The same framework covers the special case $\Delta^d$10, $\Delta^d$11, where the Orlicz norm reduces to the usual $\Delta^d$12 norm. This places simplex smoothing in a classical approximation-theoretic lineage, distinct from machine-learning relaxations but still centered on localized averaging over simplex-adapted cells.

5. Dirichlet-kernel local linear smoothing for regression surfaces

In nonparametric regression on the simplex, the sample consists of independent pairs

$\Delta^d$13

with $\Delta^d$14, $\Delta^d$15, and $\Delta^d$16. The local smoother is built from a Dirichlet kernel centered at an evaluation point $\Delta^d$17 with bandwidth $\Delta^d$18,

$\Delta^d$19

where

$\Delta^d$20

The stated properties are that $\Delta^d$21 integrates to one on $\Delta^d$22, concentrates around $\Delta^d$23 as $\Delta^d$24, and automatically tilts near a face of the simplex to avoid leaking outside the support. The local linear estimator solves

$\Delta^d$25

with fitted value $\Delta^d$26, or equivalently

$\Delta^d$27

in matrix form (Genest et al., 2024).

Under the stated smoothness assumptions $\Delta^d$28, Lipschitz $\Delta^d$29 and $\Delta^d$30, $\Delta^d$31, and $\Delta^d$32, the interior-point bias expansion begins with

$\Delta^d$33

followed by $\Delta^d$34 and $\Delta^d$35 terms. The variance is of order $\Delta^d$36. Balancing $\Delta^d$37 gives

$\Delta^d$38

and the same rate for the mean integrated squared error:

$\Delta^d$39

The boundary behavior is a central point. Local linear smoothing with the Dirichlet kernel has bias $\Delta^d$40 even at boundary points, whereas the Nadaraya–Watson estimator with Dirichlet kernel has boundary bias of order $\Delta^d$41 or $\Delta^d$42 in special settings. Practical implementation uses least-squares cross-validation, numerically stable log-Gamma evaluation for the weights, and inversion of the weighted design matrix $\Delta^d$43. In the reported Monte Carlo experiments for $\Delta^d$44, integrated squared errors were recorded over $\Delta^d$45 replications for sample sizes $\Delta^d$46. For every regression function and every $\Delta^d$47, the local linear estimator had smaller mean ISE, median ISE, standard deviation, and interquartile range than the Nadaraya–Watson competitor, with typical gains in mean ISE ranging from a factor $\Delta^d$48 to a factor $\Delta^d$49.

6. N-simplicial attention, over-smoothing, and recurrent limitations

In higher-order attention, simplex-space smoothing refers not to probabilistic diffusion or kernel estimation but to the contraction induced by aggregation over simplices of order $\Delta^d$50. Let $\Delta^d$51 denote token features. N-simplicial attention introduces $\Delta^d$52 key maps and $\Delta^d$53 value maps, forms logits on allowed $\Delta^d$54-simplices $\Delta^d$55,

$\Delta^d$56

normalizes with a softmax over $\Delta^d$57, and outputs

$\Delta^d$58

On a bounded input ball, the operator is locally Lipschitz with constant

$\Delta^d$59

The reported dependence is polynomial in the token count $\Delta^d$60, with multiplicative growth in the value and key norms and in the interaction order $\Delta^d$61 (Dussolle et al., 17 Dec 2025).

The principal theoretical result is negative with respect to a common intuition: higher-order aggregation does not remove over-smoothing. In the unmasked complete-hypergraph case, repeated application of pure N-simplicial attention contracts the residual part of the representation toward a constant row, and with residual connections the residual norm is stated to decay doubly-exponentially in the number of layers. In the masked quasi-strongly connected case of radius $\Delta^d$62, the residual obeys an exponential bound of the form

$\Delta^d$63

The paper therefore concludes that N-simplicial attention remains a contracting operator on the residual component of token features.

Because fully dense N-simplicial attention costs $\Delta^d$64 per head, two sparse routing schemes are proposed. Expert-choice Top-$\Delta^d$65 learns a scalar score per token, retains only the top-$\Delta^d$66 tokens, and reduces complexity to $\Delta^d$67. Simplicial Path Sparse Attention first applies pairwise sparse attention, then forms each $\Delta^d$68-simplex by walking along $\Delta^d$69 edges in the sparse graph, yielding complexity approximately $\Delta^d$70 for $\Delta^d$71. The empirical illustration uses discrete Forman–Ricci curvature on $\Delta^d$72 QM9 molecules: the average curvature of the line-graph is higher than that of the original graph in $\Delta^d$73 of cases, and the minimum edge curvature improves in at least $\Delta^d$74 of molecules.

Taken together, these literatures suggest several recurring technical distinctions. First, simplex-aware smoothing may be vertex-seeking, as in smoothed vector quantization, boundary-adaptive, as in Dirichlet-kernel regression, or lattice-interpolative, as in permutohedral MVMs. Second, smoothing is often beneficial only when paired with a mechanism that preserves structural coverage: the KNN regularizer enforces per-vertex usage in vector quantization, while sparse simplex selection in N-simplicial attention is introduced to control combinatorial growth. Third, simplex formulations do not by themselves remove failure modes. The supplied results explicitly retain code collapse as a risk in quantization without per-vertex coverage, high-dimensional instability in simplex diffusion, and over-smoothing in higher-order attention.