Closed-Form Attention Kernel

• Closed-Form Attention Kernels are mathematically defined mechanisms that use explicit kernel functions to compute attention weights without learned parameters.
• They integrate kernel methods with nonparametric regression, facilitating efficient, sparse, and interpretable designs in models for sequence, vision, and geometric learning.
• By employing classical kernels such as Epanechnikov and Gaussian, these methods offer principled control over sparsity, parameter efficiency, and theoretical analysis in neural architectures.

Closed-form attention kernels are a class of attention mechanisms in neural architectures—especially Transformers—where the attention map is specified by an explicit, mathematically defined kernel function rather than a learned or implicitly parameterized form. This paradigm establishes rigorous connections between attention, kernel methods, and nonparametric regression, and enables a spectrum of efficient, interpretable, and sometimes invariant or sparse attention mechanisms derived from the underlying kernel structure. Closed-form kernels enable deeper understanding, theoretical analysis, and principled generalizations of attention mechanisms across domains including sequence modeling, generative modeling, vision, and geometric learning.

1. Kernel Smoother View of Attention

The foundational perspective formalizes softmax attention as kernel smoothing. For a single query–key–value setup, the standard scaled dot-product attention can be rewritten as

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}(Q K^\top / \sqrt{d_k})\,V$$

If $q_i, k_j$ are the $i$-th query and $j$-th key, this leads to a kernel-based interpretation (Tsai et al., 2019): $$K(q_i, k_j) = \exp(\langle q_i, k_j \rangle / \sqrt{d_k})$$

$$\alpha_{ij} = \frac{K(q_i, k_j)}{\sum_t K(q_i, k_t)}$$

$\mathrm{output}_i = \sum_j \alpha_{ij} v_j$

This casts the attention output as a normalized kernel smoother, with the exponential kernel—which is asymmetric in general—as its weighting function.

Under this lens, multi-head attention corresponds to parallel ensembles of smoothers with different learned kernel projections. For positional embeddings and related attributes, the kernel formalism allows principled compositional or multiplicative integration, e.g.,

$$K((f_q, t_q), (f_k, t_k)) = k_F(f_q, f_k)\, k_T(t_q, t_k)$$

where content and position interact through separate symmetric kernels. This product-of-kernels approach achieves strong empirical performance and significant parameter savings, highlighting the flexibility enabled by the kernel framework (Tsai et al., 2019).

2. Closed-Form Kernels via Classical Kernel Theory

The kernel attention map can be specified directly using known kernels from statistical learning. A prominent construction is the use of compact-support polynomial kernels (Epanechnikov, biweight, triweight), which yield sparse attention mechanisms with explicit analytic weights (Santos et al., 30 Jan 2026): $K^{(r)}(u) \propto [1 - \| u \|^2 ]_+^r$ with $r=1$ (Epanechnikov), $q_i, k_j$0 (biweight), $q_i, k_j$1 (triweight). The kernel is zero outside the unit ball, enforcing locality and sparsity in attention. When scores are parameterized as inner products $q_i, k_j$2, the attention weights become closed-form functions of the scores, giving rise to mechanisms like normalized-ReLU attention (fixed Epanechnikov) and sparsemax (adaptive bandwidth Epanechnikov).

α-entmax generalizes these further, interpolating between sparsemax ($q_i, k_j$3) and softmax ($q_i, k_j$4) via higher-order polynomial kernels. The entire family thus admits closed-form attention weight computation, controlled by the parameter $q_i, k_j$5 (or equivalently, $q_i, k_j$6), with softmax/Gaussian as the infinite-order/limit of increasingly sharp polynomial kernels (Santos et al., 30 Jan 2026).

3. Closed-Form Gaussian and Nadaraya–Watson Attention

Under generative models with flow-matching or optimal-transport conditioning, attention is precisely realized as a Nadaraya–Watson kernel smoother: $q_i, k_j$7 with bandwidth $q_i, k_j$8 varying over the flow (e.g., $q_i, k_j$9). The output is a local mean of context points $i$0, with the degree of smoothing explicitly controlled by the time-dependent bandwidth. This formulation allows exact closed-form realization of the conditional velocity field in continuous flows using a single attention head parameterized by the Gaussian kernel, with clear asymptotic regimes (global averaging to one-nearest-neighbor) as $i$1 varies (Smola, 13 May 2026).

Failures in such closed-form setups—nearest-neighbor collapse in high dimension, kernel–data geometry mismatch, and support scarcity—are fully predictable from classical nonparametric theory and can be mitigated by employing subspace projections or learned kernels.

4. Closed-Form Kernels in Geometric and Lie Group Attention

In geometric contexts, closed-form attention kernels allow for intrinsic, equivariant token interaction over matrix groups. The Lie-Algebra Attention paradigm places tokens directly on a matrix Lie group $i$2, e.g., $i$3, $i$4, or $i$5. The pairwise attention score adopts a canonical, intrinsic form: $i$6 with the logarithm map returning the relative pose in the Lie algebra, $i$7 denoting block weights respecting the natural group decomposition, and $i$8 a temperature. This approach is free of learned kernels, MLP parameterizations, or explicit representation-theoretic machinery, and extends closed-form, block-weighted proximity measures to non-compact and non-abelian groups (inc. $i$9) (Musialski, 18 Jun 2026).

Empirical results indicate substantial parameter efficiency ($j$0–$j$1 fewer score parameters) and exact machine-precision equivariance compared to neural or flat feature-based alternatives.

5. Closed-Form Spatial and Explicit Kernels in Vision

In visual contexts, content-independent closed-form kernels are leveraged for spatial attention. Explicitly modeled attention maps can fix or parameterize the attention matrix $j$2 as a function of geometric distance, typically via a single Gaussian kernel over pixel coordinates: $j$3 where $j$4 is a globally or layer-shared, learnable scale parameter (Tan et al., 2020). Empirical results show that such closed-form kernels—with as few as one learnable parameter per block—outperform content-dependent self-attention in standard ResNet and MobileNet contexts, while reducing FLOPs and parameter count.

Ablations further indicate that the geometric prior alone captures the major signal, that fixed kernels can suffice (even outperforming learned alternatives), and that sharing the kernel across heads can be superior for optimization.

6. Closed-Form Approximations via Random Features and Polynomial Expansions

Closed-form kernel approximations enable scalable, efficient attention by projecting $j$5 dense kernels to $j$6 low-rank representations. Two approaches stand out:

• Random Features for Gaussian/Softmax/Affine Kernels: Positive, parameter-optimized random feature maps with explicit variance-minimizing closed-form parameters (e.g., Dense-Exponential Random Features, DERF family) achieve unbiased kernel estimates with exponentially reduced variance in self-attention (Likhosherstov et al., 2023). The FAVOR# method generalizes classical low-rank approximations and enables linear-time, positive-only attention via careful closed-form parameterization.
• Polynomial (Schoenberg/Maclaurin) Expansions: Attention using harmonic analysis-inspired polynomial basis, approximated by Random Maclaurin Features, achieves unbiased kernel approximations for any positive-definite dot-product kernel. Pre- and post-normalizations enforce assumptions required for the expansion, and theoretical guarantees on concentration/error are provided (Guo et al., 18 May 2025). These mechanisms provide drop-in replacements for standard attention blocks with lower memory/time complexity and preserved predictive performance.

7. Unified Theory and Implications

The kernel-based formalism unifies diverse attention mechanisms—standard dot-product, sparse, geometric, equivariant, and efficient approximations—under a rigorous mathematical framework. It explains observed empirical phenomena (e.g., the dominance of geometric over content signals in vision, the emergence of sparsity, or failure modes in high-dimension), enables systematic design using known kernels from statistical and harmonic analysis literature, and opens avenues for principled extension (e.g., multi-factor, symmetric/asymmetric, relative/absolute, and learned/analytic structures) (Tsai et al., 2019, Santos et al., 30 Jan 2026).

Key comparative insights are summarized below:

Kernel Type	Attention Regime	Parameterization
Exponential (Gaussian/softmax)	Dense, smooth, standard Transformer	Full/low-rank analytic, random feature
Compact polynomial	Sparse (α-entmax, ReLU)	Analytic, closed-form
Lie-algebra norm	Geometric, group-equivariant	No learned kernel
Spatial/geometric	Vision, explicit maps	Analytic, 1 parameter
Random polynomial	Efficient, low-rank	Kernel Maclaurin

This perspective enables direct evaluation of trade-offs (sparsity, efficiency, equivariance, approximation error), links to statistical learning theory (kernel regression), and informs practical design and theoretical analyses of next-generation attention architectures.