---
title: Continuous α-Entmax Attention
url: https://www.emergentmind.com/topics/continuous-entmax-attention
type: topic
---

# Continuous α-Entmax Attention

Continuous α-entmax attention is a continuous-domain generalization of discrete α-entmax and sparsemax attention in which attention is represented by a probability density \(p(t)\) on a measure space \(S\), obtained from a regularized prediction map with Tsallis-\(\alpha\) negentropy, and used to compute a context vector as a continuous expectation \(c=\int_S p(t)V(t)\,d\nu(t)\). For \(\alpha=1\) it recovers continuous softmax attention; for \(\alpha>1\) it yields densities with varying, possibly compact support; and for \(\alpha=2\) it becomes continuous sparsemax, producing truncated quadratic densities such as truncated parabolas and truncated paraboloids [2006.07214].

## 1. Discrete origins and variational formulation

In the discrete case, attention over a finite set \(S=\{1,\dots,L\}\) starts from a score vector \(f\in\mathbb{R}^L\). Softmax maps scores to probabilities by
\[
\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},
\]
and therefore always produces dense distributions. Sparse alternatives such as sparsemax and \(\alpha\)-entmax arise from the framework of regularized prediction maps (RPMs): for a strictly convex regularizer \(\Omega:\triangle^L\to\mathbb{R}\),
\[
\hat p_\Omega[f]=\arg\max_{p\in\triangle^L} p^\top f-\Omega(p).
\]
When \(\Omega\) is the Tsallis-\(\alpha\) negentropy,
\[
\Omega_{\alpha}(p)=
\begin{cases}
\dfrac{1}{\alpha(\alpha-1)}\left(\sum_{k=1}^L p_k^\alpha-1\right), & \alpha\neq 1,\\[3pt]
\sum_{k=1}^L p_k\log p_k, & \alpha=1,
\end{cases}
\]
the resulting RPM is the discrete \(\alpha\)-entmax transformation [2006.07214].

The special cases organize the family. At \(\alpha=1\), the Shannon negentropy is recovered and the RPM gives softmax. At \(\alpha=2\), the map becomes sparsemax,
\[
\hat p_{\Omega_2}[f]=\operatorname{sparsemax}(f)=\arg\min_{p\in\triangle^L}\|p-f\|^2,
\]
that is, Euclidean projection onto the simplex. Intermediate values \(1<\alpha<2\) interpolate between softmax and sparsemax, controlling sparsity. In this discrete setting, varying support means that the number and identity of indices with nonzero probability change with the scores \(f\) or the parameter \(\alpha\). The practical significance is twofold: only a few positions may be truly attended, and the attention map can encode a structural prior toward compact regions or few tokens [2006.07214].

A related but distinct line of work later used \(\alpha\)-entmax in standard Transformer attention and learned \(\alpha\) per head, showing that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers; this literature concerns discrete attention over tokens rather than continuous domains [1909.00015].

## 2. Continuous extension via Tsallis-regularized prediction maps

The continuous construction replaces the finite domain by an arbitrary measure space \((S,\mathcal A,\nu)\). A probability distribution is represented by a density \(p\in\mathcal M_+^1(S)\), meaning \(p:S\to\mathbb R_+\) and \(\int_S p(t)\,d\nu(t)=1\). For a strictly convex functional \(\Omega:\mathcal M_+^1(S)\to\mathbb R\), the continuous RPM is
\[
\hat p_\Omega[f]=\arg\max_{p\in\mathcal M_+^1(S)} \mathbb E_p[f(t)]-\Omega(p),
\qquad
\mathbb E_p[f(t)]=\int_S p(t)f(t)\,d\nu(t).
\]
This turns attention normalization into a functional optimization problem over densities rather than a finite-dimensional projection over the simplex [2006.07214].

The construction uses the \(\beta\)-logarithm and \(\beta\)-exponential,
\[
\log_{\beta}(u)=
\begin{cases}
\dfrac{u^{1-\beta}-1}{1-\beta}, & \beta\neq 1,\\[3pt]
\log u, & \beta=1,
\end{cases}
\qquad
\exp_{\beta}(u)=
\begin{cases}
[1+(1-\beta)u]_+^{1/(1-\beta)}, & \beta\neq 1,\\[3pt]
\exp u, & \beta=1.
\end{cases}
\]
The continuous Tsallis-\(\alpha\) negentropy is
\[
\Omega_{\alpha}(p)=\tfrac{1}{\alpha}\mathbb E_p[\log_{2-\alpha}(p(t))]
=
\begin{cases}
\dfrac{1}{\alpha(\alpha-1)}\left(\int_S p(t)^\alpha-1\right), & \alpha\neq 1,\\[3pt]
\int_S p(t)\log p(t), & \alpha=1.
\end{cases}
\]
For \(\alpha>0\), this functional is strictly convex, so the continuous RPM is well defined and unique [2006.07214].

The resulting solution has the deformed-exponential form
\[
\hat p_{\Omega_\alpha}[f](t)=\exp_{2-\alpha}\bigl(f(t)-A_\alpha(f)\bigr),
\]
where \(A_\alpha(f)\) is the normalizing scalar enforcing unit mass. When \(\alpha=1\), this becomes the Boltzmann-Gibbs density
\[
\hat p_\Omega[f](t)=\frac{\exp(f(t))}{\int_S \exp(f(t'))\,d\nu(t')}=\exp\bigl(f(t)-A(f)\bigr).
\]
When \(\alpha>1\), the deformed exponential is truncated, so the density is exactly zero wherever the bracket is negative. In the continuous setting, sparsity is therefore expressed as compact or varying support rather than as a finite set of exact zero coordinates [2006.07214].

With a linear score parametrization \(f_\theta(t)=\theta^\top\phi(t)\), these densities form a deformed exponential family, or \(q\)-exponential / \(t\)-exponential family. For \(\alpha>1\) they are called \(\alpha\)-sparse families. A central auxiliary object is the \(\beta\)-escort distribution,
\[
\tilde p^\beta(t)=\frac{p(t)^\beta}{\|p\|_\beta^\beta},
\qquad
\|p\|_\beta^\beta=\int_S p(t')^\beta\,d\nu(t'),
\]
which appears in derivatives of the normalizer and in Jacobians of the attention map [2006.07214].

## 3. Attention mechanism on continuous domains

A standard discrete attention mechanism uses a value matrix \(V\in\mathbb R^{D\times L}\), computes scores on \(L\) positions, normalizes them, and returns \(c=Vp\). In the continuous setting, the domain is continuous, such as \(S\subseteq\mathbb R\) for time or \(S\subseteq\mathbb R^2\) for the image plane. The values become a function \(V:S\to\mathbb R^D\), the scores become a function \(f:S\to\mathbb R\), the attention becomes a density \(p(t)\), and the context is
\[
c=\mathbb E_p[V(t)]=\int_S p(t)V(t)\,d\nu(t).
\]
This replaces token-level weighted sums by expectations over a continuous domain [2006.07214].

The practical parameterization in the original construction uses basis functions:
\[
f_\theta(t)=\theta^\top\phi(t),
\qquad
V_B(t)=B\psi(t), \quad B\in\mathbb R^{D\times N}.
\]
Attention is then built in three steps: compute \(p(t)=\hat p_\Omega[f_\theta](t)\); compute basis expectations \(r=\mathbb E_p[\psi(t)]\); and return the context \(c=Br\). When \(\Omega=\Omega_\alpha\), the resulting map is continuous \(\alpha\)-entmax attention, denoted \(\rho_\alpha\) [2006.07214].

The most explicit instances occur at \(\alpha=1\) and \(\alpha=2\). For \(\alpha=1\), if \(S=\mathbb R^D\), \(\phi(t)=[t,\mathrm{vec}(tt^\top)]\), and
\[
f_\theta(t)=-\tfrac12(t-\mu)^\top\Sigma^{-1}(t-\mu),
\]
then the density is a multivariate Gaussian,
\[
p(t)=\mathcal N(t;\mu,\Sigma).
\]
For \(\alpha=2\), the map simplifies to thresholding,
\[
\hat p_{\Omega_2}[f](t)=[f(t)-\lambda]_+,
\qquad
\int_S [f(t)-\lambda]_+ = 1,
\]
so quadratic score functions yield truncated quadratics: on \(S=\mathbb R\), a truncated parabola \(\mathrm{TP}(t;\mu,\sigma^2)\); on \(S=\mathbb R^N\), a truncated paraboloid supported on an ellipsoid centered at \(\mu\) [2006.07214].

The support geometry is one of the defining distinctions from continuous softmax attention. In 1D and 2D quadratic cases, the support becomes a compact interval or an ellipse. As \(\alpha\) increases above \(1\), the attentional bump becomes more compact, focusing on smaller regions. This suggests a direct geometric interpretation of sparse continuous attention as attention to intervals, ellipses, or other compact regions rather than to isolated discrete indices [2006.07214].

A later extension generalized these unimodal continuous deformed exponential families to kernel deformed exponential families, replacing finite-dimensional score functions by RKHS functions and allowing multimodal, possibly disconnected compact support. That work showed that kernel deformed exponential families can attend to multiple compact regions of the data domain [2111.01222].

## 4. Differentiation, optimization, and efficient computation

Training continuous attention requires derivatives of \(\rho_\alpha\) with respect to the score parameters. The central Jacobian identity is written in terms of a generalized \(\beta\)-covariance:
\[
\mathrm{cov}_{p,\beta}[\phi(t),\psi(t)]
=
\|p\|_\beta^\beta
\Big(
\mathbb E_{\tilde p_\beta}[\phi(t)\psi(t)^\top]
-
\mathbb E_{\tilde p_\beta}[\phi(t)]\,\mathbb E_{\tilde p_\beta}[\psi(t)]^\top
\Big),
\]
where \(\tilde p_\beta\) is the \(\beta\)-escort distribution. For continuous \(\alpha\)-entmax attention,
\[
J_{\rho_\alpha}(\theta)=\frac{\partial \rho_\alpha(\theta)}{\partial \theta}
=
\mathrm{cov}_{p,\,2-\alpha}\bigl(\phi(t),\psi(t)\bigr).
\]
This is the continuous analogue of the familiar discrete softmax and sparsemax Jacobians: derivatives become moments or generalized covariances of basis functions under the relevant escort distribution [2006.07214].

For \(\alpha=1\) with Gaussian densities and Gaussian RBF value bases, both forward and backward passes are available in closed form. If \(p=\mathcal N(\mu,\Sigma)\) and \(\psi_j(t)=\mathcal N(t;\mu_j,\Sigma_j)\), then
\[
r_j=\mathbb E_p[\psi_j(t)]
=
\mathcal N(\mu;\mu_j,\Sigma+\Sigma_j),
\]
and the covariance terms needed for backpropagation can also be written analytically. For \(\alpha=2\), the density is a truncated parabola or paraboloid; in 1D, the required integrals with Gaussian RBFs have closed form in terms of the error function \(\mathrm{erf}\), while in 2D they reduce, after a change of variables, to simple 1D integrals over radius [2006.07214].

This tractability was originally established for \(\alpha\in\{1,2\}\) in 1D and 2D domains. Later work extended the same program to sparse continuous distributions and Fenchel–Young losses for \(\alpha\in\{1,4/3,3/2,2\}\), deriving efficient gradient backpropagation algorithms for those values and identifying the resulting densities with \(\beta\)-Gaussians that include the Gaussian, biweight, triweight, and Epanechnikov families as particular cases [2108.01988].

A common source of ambiguity is the phrase “continuous \(\alpha\).” In continuous-domain attention, “continuous” refers to the domain \(S\) and the density \(p(t)\). In adaptively sparse Transformer work, by contrast, “continuous” often refers to learning \(\alpha\) itself as a continuous parameter per head. Both use the same entmax family, but they address different objects: continuous measure spaces in one case, discrete token sets with learnable sparsity shape in the other [1909.00015].

## 5. Empirical behavior, interpretability, and limitations

The original empirical studies cover attention-based text classification, machine translation, and visual question answering. On IMDB, comparing discrete, continuous, and combined attention, the reported accuracies are: discrete softmax \(90.78\%\), discrete sparsemax \(90.58\%\), and combined discrete + continuous sparsemax up to \(91.18\%\) accuracy with \(N=64\), with no extra neural-network parameters versus the discrete baseline. Continuous softmax and sparsemax alone were comparable even when using many fewer RBFs than tokens, for example \(N=32,64,128\) versus average \(L\approx 280\). The interpretation given in the experiments is that continuous attention provides a smooth, global view of the document, capturing segment-level focus, and that sparsemax variants tend to be slightly better than softmax [2006.07214].

On IWSLT17 De\(\to\)En machine translation, the baseline with discrete attention achieved BLEU \(=23.92\), combined discrete + continuous softmax reached \(24.00\), and combined discrete + continuous sparsemax reached \(24.25\). Qualitative attention maps show that continuous Gaussian attention tends to center near the regular alignment diagonal and adjust its variance according to local uncertainty, while continuous sparsemax often focuses tightly on segments, matching phrase-level alignment intuition [2006.07214].

On VQA-v2 with MCAN and \(14\times 14\) grid features, the overall Test-dev accuracies were \(65.83\) for discrete softmax attention, \(65.96\) for 2D continuous softmax, and \(65.79\) for 2D continuous sparsemax. The qualitative interpretation is sharper than the numerical gap: discrete attention is sometimes spread over multiple, possibly irrelevant regions; continuous Gaussian attention creates a single elliptical focus; and continuous sparsemax places an ellipse exactly over the relevant object or region, with sharp boundaries determined by the support of the truncated paraboloid [2006.07214].

The same framework also yielded interval-style sparse continuous distributions outside attention proper. In daily temperature interval prediction with Fenchel–Young losses, the truncated parabola (\(\alpha=2\)) achieved the best tradeoff, with lowest MSE and highest Jaccard similarity on predicted versus true intervals. This suggests that sparse continuous densities are particularly suitable when the target object is itself an interval or compact region rather than a diffuse density [2006.07214].

The limitations are explicit. The paper focuses on 1D and 2D Euclidean domains. For \(\alpha=1\), closed-form formulas are available with Gaussian RBFs; for \(\alpha=2\) in 2D, one-dimensional numerical integration is still required. The attention families used in the experiments are single-bump distributions, so they are intrinsically unimodal: one contiguous interval in 1D or one ellipse in 2D. Multi-modal attention would require mixtures, multiple heads, or iterative or recursive attention. A common misconception is therefore to equate continuous \(\alpha\)-entmax attention with arbitrary continuous sparse attention; in the original construction, the implemented families are sparse and continuous, but mostly unimodal [2006.07214].

## 6. Relation to later entmax research and broader interpretation

Continuous \(\alpha\)-entmax attention occupies a specific place within the broader entmax literature. One direction stayed in discrete domains and studied learnable sparsity in Transformers, introducing per-head \(\alpha\in(1,2)\) so that some heads could remain dense while others became sparse. That work emphasized head specialization, higher Jensen–Shannon divergence across heads, and no cost in accuracy relative to softmax on machine translation benchmarks [1909.00015]. A different direction connected \(\alpha\)-entmax to compact-support kernels, showing that sparsemax and more general \(\alpha\)-entmax transformations correspond to Epanechnikov, biweight, and triweight kernel regression under suitable normalization; this supplies a kernel-theoretic interpretation of why sparsity appears as bounded support [2601.22766].

Within the continuous-domain line itself, later work expanded both expressivity and theory. Kernel deformed exponential families replaced low-dimensional score functions by RKHS functions and showed that sparse continuous attention can be multimodal, with support on multiple non-overlapping intervals or blobs, while retaining the Tsallis-regularized variational formulation [2111.01222]. Another related development introduced sparse continuous distributions and Fenchel–Young losses on arbitrary domains, proving that minimization of Fenchel–Young losses is equivalent to moment matching of the statistics for linearly parametrized families and extending efficient continuous attention to additional \(\alpha\) values [2108.01988].

A further, conceptually adjacent literature used discrete \(\alpha\)-entmax to address long-context dispersion and over-squashing. In that setting, Adaptive-Scalable Entmax endowed \(\alpha\)-entmax with a learnable temperature parameter and showed that sparse attention can avoid attention dispersion and representational collapse on long-context generalization tasks. This suggests that the same Tsallis-based sparsity mechanism can be used either to define continuous support in time and space or to stabilize selective attention over very long token sequences [2506.16640].

Taken together, these developments establish continuous \(\alpha\)-entmax attention as a measure-theoretic extension of sparse attention in which deformed exponential families, escort distributions, and Tsallis statistics provide the mathematical core; quadratic or kernelized score functions provide tractable families of densities; and sparsity is expressed as compact, varying support in continuous domains. Its defining contribution is not merely to replace a finite vector of attention weights by a density, but to do so in a way that preserves exact zeros, admits analytic or low-dimensional gradient formulas in important cases, and makes “attending to intervals and compact regions” a first-class operation in neural attention models [2006.07214].

Source: https://www.emergentmind.com/topics/continuous-entmax-attention