Continuous α-Entmax Attention

• Continuous α-entmax attention is a generalization of discrete softmax and sparsemax, replacing token-level weights with probability densities over continuous measure spaces.
• It employs Tsallis-α negentropy and deformed exponential functions to generate sparse, compact-support attention maps that balance smoothness with selective focus.
• The method is applied across text, image, and temporal domains and offers closed-form solutions and efficient gradient computations for certain α values, enhancing interpretability and performance.

Continuous α-entmax attention is a continuous-domain generalization of discrete α-entmax and sparsemax attention in which attention is represented by a probability density $p(t)$ on a measure space $S$, obtained from a regularized prediction map with Tsallis-$\alpha$ negentropy, and used to compute a context vector as a continuous expectation $c=\int_S p(t)V(t)\,d\nu(t)$. For $\alpha=1$ it recovers continuous softmax attention; for $\alpha>1$ it yields densities with varying, possibly compact support; and for $\alpha=2$ it becomes continuous sparsemax, producing truncated quadratic densities such as truncated parabolas and truncated paraboloids (Martins et al., 2020).

1. Discrete origins and variational formulation

In the discrete case, attention over a finite set $S=\{1,\dots,L\}$ starts from a score vector $f\in\mathbb{R}^L$. Softmax maps scores to probabilities by

$$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$

and therefore always produces dense distributions. Sparse alternatives such as sparsemax and $S$0-entmax arise from the framework of regularized prediction maps (RPMs): for a strictly convex regularizer $S$1,

$S$2

When $S$3 is the Tsallis-$S$4 negentropy,

$S$5

the resulting RPM is the discrete $S$6-entmax transformation (Martins et al., 2020).

The special cases organize the family. At $S$7, the Shannon negentropy is recovered and the RPM gives softmax. At $S$8, the map becomes sparsemax,

$S$9

that is, Euclidean projection onto the simplex. Intermediate values $\alpha$0 interpolate between softmax and sparsemax, controlling sparsity. In this discrete setting, varying support means that the number and identity of indices with nonzero probability change with the scores $\alpha$1 or the parameter $\alpha$2. The practical significance is twofold: only a few positions may be truly attended, and the attention map can encode a structural prior toward compact regions or few tokens (Martins et al., 2020).

A related but distinct line of work later used $\alpha$3-entmax in standard Transformer attention and learned $\alpha$4 per head, showing that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers; this literature concerns discrete attention over tokens rather than continuous domains (Correia et al., 2019).

2. Continuous extension via Tsallis-regularized prediction maps

The continuous construction replaces the finite domain by an arbitrary measure space $\alpha$5. A probability distribution is represented by a density $\alpha$6, meaning $\alpha$7 and $\alpha$8. For a strictly convex functional $\alpha$9, the continuous RPM is

$c=\int_S p(t)V(t)\,d\nu(t)$0

This turns attention normalization into a functional optimization problem over densities rather than a finite-dimensional projection over the simplex (Martins et al., 2020).

The construction uses the $c=\int_S p(t)V(t)\,d\nu(t)$1-logarithm and $c=\int_S p(t)V(t)\,d\nu(t)$2-exponential,

$c=\int_S p(t)V(t)\,d\nu(t)$3

The continuous Tsallis-$c=\int_S p(t)V(t)\,d\nu(t)$4 negentropy is

$c=\int_S p(t)V(t)\,d\nu(t)$5

For $c=\int_S p(t)V(t)\,d\nu(t)$6, this functional is strictly convex, so the continuous RPM is well defined and unique (Martins et al., 2020).

The resulting solution has the deformed-exponential form

$c=\int_S p(t)V(t)\,d\nu(t)$7

where $c=\int_S p(t)V(t)\,d\nu(t)$8 is the normalizing scalar enforcing unit mass. When $c=\int_S p(t)V(t)\,d\nu(t)$9, this becomes the Boltzmann-Gibbs density

$\alpha=1$0

When $\alpha=1$1, the deformed exponential is truncated, so the density is exactly zero wherever the bracket is negative. In the continuous setting, sparsity is therefore expressed as compact or varying support rather than as a finite set of exact zero coordinates (Martins et al., 2020).

With a linear score parametrization $\alpha=1$2, these densities form a deformed exponential family, or $\alpha=1$3-exponential / $\alpha=1$4-exponential family. For $\alpha=1$5 they are called $\alpha=1$6-sparse families. A central auxiliary object is the $\alpha=1$7-escort distribution,

$\alpha=1$8

which appears in derivatives of the normalizer and in Jacobians of the attention map (Martins et al., 2020).

3. Attention mechanism on continuous domains

A standard discrete attention mechanism uses a value matrix $\alpha=1$9, computes scores on $\alpha>1$0 positions, normalizes them, and returns $\alpha>1$1. In the continuous setting, the domain is continuous, such as $\alpha>1$2 for time or $\alpha>1$3 for the image plane. The values become a function $\alpha>1$4, the scores become a function $\alpha>1$5, the attention becomes a density $\alpha>1$6, and the context is

$\alpha>1$7

This replaces token-level weighted sums by expectations over a continuous domain (Martins et al., 2020).

The practical parameterization in the original construction uses basis functions: $\alpha>1$8 Attention is then built in three steps: compute $\alpha>1$9; compute basis expectations $\alpha=2$0; and return the context $\alpha=2$1. When $\alpha=2$2, the resulting map is continuous $\alpha=2$3-entmax attention, denoted $\alpha=2$4 (Martins et al., 2020).

The most explicit instances occur at $\alpha=2$5 and $\alpha=2$6. For $\alpha=2$7, if $\alpha=2$8, $\alpha=2$9, and

$S=\{1,\dots,L\}$0

then the density is a multivariate Gaussian,

$S=\{1,\dots,L\}$1

For $S=\{1,\dots,L\}$2, the map simplifies to thresholding,

$S=\{1,\dots,L\}$3

so quadratic score functions yield truncated quadratics: on $S=\{1,\dots,L\}$4, a truncated parabola $S=\{1,\dots,L\}$5; on $S=\{1,\dots,L\}$6, a truncated paraboloid supported on an ellipsoid centered at $S=\{1,\dots,L\}$7 (Martins et al., 2020).

The support geometry is one of the defining distinctions from continuous softmax attention. In 1D and 2D quadratic cases, the support becomes a compact interval or an ellipse. As $S=\{1,\dots,L\}$8 increases above $S=\{1,\dots,L\}$9, the attentional bump becomes more compact, focusing on smaller regions. This suggests a direct geometric interpretation of sparse continuous attention as attention to intervals, ellipses, or other compact regions rather than to isolated discrete indices (Martins et al., 2020).

A later extension generalized these unimodal continuous deformed exponential families to kernel deformed exponential families, replacing finite-dimensional score functions by RKHS functions and allowing multimodal, possibly disconnected compact support. That work showed that kernel deformed exponential families can attend to multiple compact regions of the data domain (Moreno et al., 2021).

4. Differentiation, optimization, and efficient computation

Training continuous attention requires derivatives of $f\in\mathbb{R}^L$0 with respect to the score parameters. The central Jacobian identity is written in terms of a generalized $f\in\mathbb{R}^L$1-covariance: $f\in\mathbb{R}^L$2 where $f\in\mathbb{R}^L$3 is the $f\in\mathbb{R}^L$4-escort distribution. For continuous $f\in\mathbb{R}^L$5-entmax attention,

$f\in\mathbb{R}^L$6

This is the continuous analogue of the familiar discrete softmax and sparsemax Jacobians: derivatives become moments or generalized covariances of basis functions under the relevant escort distribution (Martins et al., 2020).

For $f\in\mathbb{R}^L$7 with Gaussian densities and Gaussian RBF value bases, both forward and backward passes are available in closed form. If $f\in\mathbb{R}^L$8 and $f\in\mathbb{R}^L$9, then

$$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$0

and the covariance terms needed for backpropagation can also be written analytically. For $$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$1, the density is a truncated parabola or paraboloid; in 1D, the required integrals with Gaussian RBFs have closed form in terms of the error function $$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$2, while in 2D they reduce, after a change of variables, to simple 1D integrals over radius (Martins et al., 2020).

This tractability was originally established for $$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$3 in 1D and 2D domains. Later work extended the same program to sparse continuous distributions and Fenchel–Young losses for $$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$4, deriving efficient gradient backpropagation algorithms for those values and identifying the resulting densities with $$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$5-Gaussians that include the Gaussian, biweight, triweight, and Epanechnikov families as particular cases (Martins et al., 2021).

A common source of ambiguity is the phrase “continuous $$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$6.” In continuous-domain attention, “continuous” refers to the domain $$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$7 and the density $$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$8. In adaptively sparse Transformer work, by contrast, “continuous” often refers to learning $$\mathrm{softmax}(f)_k=\frac{e^{f_k}}{\sum_{j=1}^L e^{f_j}},$$9 itself as a continuous parameter per head. Both use the same entmax family, but they address different objects: continuous measure spaces in one case, discrete token sets with learnable sparsity shape in the other (Correia et al., 2019).

5. Empirical behavior, interpretability, and limitations

The original empirical studies cover attention-based text classification, machine translation, and visual question answering. On IMDB, comparing discrete, continuous, and combined attention, the reported accuracies are: discrete softmax $S$00, discrete sparsemax $S$01, and combined discrete + continuous sparsemax up to $S$02 accuracy with $S$03, with no extra neural-network parameters versus the discrete baseline. Continuous softmax and sparsemax alone were comparable even when using many fewer RBFs than tokens, for example $S$04 versus average $S$05. The interpretation given in the experiments is that continuous attention provides a smooth, global view of the document, capturing segment-level focus, and that sparsemax variants tend to be slightly better than softmax (Martins et al., 2020).

On IWSLT17 De$S$06En machine translation, the baseline with discrete attention achieved BLEU $S$07, combined discrete + continuous softmax reached $S$08, and combined discrete + continuous sparsemax reached $S$09. Qualitative attention maps show that continuous Gaussian attention tends to center near the regular alignment diagonal and adjust its variance according to local uncertainty, while continuous sparsemax often focuses tightly on segments, matching phrase-level alignment intuition (Martins et al., 2020).

On VQA-v2 with MCAN and $S$10 grid features, the overall Test-dev accuracies were $S$11 for discrete softmax attention, $S$12 for 2D continuous softmax, and $S$13 for 2D continuous sparsemax. The qualitative interpretation is sharper than the numerical gap: discrete attention is sometimes spread over multiple, possibly irrelevant regions; continuous Gaussian attention creates a single elliptical focus; and continuous sparsemax places an ellipse exactly over the relevant object or region, with sharp boundaries determined by the support of the truncated paraboloid (Martins et al., 2020).

The same framework also yielded interval-style sparse continuous distributions outside attention proper. In daily temperature interval prediction with Fenchel–Young losses, the truncated parabola ($S$14) achieved the best tradeoff, with lowest MSE and highest Jaccard similarity on predicted versus true intervals. This suggests that sparse continuous densities are particularly suitable when the target object is itself an interval or compact region rather than a diffuse density (Martins et al., 2020).

The limitations are explicit. The paper focuses on 1D and 2D Euclidean domains. For $S$15, closed-form formulas are available with Gaussian RBFs; for $S$16 in 2D, one-dimensional numerical integration is still required. The attention families used in the experiments are single-bump distributions, so they are intrinsically unimodal: one contiguous interval in 1D or one ellipse in 2D. Multi-modal attention would require mixtures, multiple heads, or iterative or recursive attention. A common misconception is therefore to equate continuous $S$17-entmax attention with arbitrary continuous sparse attention; in the original construction, the implemented families are sparse and continuous, but mostly unimodal (Martins et al., 2020).

6. Relation to later entmax research and broader interpretation

Continuous $S$18-entmax attention occupies a specific place within the broader entmax literature. One direction stayed in discrete domains and studied learnable sparsity in Transformers, introducing per-head $S$19 so that some heads could remain dense while others became sparse. That work emphasized head specialization, higher Jensen–Shannon divergence across heads, and no cost in accuracy relative to softmax on machine translation benchmarks (Correia et al., 2019). A different direction connected $S$20-entmax to compact-support kernels, showing that sparsemax and more general $S$21-entmax transformations correspond to Epanechnikov, biweight, and triweight kernel regression under suitable normalization; this supplies a kernel-theoretic interpretation of why sparsity appears as bounded support (Santos et al., 30 Jan 2026).

Within the continuous-domain line itself, later work expanded both expressivity and theory. Kernel deformed exponential families replaced low-dimensional score functions by RKHS functions and showed that sparse continuous attention can be multimodal, with support on multiple non-overlapping intervals or blobs, while retaining the Tsallis-regularized variational formulation (Moreno et al., 2021). Another related development introduced sparse continuous distributions and Fenchel–Young losses on arbitrary domains, proving that minimization of Fenchel–Young losses is equivalent to moment matching of the statistics for linearly parametrized families and extending efficient continuous attention to additional $S$22 values (Martins et al., 2021).

A further, conceptually adjacent literature used discrete $S$23-entmax to address long-context dispersion and over-squashing. In that setting, Adaptive-Scalable Entmax endowed $S$24-entmax with a learnable temperature parameter and showed that sparse attention can avoid attention dispersion and representational collapse on long-context generalization tasks. This suggests that the same Tsallis-based sparsity mechanism can be used either to define continuous support in time and space or to stabilize selective attention over very long token sequences (Vasylenko et al., 19 Jun 2025).

Taken together, these developments establish continuous $S$25-entmax attention as a measure-theoretic extension of sparse attention in which deformed exponential families, escort distributions, and Tsallis statistics provide the mathematical core; quadratic or kernelized score functions provide tractable families of densities; and sparsity is expressed as compact, varying support in continuous domains. Its defining contribution is not merely to replace a finite vector of attention weights by a density, but to do so in a way that preserves exact zeros, admits analytic or low-dimensional gradient formulas in important cases, and makes “attending to intervals and compact regions” a first-class operation in neural attention models (Martins et al., 2020).