Entmax-1.5: Sparse Differentiable Transformation

• Entmax-1.5 is a specialized α-entmax transformation (with α=1.5) that applies a squared-threshold function to assign exact zeros to low-scoring logits.
• It is employed in transformer attention and seq2seq output layers to yield sparse activations that reduce search space and alleviate issues like attention dispersion.
• Its variational formulation and Fenchel-Young loss structure provide clear gradient signals, while computational surrogates like 1.5-ReLU and AdaSplash optimize performance.

Searching arXiv for foundational and papers on Entmax-1.5 and related sparse attention. Entmax-1.5 is the $\alpha$-entmax transformation specialized to $\alpha=1.5$, used to map a logit vector to a normalized distribution with exact zeros. In the sources considered here, it occupies an intermediate regime between the dense softmax limit and the sparsemax case, and appears in several distinct roles: as a row-wise attention normalization in transformers, as an output-layer distribution and training loss in seq2seq models, as the fixed sparse baseline against which adaptive variants are compared, and as a sparse weighting mechanism outside sequence modeling. Its defining practical property is support selection by thresholding: low-scoring coordinates are assigned probability exactly zero, while the surviving coordinates are normalized through a squared-threshold form (Vasylenko et al., 19 Jun 2025, Peters et al., 2021, Tezekbayev et al., 2021).

1. Mathematical definition and equivalent parameterizations

For $\alpha>1$, recent transformer work defines $\alpha$-entmax as

$$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$

with $\tau(\bm z)$ chosen so that the output sums to $1$. In attention, this map is applied row-wise to a query’s score vector, typically with logits

$$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$

Setting $\alpha=1.5$ yields

$$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$

The support condition becomes

$\alpha=1.5$0

so Entmax-1.5 keeps only coordinates above an adaptive threshold and sets the rest exactly to zero (Vasylenko et al., 19 Jun 2025).

A seq2seq treatment writes the same family through the Tsallis/$\alpha=1.5$1-exponential: $\alpha=1.5$2 and for $\alpha=1.5$3 expands it as

$\alpha=1.5$4

Under this parameterization, the $\alpha=1.5$5 case is written as

$\alpha=1.5$6

This suggests that the apparent discrepancy between the two formulas is a difference in threshold parameterization rather than a change in the underlying map (Peters et al., 2021).

The practically important specialization to $\alpha=1.5$7 also admits a closed-form threshold expression once the support size $\alpha=1.5$8 is known. With sorted logits $\alpha=1.5$9,

$\alpha>1$0

where

$\alpha>1$1

The valid support size satisfies

$\alpha>1$2

This explicit structure is mathematically attractive, but it also exposes the computational difficulty: identifying $\alpha>1$3 requires information from sorted logits (Tezekbayev et al., 2021).

2. Variational form, losses, and differential structure

A variational definition used in the long-context attention literature is

$\alpha>1$4

where $\alpha>1$5 is the Tsallis$\alpha>1$6 entropy. The same family is also presented through Fenchel-Young regularized prediction: $\alpha>1$7 with entmax recovered by taking $\alpha>1$8 to be the negative Tsallis entropy. For $\alpha>1$9,

$\alpha$0

These formulations are important because they make Entmax-1.5 not merely a decoding transform but also the prediction map associated with a specific training loss (Vasylenko et al., 19 Jun 2025, Peters et al., 2021).

The corresponding Fenchel-Young loss is

$\alpha$1

with gradient

$\alpha$2

For Entmax-1.5 this gives the familiar

$\alpha$3

The same framework supports label smoothing by replacing the one-hot target with $\alpha$4, yielding

$\alpha$5

and gradient

$\alpha$6

One motivation given for this Fenchel-Young smoothing is that vanilla KL-style smoothing is problematic for sparse distributions because KL diverges when the model has zeros and the target does not (Peters et al., 2021).

This gradient structure also motivates later surrogates. A computational paper on sparse output layers notes that when entmax is coupled with the entmax loss, differentiation again produces the usual “prediction minus one-hot target” form, and proposes a non-normalized surrogate designed to preserve that training behavior while avoiding the cost of adaptive thresholding. The surrogate is not exact Entmax-1.5, but its loss is engineered so that the gradient retains the same algebraic pattern (Tezekbayev et al., 2021).

3. Entmax-1.5 in attention and long-context transformers

In transformer attention, Entmax-1.5 is used by replacing the softmax map in

$\alpha$7

with $\alpha$8-entmax, typically applied row-wise to the standard scaled dot-product scores and combined with positional choices such as NoPE, ALiBi, RoPE, or the hybrid NAPE. A recent long-context study states explicitly: “For our experiments with $\alpha$9-entmax, we use $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$0 as the default value for the $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$1-entmax models, unless mentioned otherwise.” In that work, Entmax-1.5 functions primarily as the default fixed sparse transformer attention mechanism and as the baseline recovered by ASEntmax when $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$2 (Vasylenko et al., 19 Jun 2025).

The theoretical case for Entmax-1.5 in long contexts is the general case for $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$3-entmax with $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$4. Softmax attention is dense and must distribute mass over all tokens; entmax can assign exact zeros. The paper emphasizes that this helps avoid attention dispersion, representational collapse, and over-squashing. One lemma establishes that when an extra token is added, softmax must strictly reduce the probability of an existing token, whereas $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$5-entmax may leave that probability unchanged if the new token falls below threshold. The same work argues that with bounded logits softmax entropy tends to $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$6, while $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$7-entmax entropy scales like $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$8 when the support size is $$\alpha\text{-entmax}(\bm z)_i=\left[(\alpha-1)z_i-\tau(\bm z)\right]_+^{\frac{1}{\alpha-1}}, \qquad [\cdot]_+=\max(0,\cdot),$$9, so attention can remain focused even as total sequence length grows. It also reports stronger layerwise gradients, up to $\tau(\bm z)$0 larger than softmax, and gives an ALiBi analysis in which $\tau(\bm z)$1-entmax induces a hard attention window (Vasylenko et al., 19 Jun 2025).

The empirical behavior of fixed Entmax-1.5 in long-context generalization is mixed but often strong. On MQMTAR with NAPE, fixed Entmax scored $\tau(\bm z)$2 at lengths $\tau(\bm z)$3 and $\tau(\bm z)$4, remained at $\tau(\bm z)$5 at $\tau(\bm z)$6, and reached $\tau(\bm z)$7 at $\tau(\bm z)$8K, compared with NAPE softmax at $\tau(\bm z)$9 at $1$0K; however, at $1$1K it scored $1$2 versus softmax’s $1$3. On Reverse with NAPE, fixed Entmax gave $1$4 at length $1$5, $1$6 at $1$7, $1$8 at $1$9, and $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$0 at $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$1, substantially above softmax. On Sort with NAPE, fixed Entmax achieved $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$2 at $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$3 and $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$4 at $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$5, while softmax was already $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$6 from $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$7 onward. By contrast, Copy is the clearest case in which fixed sparse attention is “too sparse”: under NAPE, fixed Entmax fell from $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$8 in-distribution to $$z_{ij}^{(\ell)}=\langle \bm q_i^{(\ell)},\bm k_j^{(\ell)}\rangle/\sqrt d.$$9 by $\alpha=1.5$0 and $\alpha=1.5$1, while softmax was much stronger. On Max Retrieval at $\alpha=1.5$2, fixed Entmax-$\alpha=1.5$3 scored $\alpha=1.5$4; larger fixed $\alpha=1.5$5 values improved concentration up to $\alpha=1.5$6 at $\alpha=1.5$7 and then degraded, while ASEntmax with $\alpha=1.5$8 and $\alpha=1.5$9 reached $$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$0. This suggests that Entmax-1.5 is a useful fixed sparse operating point, but not a universally optimal one across task structure and context length (Vasylenko et al., 19 Jun 2025).

4. Output-layer normalization, search-space shrinking, and seq2seq use

A separate line of work studies Entmax-1.5 as an output-layer replacement for softmax in locally normalized seq2seq models: $$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$1 Here the key claim is that exact zeros shrink the search space. Under softmax, every vocabulary item retains nonzero probability; under Entmax-1.5, many local continuations receive zero probability and can never appear in any hypothesis. The paper frames this as a remedy for length bias and for the “cat got your tongue” pathology, where the empty string is assigned excessively high probability (Peters et al., 2021).

In machine translation, Entmax-1.5 is the only sparse entmax setting evaluated. The paper reports that with $$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$2, the percentage of development examples where the empty string is more probable than the beam hypothesis collapses to near zero: $$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$3 for de$$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$4en, $$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$5 for en$$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$6de, $$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$7 for ja$$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$8en, $$\mathrm{entmax}_{1.5}(\bm z)_i=\left[\frac12 z_i-\tau(\bm z)\right]_+^2, \qquad \sum_i \mathrm{entmax}_{1.5}(\bm z)_i=1.$$9 for en$\alpha=1.5$00ja, $\alpha=1.5$01 for ro$\alpha=1.5$02en, and $\alpha=1.5$03 for en$\alpha=1.5$04ro. By contrast, softmax models exhibit much larger values, and label smoothing can worsen them substantially. The same study reports that unsmoothed Entmax-1.5 beats smoothed softmax on $\alpha=1.5$05 MT directions, while label-smoothed Entmax-1.5 is best on $\alpha=1.5$06, unsmoothed Entmax-1.5 best on $\alpha=1.5$07, and tied on $\alpha=1.5$08. Representative BLEU numbers include de$\alpha=1.5$09en: $\alpha=1.5$10 for Entmax-1.5 versus $\alpha=1.5$11 for smoothed softmax; ro$\alpha=1.5$12en: $\alpha=1.5$13 for smoothed Entmax-1.5 versus $\alpha=1.5$14 for smoothed softmax; and en$\alpha=1.5$15ro: $\alpha=1.5$16 for unsmoothed Entmax-1.5 versus $\alpha=1.5$17 for smoothed softmax (Peters et al., 2021).

The same paper measures how sparse these output distributions are under forced decoding. For Entmax-1.5, the average percentage of the target vocabulary with nonzero probability is $\alpha=1.5$18 to $\alpha=1.5$19 at $\alpha=1.5$20, $\alpha=1.5$21 to $\alpha=1.5$22 at $\alpha=1.5$23, and $\alpha=1.5$24 to $\alpha=1.5$25 at $\alpha=1.5$26, whereas cross-entropy models remain $\alpha=1.5$27 dense. It also reports better calibration than softmax at the same smoothing level. Moderate smoothing improves performance and calibration, but the paper is explicit that smoothing and sparsity trade off: heavier smoothing makes the tail heavier and can reintroduce empty-string problems (Peters et al., 2021).

Entmax-1.5 is also used beyond MT within seq2seq. On multilingual grapheme-to-phoneme conversion, $\alpha=1.5$28 with $\alpha=1.5$29 achieved WER $\alpha=1.5$30 and PER $\alpha=1.5$31, and the ensemble reached WER $\alpha=1.5$32 and PER $\alpha=1.5$33, improving on the previous best ensemble baseline. On crosslingual morphological inflection, unsmoothed Entmax-1.5 reached accuracy $\alpha=1.5$34 and Levenshtein distance $\alpha=1.5$35, while smoothed Entmax-1.5 improved to accuracy $\alpha=1.5$36 and distance $\alpha=1.5$37. The same paper additionally notes that its G2P model uses Entmax-1.5 attention, so the $\alpha=1.5$38 operating point appears in both output normalization and attention within that architecture (Peters et al., 2021).

5. Computation, kernels, and support-aware inference

The principal computational bottleneck of Entmax-1.5 is the adaptive threshold $\alpha=1.5$39. One paper states this directly: “It is this threshold $\alpha=1.5$40 that makes the computation of $\alpha=1.5$41-entmax slow, because one needs to sort the components of $\alpha=1.5$42 to find $\alpha=1.5$43.” In large-vocabulary settings, this overhead is substantial. Rather than proposing a faster exact Entmax-1.5 algorithm, that work introduces the surrogate

$\alpha=1.5$44

where $\alpha=1.5$45 is constant, not input-dependent. The method is therefore not normalized and not exact Entmax-1.5, but is intended to mimic its sparse support and training dynamics. Training time results show the scale of the original bottleneck: on IWSLT'14 En$\alpha=1.5$46De, softmax took $\alpha=1.5$47 h, 1.5-entmax $\alpha=1.5$48 h, and 1.5-ReLU $\alpha=1.5$49 h; on WMT'14 De$\alpha=1.5$50En, softmax took $\alpha=1.5$51 h, 1.5-entmax $\alpha=1.5$52 h, and 1.5-ReLU $\alpha=1.5$53 h. In inference on WMT En$\alpha=1.5$54Ru, softmax is reported as almost twice faster than full-sort 1.5-entmax, while 1.5-ReLU runs at softmax speed (Tezekbayev et al., 2021).

Exact attention-side acceleration is addressed by AdaSplash, which implements $\alpha=1.5$55-entmax attention in Triton and introduces a hybrid Halley-bisection solver for the root problem

$\alpha=1.5$56

in the $\alpha=1.5$57 case. The paper reports a $\alpha=1.5$58-fold reduction in the number of iterations needed to compute $\alpha=1.5$59-entmax and states that for $\alpha=1.5$60 the hybrid method reaches machine precision for output and gradient in $\alpha=1.5$61 iterations, whereas standard bisection needs $\alpha=1.5$62. AdaSplash then combines this solver with Flash-style tiling and a sparsity-aware block mask so that exact zeros can be skipped in forward and backward passes. In GPT-2 training, FlashAttention-2 ran at $\alpha=1.5$63 s/step and $\alpha=1.5$64 GB, while Entmax-1.5 via AdaSplash ran at $\alpha=1.5$65 s/step with the same $\alpha=1.5$66 GB, dramatically better than prior entmax implementations. On long-sequence benchmarks the paper states that AdaSplash approaches, and in some cases surpasses, the efficiency of FlashAttention-2 when sparsity is high (Gonçalves et al., 17 Feb 2025).

AdaSplash-2 pushes the same agenda further with histogram-based initialization of $\alpha=1.5$67, reducing the number of refinement iterations to typically $\alpha=1.5$68--$\alpha=1.5$69. Its central guarantee is

$\alpha=1.5$70

where $\alpha=1.5$71 is the histogram estimate. For $\alpha=1.5$72, the paper emphasizes that the histogram problem can be solved exactly from bin counts without sorting, and that Halley refinement remains stable because $\alpha=1.5$73 is bounded for $\alpha=1.5$74. The reported systems-level outcome is that AdaSplash-2 matches or improves per-step training time relative to FlashAttention-2 when block sparsity is moderate-to-high, for example above $\alpha=1.5$75. All major modeling experiments in that paper use $\alpha=1.5$76, and the strongest long-context results come from Entmax-1.5 paired with NAPE rather than RoPE (Gonçalves et al., 16 Apr 2026).

Decode-time efficiency introduces a different question: not how to compute dense Entmax-1.5 faster, but how to avoid reading irrelevant KV cache pages before sparsity is known. EntmaxKV formulates this as support recovery. For general $\alpha=1.5$77-entmax, if a kept token set $\alpha=1.5$78 contains the true support $\alpha=1.5$79, then sparse decoding is exact; for Entmax-1.5 this means exactness once all indices satisfying $\alpha=1.5$80 are retained. The paper measures error through the dropped mass $\alpha=1.5$81 and proves

$\alpha=1.5$82

with exact equality of sparse and dense outputs when $\alpha=1.5$83. Empirically, EntmaxKV reports better support retention, lower dropped mass, and lower output error than softmax-based sparse decoding at matched KV budgets, and at $\alpha=1.5$84M context length achieves up to $\alpha=1.5$85 speedup over full softmax and $\alpha=1.5$86 over full entmax while closely matching full-cache entmax (Duarte et al., 20 May 2026).

6. Broader uses, operating-point interpretation, and limits

Entmax-1.5 also appears outside attention and output normalization. In Forward-Forward learning, an “entmax-weighted energy” goodness function is defined by

$\alpha=1.5$87

Here $\alpha=1.5$88 recovers softmax, $\alpha=1.5$89 gives sparsemax, and intermediate values produce adaptive sparsity. The paper’s central empirical conclusion is that $\alpha=1.5$90 is best: on Fashion-MNIST with a $\alpha=1.5$91 network, standard FF with Entmax-1.5 energy achieved $\alpha=1.5$92, and FFCL with Entmax-1.5 reached $\alpha=1.5$93, the best overall result in the paper. The corresponding $\alpha=1.5$94-sweep exhibits a clear inverted-U, with $\alpha=1.5$95 outperforming both the dense endpoint $\alpha=1.5$96 and the harder sparse endpoint $\alpha=1.5$97 (Yuksel et al., 28 Mar 2026).

This broader evidence reinforces a recurring interpretation of Entmax-1.5 as a moderate sparse operating point rather than an extremal one. In the FF paper, $\alpha=1.5$98 corresponds to roughly $\alpha=1.5$99--$\alpha>1$00 of neurons receiving nonzero weight, varying per input. In long-context transformers, Entmax-1.5 is repeatedly used as the default fixed sparse baseline, while later methods such as ASEntmax are introduced precisely because fixed sparsity can be either too weak or too strong depending on the task. In machine translation, Entmax-1.5 sharply reduces empty-string pathologies and shrinks the beam-search space, but heavier label smoothing can erode those benefits. In computational work, 1.5-ReLU is proposed as a fast surrogate precisely because exact Entmax-1.5 thresholding remains expensive unless specialized kernels are used (Vasylenko et al., 19 Jun 2025, Peters et al., 2021, Tezekbayev et al., 2021).

Several common misunderstandings are corrected by this recent literature. Entmax-1.5 is not the main novelty in all papers that use it: some papers treat it chiefly as a baseline or default sparse choice, with the primary contribution lying in adaptive scaling, efficient kernels, or support-aware decoding. It is not merely “softmax with fewer nonzeros”: its variational definition, Fenchel-Young loss, and exact support recovery behavior have algorithmic consequences not shared by dense truncation heuristics. Nor is it uniformly superior to either denser or sparser alternatives. The most consistent picture across the cited work is narrower and more technical: Entmax-1.5 is a sparse, differentiable, squared-threshold normalization that often provides a robust middle ground between dispersion and over-hard selection, and whose value becomes especially pronounced when exact zeros can be exploited by the model, the loss, or the system implementation (Vasylenko et al., 19 Jun 2025, Yuksel et al., 28 Mar 2026).