---
title: Sparsemax Pointwise Attention in Neural Networks
url: https://www.emergentmind.com/topics/sparsemax-pointwise-attention
type: topic
---

# Sparsemax Pointwise Attention in Neural Networks

Sparsemax pointwise attention refers to a family of attention transformations in neural networks where, at each application, the transformation maps an unnormalized score vector to a sparse probability distribution. Unlike the canonical softmax, which assigns nonzero weights to all positions, sparsemax outputs have exact zeros—performing hard selection of the most salient elements. This property leads to improved interpretability, potential efficiency gains, and, when combined with additional constraints or structure, it can enable coverage control and better model behavior in various applications.

## 1. Definition and Mathematical Formulation

Sparsemax is the Euclidean projection of a real-valued vector onto the probability simplex. Given an attention score vector $z \in \mathbb{R}^K$, the sparsemax mapping is
\[
\text{sparsemax}(z) = \arg\min_{p \in \Delta^{K-1}} \|p - z\|^2, \quad \text{where } \Delta^{K-1} = \{p \in \mathbb{R}^K: p_i \geq 0,\, \sum_{i=1}^K p_i = 1\}.
\]
The closed-form for each coordinate is
\[
[\text{sparsemax}(z)]_i = \max\{0,\, z_i - \tau(z)\},
\]
where $\tau(z)$ is a threshold chosen so $\sum_i \max\{0, z_i - \tau(z)\} = 1$. Computation of $\tau(z)$ requires sorting $z$ and identifying the largest $k$ for which $z_{(k)} - \frac{1}{k}\left(\sum_{j=1}^k z_{(j)} - 1\right) > 0$ with $z_{(1)} \ge \dots \ge z_{(K)}$; then $\tau(z) = (\sum_{j=1}^k z_{(j)} - 1)/k$ [1602.02068][2002.05556].

This hard-projection property yields sparse outputs: only the most significant entries in $z$ contribute nonzero mass, with the number of nonzeros determined adaptively by the support threshold.

## 2. Forward, Backward, and Computational Properties

The typical workflow for sparsemax pointwise attention is:
1. Compute raw attention scores (e.g., via a dot product $q^\top k_i$ or any other scoring function).
2. Apply sparsemax to obtain normalized weights.
3. Form the weighted sum of values (attended representation).

**Forward Pass**: Sorting dominates the computational complexity at $O(K \log K)$, but selection algorithms can achieve $O(K)$ expected time, especially practical for moderate $K$.

**Backward Pass**: The gradient (Jacobian) with respect to the input scores is sparse. For support $S = \{i : p_i > 0\},\, m = |S|$:
\[
\frac{\partial p_i}{\partial z_j} =
\begin{cases}
1 - 1/m & \text{if } i = j \in S, \\
-1/m    & \text{if } i \neq j,\, i, j \in S, \\
0       & \text{otherwise}.
\end{cases}
\]
Consequently, during backpropagation, nonzero gradients are propagated only along the non-pruned positions—yielding potential reductions in computational and memory requirements [1602.02068][2006.07214].

**Comparison with Softmax**: Softmax is $O(K)$ (no sort), always dense, and differentiable everywhere. Sparsemax is piecewise linear, 1-Lipschitz, and has sublinear backward cost in the support size [1705.07704][2002.05556].

## 3. Extensions: Constrained and Structured Sparsemax

Sparsemax can be extended to include upper bounds on the probability mass available per position—addressing coverage and fertility constraints. The *constrained sparsemax* (csparsemax) maps $z \in \mathbb{R}^K$ and an upper-bound vector $u \in \mathbb{R}_+^K$:
\[
\text{csparsemax}(z; u) = \arg\min_{p \in \Delta^{K-1},\, 0 \le p \le u} \|p - z\|^2
\]
The solution remains unique and admits a thresholded form
\[
p_i = \max\{0,\,\min\{u_i,\, z_i - \tau\}\}
\]
with the scalar $\tau$ chosen such that $\sum_i p_i = 1$. Rigorous optimization procedures, e.g., median-finding algorithms for breakpoints, enable $O(K)$-time implementations [1805.08241].

For structured attention, total-variation penalties (e.g., TVmax) are added to promote group or segment sparsity. The solution combines a proximal step for structure (e.g., spatial contiguity) and projection onto the simplex, composing sparsemax with structured sparsity [2002.05556][1705.07704].

## 4. Implementation in Neural Attention Models

Sparsemax is a drop-in replacement for softmax in pointwise (dot-product) attention. Formally, for queries $Q \in \mathbb{R}^{L \times D}$, keys $K \in \mathbb{R}^{N \times D}$, values $V \in \mathbb{R}^{N \times D}$:
\[
Z = QK^\top,\quad A = \text{row-wise sparsemax}(Z), \quad R = AV
\]
In hierarchical, multi-head, or convolutional attention, sparsemax can be applied at word-level, channel-level, or over spatial regions. In constrained variants, fertility vectors are updated online based on previous attentions, absorbing attention excess in a sink node to manage normalization [1805.08241][2107.00178][2004.04343].

Practical deep learning libraries have native support for sparsemax; otherwise, implementation requires careful sort/thresh routines and O($|\text{support}|$) backward passes.

## 5. Empirical Applications and Behavior

Sparsemax-based pointwise attention has been applied in several settings:
- **Machine Translation**: Constrained sparsemax incorporated into NMT prevents repeated/undertranslation. On De–En, csparsemax attains BLEU ≈ 29.85 (vs. softmax ≈ 29.51), lower REP (2.67 vs. 3.37), and lower DROP (5.23% vs. 5.89%) [1805.08241].
- **Text Classification**: In sentiment classification, sparsemax-attention models induce ∼40% word-level sparsity, achieving slightly lower accuracy than softmax but with increased interpretability [2004.04343].
- **Speaker Verification**: In ad-hoc arrays, sparsemax excels at removing weights from noisy microphone channels, leading to 3–6% relative EER gains over already strong softmax-attention baselines [2107.00178].
- **Visual Question Answering**: Sparsemax attention on spatial grids selects only the most relevant regions, improving interpretability and matching human-like attention patterns [2002.05556].
- **Kernel Regression Perspective**: There is a formal correspondence between sparsemax and Epanechnikov kernel regression with adaptive normalization. Sparse attention mechanisms generalize principle alternatives to heuristic top-$k$ selection [2601.22766].

A notable property is that sparsemax produces "hard" sparsity—only the top-$k$ entries per query receive nonzero mass, with $k$ determined textually. This contrasts with softmax, where all entries are nonzero and often difficult to interpret.

## 6. Theoretical Foundations and Generalizations

Sparsemax is a special case of the $\alpha$-entmax family (specifically, $\alpha=2$), which interpolates between softmax ($\alpha=1$) and polynomially sparse activations (e.g., biweight at $\alpha=1.5$). The general $\alpha$-entmax has closed-form:
\[
\text{entmax}_\alpha(z)_i = \left[((\alpha-1)z_i - \tau)_+\right]^{1/(\alpha-1)}
\]
Sparsemax realizes the $\ell_2$-projection; its support adapts to the input nonlinearly, producing hard zeros [1909.00015][2006.07214]. This yields a bridge between convex duality, Tsallis entropy regularization, and deformed exponential families.

Furthermore, from a variational viewpoint, sparsemax is the gradient of the maximization of a linear form minus squared $\ell_2$ norm over the simplex—a smoothed approximation of the max-operator distinct from softmax, whose smoothing is through negative entropy [1705.07704].

## 7. Interpretability, Design Implications, and Related Variants

Sparsemax enables direct inspection of which tokens, channels, or regions the model "attended to," since zeros in the output map correspond to pruned, ignored locations. This effect improves transparency especially in structured or explainable settings.

However, sparsemax can at times collapse multimodal attention into single-mode, depending on the input. Other variants, such as MultiMax [2406.01189], introduce piecewise-linear modulators to achieve a trade-off between sparsity and multimodality, and $\alpha$-entmax allows for soft interpolation between behaviors.

Sparsemax can be further modified for additional constraints (e.g., upper bounds, structured penalties) or generalized to continuous domains, in which case density functions respect support constraints analogous to their discrete counterparts [2006.07214].

---

**References**:  
- [1602.02068] "From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification"  
- [2002.05556] "Sparse and Structured Visual Attention"  
- [1705.07704] "A Regularized Framework for Sparse and Structured Neural Attention"  
- [1805.08241] "Sparse and Constrained Attention for Neural Machine Translation"  
- [1909.00015] "Adaptively Sparse Transformers"  
- [2107.00178] "Attention-based multi-channel speaker verification with ad-hoc microphone arrays"  
- [2006.07214] "Sparse and Continuous Attention Mechanisms"  
- [2004.04343] "Pruning and Sparsemax Methods for Hierarchical Attention Networks"  
- [2601.22766] "Sparse Attention as Compact Kernel Regression"  
- [2406.01189] "MultiMax: Sparse and Multi-Modal Attention Learning"

Source: https://www.emergentmind.com/topics/sparsemax-pointwise-attention