Native Sparse Attention

• Native sparse attention is a technique that produces interpretable, resource-efficient attention distributions by directly replacing dense softmax with sparse alternatives.
• It employs transformations like sparsemax, constrained softmax, and constrained sparsemax to enforce coverage constraints and control over- and under-translation.
• Empirical evaluations in neural machine translation show improved BLEU, METEOR scores along with reduced repetition and dropped source words.

Native sparse attention refers to methods for directly producing sparse, bounded, and interpretable attention distributions within neural models, replacing or augmenting standard dense mechanisms such as softmax. In neural machine translation (NMT), dense softmax attention can lead to dropped source words (under-translation) or excessive repetition (over-translation) due to its inability to focus exclusively or bound the attention mass allocated to each source token. Native sparse attention mechanisms modify the probabilistic transformation underlying attention, yielding sparse distributions and explicit coverage constraints, and can be seamlessly integrated into existing sequence-to-sequence architectures.

1. Sparse and Constrained Attention Mechanisms

The foundational approach replaces the softmax attention transformation, which produces a strictly positive and dense distribution, with functions that promote sparsity and/or bounded allocations:

• Sparsemax: Computes attention as the Euclidean projection of the input scores onto the probability simplex,

$$\text{sparsemax}(\mathbf{z}) := \arg\min_{\boldsymbol{\alpha} \in \Delta^J} \|\boldsymbol{\alpha} - \mathbf{z}\|^2,$$

where $\Delta^J$ is the $J$-dimensional probability simplex. Sparsemax assigns exactly zero attention to many irrelevant source words in each decoding step, yielding hard selection behavior.

• Constrained Softmax: Imposes upper bounds (fertility constraints) per source word,

$$\text{csoftmax}(\mathbf{z}; \mathbf{u}) := \arg\min_{\boldsymbol{\alpha} \in \Delta^J,\, \boldsymbol{\alpha} \leq \mathbf{u}} \mathrm{KL}(\boldsymbol{\alpha} \| \text{softmax}(\mathbf{z})),$$

where $\mathbf{u}$ is a vector of per-word limits. This controls over-translation by bounding how much total attention a word can receive.

• Constrained Sparsemax: Newly introduced, enforces both sparsity and upper bounds,

$$\text{csparsemax}(\mathbf{z}; \mathbf{u}) := \arg\min_{\boldsymbol{\alpha} \in \Delta^J,\, \boldsymbol{\alpha} \leq \mathbf{u}} \|\boldsymbol{\alpha} - \mathbf{z}\|^2.$$

This mechanism directly produces an attention vector with many zeros and explicit limits per source token.

These constructions shift attention computation from indirectly shaping distributions with penalties or extra parameters to natively encoding desired properties into the transformation itself.

2. Constrained Sparsemax: Properties and Computation

Constrained sparsemax offers a differentiable solution, compatible with modern deep learning frameworks, with desirable characteristics for sequence transduction:

• Exact Sparsity and Coverage Bound: The output

$$\alpha_j^* = \max \left\{ 0, \min \{ u_j, z_j - \tau \} \right\}$$

(with normalization parameter $\tau$) is exactly zero for most $j$ and saturates at the per-token limit $u_j$ when needed.

• Differentiability: Full support for gradient-based optimization.
• Computational Efficiency: Forward step is $O(J)$, with backward propagation cost dependent only on the size of the sparse support.
• Interpretability: Attention aligns with linguistic intuition; attention maps become sharper and more interpretable, revealing explicit token alignments.

In comparison, softmax always assigns nonzero weight to every possible source word, while classic sparsemax may allow unlimited repeated usage, and constrained softmax does not induce within-step sparsity.

3. Empirical Results and Coverage Improvements

Native sparse attention mechanisms, especially constrained sparsemax, have been empirically evaluated on three language pairs:

• German–English (IWSLT14), Japanese–English (KFTT), and Romanian–English (WMT16).
• Constrained sparsemax with predicted fertility consistently achieves top performance in BLEU and METEOR, while minimizing REP-score (target repetition) and DROP-score (dropped source words).
• For instance, on De-En:
  • BLEU: 29.85
  • METEOR: 31.76
  • REP: 2.67
  • DROP: 5.23

Coverage issues—excess repetition and word omission—are substantially reduced compared to softmax and softmax with post-hoc coverage penalties. Sparse and constrained attention also yield more interpretable alignments.

4. Fertility Strategies and Implementation

A central feature of native sparse attention is the fertility parameter, which caps attention allocation per source word across the translation:

• Constant fertility: Assigns a fixed upper bound for all source words.
• Guided fertility: Sets upper bounds based on word alignment statistics from external aligners (e.g., fast_align).
• Predicted fertility: Uses a separately trained tagger to predict word-specific fertilities from alignment data.

At each decoding step, the maximum allowable attention mass for word $j$ is

$u_{t, j} = f_j - \beta_{j, t-1},$

where $f_j$ is the assigned fertility, and $\beta_{j, t-1}$ is the accumulated attention up to the previous step.

This explicit budget mechanism ensures neither over-attention nor neglect, integrating linguistic prior knowledge (via alignments) into attention computation.

5. Implications for Native Sparse Attention Paradigms

"Native sparse attention" describes mechanisms where sparsity and coverage are hard-coded into the attention map, not enforced indirectly:

• Direct sparsity: Zeroing of entries arises naturally; no need for post-processing, thresholding, or additional architectural components.
• Coverage control: Fertility bounds systematically prevent degeneration modes common in unconstrained attention.
• Efficiency: Tractable forward and backward passes; sparse support offers potential for hardware acceleration.
• Ease of adoption: The transformations can directly substitute for softmax in standard encoder-decoder models, requiring minimal change to model architecture.

These mechanisms can be generalized beyond NMT to any structured prediction, sequence-to-sequence, or alignment task requiring interpretable, resource-constrained allocation (e.g., summarization).

6. Challenges and Limitations

• Fertility estimation: While constant or alignment-guided fertilities are possible, accurate, data-driven or end-to-end fertility prediction methods may further improve adequacy.
• Scalability: The work demonstrates strong results in standard and low-resource settings; validation at the scale of very large data/models remains to be shown.
• Multi-head/multi-layer extension: Interactions between constrained sparse attention and transformer-based architectures with multiple attention heads and deep stacks invite further paper.

7. Broader Research Context and Future Directions

This approach grounds native sparse attention as a primitive in neural sequence modeling. Future research may focus on:

• Joint learning of fertility and translation: Integrating fertility prediction fully into the model's end-to-end training.
• Generalization to multi-head transformers: Exploring native sparse attention's effectiveness in modern large-scale architectures.
• Extensions to multilingual and multi-modal settings: Adapting the primitive to tasks beyond NMT, including speech, vision, or structured data analysis.

Native sparse attention as formalized via sparsemax, constrained softmax, and constrained sparsemax provides a theoretically motivated and practically validated foundation for interpretable, efficient, and adequately controlled attention computation, offering benefits in accuracy, coverage, and model transparency for sequence-to-sequence tasks and beyond.