---
title: Rectified Sparse Attention (ReSA)
url: https://www.emergentmind.com/topics/rectified-sparse-attention-resa
type: topic
---

# Rectified Sparse Attention (ReSA)

Rectified Sparse Attention (ReSA) encompasses a class of attention mechanisms in neural architectures where explicit rectification operations are applied to enforce sparsity, suppress pathological behaviors of classic softmax normalization, and address approximation biases arising in block-sparse regimes. Recent literature distinguishes ReSA in two major contexts: (1) *elementwise rectification* applied to attention logits to produce sparse, non-sum-to-one distributions (notably Softpick and related functions), and (2) *rectification of block-sparse attention outputs* via careful reallocation and gain-aware compensation to recover fidelity lost during sparse approximation. This article covers both paradigms, their mathematical formulations, algorithmic details, empirical results, and their broader implications for scalable sequence modeling, quantization, and efficient high-dimensional generative models.

## 1. Mathematical Formulations and Core Variants

### 1.1 Softpick: Rectified Non–Sum-to-One Attention

Given logits $\mathbf{x} \in \mathbb{R}^N$, Softpick replaces softmax normalization with a rectified, ReLU-shifted, and non-sum-to-one mapping:
\[
\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}
\]
For numerical stability, logits are shifted by $m = \max_k x_k$, yielding
\[
\text{Softpick}(\mathbf{x})_i = \frac{\max(e^{x_i-m} - e^{-m}, 0)}{\sum_{j=1}^N |e^{x_j-m} - e^{-m}| + \epsilon}
\]
In self-attention, this yields:
\[
\mathrm{Attention}(Q, K, V) = \text{Softpick}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V
\]
Unlike softmax, Softpick does **not** guarantee that $\sum_i \alpha_i = 1$.

### 1.2 Rectification in Block-Sparse Attention

Block-sparse attention computes attention over a subset of blocks, yielding a reweighted softmax that distorts the original distribution. If $A_{n,m}$ is full softmax over all keys $m$ and $A^{spa}_{n,m}$ uses a sparse mask $\widehat{M}_{n,m}$:
\[
A^{spa}_{n,m} = 
\frac{e^{S_{n,m}} \cdot \widehat{M}_{n,m}}{\sum_{m': \widehat{M}_{n,m'}=1} e^{S_{n,m'}}}
\]
This systematically amplifies the kept (“critical”) tokens and neglects “non-critical” ones, introducing bias. Rectification seeks to restore fidelity by re-scaling $A^{spa}$ and optionally compensating for dropped tokens, often using a pooled proxy of the full attention for efficiency [2511.19835].

## 2. Algorithmic Implementations and Workflows

### 2.1 Elementwise Rectified Attention

The Softpick workflow in transformer heads [2504.20966]:

```python
# Inputs: Q, K, V ∈ ℝ^{L×d}, ε>0
S = Q @ K.T / sqrt(d)        # Raw attention scores
m = S.max(dim=-1, keepdim=True)
P = exp(S - m) - exp(-m)
R = torch.relu(P)
D = R.abs().sum(dim=-1, keepdim=True) + ε
A = R / D                    # Sparse, rectified attention weights
O = A @ V
```

Memory and compute complexity remain strictly $O(L^2 d)$. When implemented via streaming blocks (as in FlashAttention), peak memory is $O(L d)$.

### 2.2 Block-Sparse ReSA with Periodic Rectification

In sequence models handling very long contexts, ReSA alternates fast sparse decoding with periodic dense rectification:
```python
# Algorithm: Rectified Sparse Attention Inference
1. Encode prompt densely → KV cache
2. For each generation step:
    a. Apply sparse block-wise attention (GBSA) to generate next token
    b. Update cache with new KV
    c. Every f steps: run dense forward pass on last f tokens to refresh cache (rectification)
```
For block size $b$, sparsity $p$, rectification frequency $f$, the practical memory load per token is $n (\frac{1}{b} + p + \frac{1}{f})$, with speedup factor $S = 1/(\frac{1}{b} + p + \frac{1}{f})$ [2506.04108].

### 2.3 Block-wise Rectification (IPAR, GAPR)

In block-sparse video-scale diffusion models, ReSA requires two explicit rectification steps [2511.19835]:

- **Isolated-Pooling Attention Reallocation (IPAR):** Pools queries/keys per block and computes a reference softmax. This approximates the true contribution (“renormalization constant”) of each kept block and rescales sparse attention weights.
- **Gain-Aware Pooling Rectification (GAPR):** For dropped (“non-critical”) blocks, computes the estimated gain from pooling and compares it to pooling error. Only blocks where gain exceeds error are compensated in the output.

## 3. Empirical Performance and Comparative Metrics

| Model/Task       | Sparsity (%) | Speedup | Fidelity Loss | Notable Metrics                               |
|------------------|-------------|---------|--------------|-----------------------------------------------|
| Softpick–340M LM | 99.34       | n/a     | 0% sink rate | Kurtosis drop 33510.8→340.96, 2–30 pt LM acc |
| ReSA–LLM (1.5B)  | $\sim$90    | 2.42×   | <1% acc loss | Top-3 next-token acc $\uparrow$  vs. sparse   |
| ReSA–Video T2V   | 88.95       | 3.33×   | $\Delta$VR <.02 | VBench gap 0.6 pt vs. dense                 |

- **Softpick** eliminates attention sinks entirely and achieves $\sim$99% true sparsity in attention maps [2504.20966], with hidden-state kurtosis reduced by two orders of magnitude and substantial improvements in quantized (low-precision) model benchmarks.
- **Block-wise ReSA** in LLMs yields near-lossless fidelity (≤1% acc drop at 256K context) and up to $2.42\times$ speedup without retraining [2506.04108].
- **Video ReSA (SpaAttn)** achieves speedups of 2.08–3.33× while maintaining high sample quality at $\sim$90% sparsity, with ablation showing that both IPAR and GAPR are essential for restoring attention output fidelity [2511.19835].

## 4. Theoretical Analyses of Sparsity, Bias, and Error Accumulation

- **Softpick** assigns *exact* zero weight to negative logits, yielding precise sparsity. Because the denominator is not a partition function, normalization is unconstrained, eliminating the “sink” behavior of softmax. The absence of mass reallocation prevents any token from concentrating excessive weight artificially [2504.20966].
- **Block ReSA–LLMs**: Without periodic dense rectification, blockwise sparse approximation errors accumulate unboundedly in the KV cache; every f-token dense refresh bounds total error by $O(f)$, independent of sequence length [2506.04108].
- **Block ReSA–Video**: Renormalization bias causes over-amplification on critical tokens and complete erasure on dropped tokens. Optimal rectification restores the distribution using an implicit pooled full attention, maintaining high alignment to the original dense distribution [2511.19835].

## 5. Comparative Analysis with Related Sparse Attention Approaches

- **Rectified Linear Attention (ReLA, [2104.07012]):** Replaces softmax with ReLU, enforcing sparsity but requiring layer-norm or gating to prevent divergence. Achieves head diversity and superior alignment error rates in machine translation, but lacks Softpick’s explicit normalization control or block-sparse correction mechanisms.
- **Entropy-regularized variants (sparsemax, entmax):** Impose sparsity via altered normalization but often at significantly lower speed and require iterative root-finding; ReSA/Softpick preserves $O(L^2d)$ throughput and operates as a direct, monotonic mapping.
- **Classic block-sparse methods:** (e.g., Quest, ClusterKV, MagicPig) focus on reducing computational load but do not address systematic bias or error accumulation in the attention map, leading to long-sequence degradation [2506.04108].

## 6. Practical Implications and Extensions

- **Quantization and Low-Precision Regimes:** Suppressing extreme activations via Softpick enables stable 2–3 bit quantization and reduces reliance on elaborate outlier-handling during quantized inference [2504.20966].
- **Interpretability:** Rectified attention, by producing highly sparse and sharp maps, improves the legibility of token flows and heatmaps, facilitating causal and structural analysis of transformer decisions.
- **Structured Pruning and Acceleration:** True zero attention weights (up to $\sim$99%) enable high-throughput sparse kernels and token/head pruning. In blockwise ReSA, compensation mechanisms identify when to restore approximate contributions from pruned paths.
- **General Applicability:** The rectified, non-sum-to-one principle applies across domains—language models, diffusion transformers, and vision/multimodal architectures—where attention sinks and outlier activations pose obstacles to efficiency and scale [2504.20966][2511.19835].

## 7. Limitations and Open Directions

- **Fixed vs. Adaptive Rectification:** Periodic dense rectification ($f$ tokens) is robust in practice; however, adaptively triggering based on an error estimate may further balance speed and fidelity [2506.04108]. This remains an open question for sequence models with widely varying context lengths.
- **Block Representation Quality:** Pooling strategies used for rectification rely on the accuracy of pooled query/key statistics. Improved low-rank or learned block summaries may extend ReSA to even higher sparsity without quality loss [2511.19835].
- **Extensibility to Joint Training:** While all ReSA approaches described are *drop-in* at inference, their integration into end-to-end training (e.g., learning sparse masks, block assignments, or rectification schedules jointly) is underexplored.

Rectified Sparse Attention thus synthesizes a spectrum of strategies for imposing true sparsity, suppressing degenerate behaviors in attention normalization, and enabling fast, interpretable, high-fidelity attention in both autoregressive sequence models and large-scale generative frameworks. The dual focus on explicit elementwise rectification and block-level fidelity restoration distinguishes modern ReSA approaches from both legacy softmax and alternative sparse normalization schemes. 

**Key references:** Softpick [2504.20966], ReSA-LLM [2506.04108], ReSA-Video (SpaAttn) [2511.19835], ReLA [2104.07012].

Source: https://www.emergentmind.com/topics/rectified-sparse-attention-resa