Rectified Sparse Attention (ReSA)

• Rectified Sparse Attention (ReSA) is a mechanism that applies explicit rectification to attention scores, enforcing sparsity and mitigating softmax biases.
• It employs both elementwise methods like Softpick and block-sparse rectification to improve model efficiency, quantization, and interpretability.
• Empirical results demonstrate near-lossless fidelity with significant speedups in large-scale transformers, making ReSA vital for scalable sequence modeling.

Rectified Sparse Attention (ReSA) encompasses a class of attention mechanisms in neural architectures where explicit rectification operations are applied to enforce sparsity, suppress pathological behaviors of classic softmax normalization, and address approximation biases arising in block-sparse regimes. Recent literature distinguishes ReSA in two major contexts: (1) elementwise rectification applied to attention logits to produce sparse, non-sum-to-one distributions (notably Softpick and related functions), and (2) rectification of block-sparse attention outputs via careful reallocation and gain-aware compensation to recover fidelity lost during sparse approximation. This article covers both paradigms, their mathematical formulations, algorithmic details, empirical results, and their broader implications for scalable sequence modeling, quantization, and efficient high-dimensional generative models.

1. Mathematical Formulations and Core Variants

1.1 Softpick: Rectified Non–Sum-to-One Attention

Given logits $\mathbf{x} \in \mathbb{R}^N$, Softpick replaces softmax normalization with a rectified, ReLU-shifted, and non-sum-to-one mapping: $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$ For numerical stability, logits are shifted by $m = \max_k x_k$, yielding

$$\text{Softpick}(\mathbf{x})_i = \frac{\max(e^{x_i-m} - e^{-m}, 0)}{\sum_{j=1}^N |e^{x_j-m} - e^{-m}| + \epsilon}$$

In self-attention, this yields: $$\mathrm{Attention}(Q, K, V) = \text{Softpick}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$ Unlike softmax, Softpick does not guarantee that $\sum_i \alpha_i = 1$.

1.2 Rectification in Block-Sparse Attention

Block-sparse attention computes attention over a subset of blocks, yielding a reweighted softmax that distorts the original distribution. If $A_{n,m}$ is full softmax over all keys $m$ and $A^{spa}_{n,m}$ uses a sparse mask $\widehat{M}_{n,m}$: $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$0 This systematically amplifies the kept (“critical”) tokens and neglects “non-critical” ones, introducing bias. Rectification seeks to restore fidelity by re-scaling $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$1 and optionally compensating for dropped tokens, often using a pooled proxy of the full attention for efficiency (Liu et al., 25 Nov 2025).

2. Algorithmic Implementations and Workflows

2.1 Elementwise Rectified Attention

The Softpick workflow in transformer heads (Zuhri et al., 29 Apr 2025):

$m = \max_k x_k$9

Memory and compute complexity remain strictly $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$2. When implemented via streaming blocks (as in FlashAttention), peak memory is $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$3.

2.2 Block-Sparse ReSA with Periodic Rectification

In sequence models handling very long contexts, ReSA alternates fast sparse decoding with periodic dense rectification: $$\text{Softpick}(\mathbf{x})_i = \frac{\max(e^{x_i-m} - e^{-m}, 0)}{\sum_{j=1}^N |e^{x_j-m} - e^{-m}| + \epsilon}$$0 For block size $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$4, sparsity $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$5, rectification frequency $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$6, the practical memory load per token is $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$7, with speedup factor $$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$8 (Sun et al., 4 Jun 2025).

2.3 Block-wise Rectification (IPAR, GAPR)

In block-sparse video-scale diffusion models, ReSA requires two explicit rectification steps (Liu et al., 25 Nov 2025):

• Isolated-Pooling Attention Reallocation (IPAR): Pools queries/keys per block and computes a reference softmax. This approximates the true contribution (“renormalization constant”) of each kept block and rescales sparse attention weights.
• Gain-Aware Pooling Rectification (GAPR): For dropped (“non-critical”) blocks, computes the estimated gain from pooling and compares it to pooling error. Only blocks where gain exceeds error are compensated in the output.

3. Empirical Performance and Comparative Metrics

Model/Task	Sparsity (%)	Speedup	Fidelity Loss	Notable Metrics
Softpick–340M LM	99.34	n/a	0% sink rate	Kurtosis drop 33510.8→340.96, 2–30 pt LM acc
ReSA–LLM (1.5B)	$$\text{Softpick}(\mathbf{x})_i = \frac{\max\!\left(e^{x_i} - 1,\,0\right)}{\sum_{j=1}^N \left|\,e^{x_j} - 1 \right| + \epsilon}$$990	2.42×	<1% acc loss	Top-3 next-token acc $m = \max_k x_k$0 vs. sparse
ReSA–Video T2V	88.95	3.33×	$m = \max_k x_k$1VR <.02	VBench gap 0.6 pt vs. dense

• Softpick eliminates attention sinks entirely and achieves $m = \max_k x_k$299% true sparsity in attention maps (Zuhri et al., 29 Apr 2025), with hidden-state kurtosis reduced by two orders of magnitude and substantial improvements in quantized (low-precision) model benchmarks.
• Block-wise ReSA in LLMs yields near-lossless fidelity (≤1% acc drop at 256K context) and up to $m = \max_k x_k$3 speedup without retraining (Sun et al., 4 Jun 2025).
• Video ReSA (SpaAttn) achieves speedups of 2.08–3.33× while maintaining high sample quality at $m = \max_k x_k$490% sparsity, with ablation showing that both IPAR and GAPR are essential for restoring attention output fidelity (Liu et al., 25 Nov 2025).

4. Theoretical Analyses of Sparsity, Bias, and Error Accumulation

• Softpick assigns exact zero weight to negative logits, yielding precise sparsity. Because the denominator is not a partition function, normalization is unconstrained, eliminating the “sink” behavior of softmax. The absence of mass reallocation prevents any token from concentrating excessive weight artificially (Zuhri et al., 29 Apr 2025).
• Block ReSA–LLMs: Without periodic dense rectification, blockwise sparse approximation errors accumulate unboundedly in the KV cache; every f-token dense refresh bounds total error by $m = \max_k x_k$5, independent of sequence length (Sun et al., 4 Jun 2025).
• Block ReSA–Video: Renormalization bias causes over-amplification on critical tokens and complete erasure on dropped tokens. Optimal rectification restores the distribution using an implicit pooled full attention, maintaining high alignment to the original dense distribution (Liu et al., 25 Nov 2025).

5. Comparative Analysis with Related Sparse Attention Approaches

• Rectified Linear Attention (ReLA, (Zhang et al., 2021)): Replaces softmax with ReLU, enforcing sparsity but requiring layer-norm or gating to prevent divergence. Achieves head diversity and superior alignment error rates in machine translation, but lacks Softpick’s explicit normalization control or block-sparse correction mechanisms.
• Entropy-regularized variants (sparsemax, entmax): Impose sparsity via altered normalization but often at significantly lower speed and require iterative root-finding; ReSA/Softpick preserves $m = \max_k x_k$6 throughput and operates as a direct, monotonic mapping.
• Classic block-sparse methods: (e.g., Quest, ClusterKV, MagicPig) focus on reducing computational load but do not address systematic bias or error accumulation in the attention map, leading to long-sequence degradation (Sun et al., 4 Jun 2025).

6. Practical Implications and Extensions

• Quantization and Low-Precision Regimes: Suppressing extreme activations via Softpick enables stable 2–3 bit quantization and reduces reliance on elaborate outlier-handling during quantized inference (Zuhri et al., 29 Apr 2025).
• Interpretability: Rectified attention, by producing highly sparse and sharp maps, improves the legibility of token flows and heatmaps, facilitating causal and structural analysis of transformer decisions.
• Structured Pruning and Acceleration: True zero attention weights (up to $m = \max_k x_k$799%) enable high-throughput sparse kernels and token/head pruning. In blockwise ReSA, compensation mechanisms identify when to restore approximate contributions from pruned paths.
• General Applicability: The rectified, non-sum-to-one principle applies across domains—LLMs, diffusion transformers, and vision/multimodal architectures—where attention sinks and outlier activations pose obstacles to efficiency and scale (Zuhri et al., 29 Apr 2025, Liu et al., 25 Nov 2025).

7. Limitations and Open Directions

• Fixed vs. Adaptive Rectification: Periodic dense rectification ($m = \max_k x_k$8 tokens) is robust in practice; however, adaptively triggering based on an error estimate may further balance speed and fidelity (Sun et al., 4 Jun 2025). This remains an open question for sequence models with widely varying context lengths.
• Block Representation Quality: Pooling strategies used for rectification rely on the accuracy of pooled query/key statistics. Improved low-rank or learned block summaries may extend ReSA to even higher sparsity without quality loss (Liu et al., 25 Nov 2025).
• Extensibility to Joint Training: While all ReSA approaches described are drop-in at inference, their integration into end-to-end training (e.g., learning sparse masks, block assignments, or rectification schedules jointly) is underexplored.

Rectified Sparse Attention thus synthesizes a spectrum of strategies for imposing true sparsity, suppressing degenerate behaviors in attention normalization, and enabling fast, interpretable, high-fidelity attention in both autoregressive sequence models and large-scale generative frameworks. The dual focus on explicit elementwise rectification and block-level fidelity restoration distinguishes modern ReSA approaches from both legacy softmax and alternative sparse normalization schemes.

Key references: Softpick (Zuhri et al., 29 Apr 2025), ReSA-LLM (Sun et al., 4 Jun 2025), ReSA-Video (SpaAttn) (Liu et al., 25 Nov 2025), ReLA (Zhang et al., 2021).