Sparse Transformer Algorithms (FlashAttention)

• Sparse transformer algorithms like FlashAttention are methods that reduce the quadratic complexity of attention by limiting token pair computations.
• They leverage hardware-aware scheduling, tiling, and block-based strategies to achieve significant speedups and memory efficiency.
• Adaptive and data-driven sparsity techniques dynamically optimize mask representations, enhancing model scalability and expressiveness.

Sparse transformer algorithms address the formidable computational and memory cost of the attention mechanism by exploiting algorithmic sparsity, memory hierarchy, data-dependent masking, and hardware-aware scheduling. FlashAttention and its extensions represent a major line of development in making attention practical at scale, alongside structured sparsification, adaptive and learned mask methods, and graph-centric formulations. This article surveys the principles and methods underpinning state-of-the-art sparse transformer algorithms as exemplified by FlashAttention and its descendants.

1. Computational Motivation and Foundations

The attention mechanism in transformers is dominated by the computation of the scaled dot-product attention: $$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right) V,$$ where $Q, K, V \in \mathbb{R}^{N \times d_k}$ for a sequence of length $N$. The matrix multiplication and subsequent softmax normalization yield $\mathcal{O}(N^2 d_k)$ complexity for both computational cost and memory footprint—placing severe limits on achievable context length.

Sparse attention algorithms reduce this cost through one or more of the following strategies:

• Limiting the set of tokens that each query attends to via a fixed or learned mask ("structured sparsity").
• Exploiting hardware memory hierarchy through tiling and blocking to reduce memory reads/writes ("IO awareness").
• Replacing dense softmax with sparse variants (e.g., $\alpha$-entmax, adaptive masking).
• Reformulating attention as a graph computation or conditional expectation, allowing for strict computational work minimization.

The goal is to reduce effective complexity, scale context length, and retain model expressiveness.

2. FlashAttention: IO-Aware Exact and Sparse Attention

FlashAttention (Dao et al., 2022) introduced an IO-aware, blockwise approach to exact (dense) softmax attention on GPUs. By partitioning $Q$, $K$, and $V$ into tiles that fit in on-chip SRAM, FlashAttention avoids materializing the $N \times N$ attention matrix, reducing global memory (HBM) accesses from $\Theta(N^2)$ to $\Theta(N^2 d^2 / M)$, with $M$ the size of fast memory. Incremental numerically stable softmax computation across blocks maintains exactness, achieving $2-4\times$ speedup and enabling context of $16$–$64$K tokens with linear memory scaling.

Block-sparse FlashAttention further skips computation on masked-out blocks, yielding IO complexity of

$\Theta\left(Nd + \frac{N^2 d^2}{M}s\right)$

where $s$ is block density. This approach demonstrated up to $3\times$ training speedup for GPT-2 and $15\%$ end-to-end improvement for BERT-large, as well as higher-quality long-sequence models (Dao et al., 2022, Dao, 2023).

FlashAttention-2 (Dao, 2023) advances by optimizing non-matmul operations, parallelizing attention over the sequence length, and reducing shared memory use in CUDA blocks, reaching $50$–$73\%$ of theoretical peak FLOPs/s and enabling up to $225$ TFLOPs/s per A100 GPU in end-to-end training.

3. Algorithmic Extensions: Mask-Awareness, Graph Processing, and Adaptive Sparsity

Mask-Aware and Block-Based Methods

Recent work has focused on making FlashAttention and similar algorithms efficiently process arbitrary or structured sparse masks:

• Binary Block Masking (Sharma et al., 23 Sep 2024): Preprocesses sparse binary masks into blockwise binary indicators; only nonzero blocks are processed, skipping fully zeroed regions, and achieves up to $9\times$ runtime improvement for complex masks in sequence packing and tree masking.
• FlashMask (Wang et al., 2 Oct 2024): Encodes masks as sparse column-wise intervals (e.g., lower/upper triangular ranges), reducing mask memory from $O(N^2)$ to $O(N)$. This approach allows linear memory and kernel time scaling, efficient block skipping, and achieves $1.65$–$3.22\times$ end-to-end training speedup.
• Flash Sparse Attention (Yan et al., 25 Aug 2025): Reorders kernel loops over query and key-value blocks to enable efficient execution of native, trainable sparse attention (NSA) for small GQA group sizes, yielding up to $3.5\times$ kernel speedup and $1.25\times$ faster end-to-end training compared to baseline NSA.

Graph-Centric and Work-Optimal Attention

Longer Attention Span (Tomczak et al., 31 Jan 2025) proposes representing attention as a sparse graph, where tokens are nodes and attention masks define edges. By computing only over explicit edges using COO or CSR storage or via pattern parametrization, and by performing online softmax statistics, the algorithm achieves theoretical and practical work optimality—computing precisely $O(S_f L^2 d)$ work, where $S_f$ is mask sparsity. On extremely long sequences (up to $160$ million tokens), this yields up to $51\times$ runtime speedup compared to FlashAttention.

Adaptive and Data-Driven Sparsity

SBM-Transformer (Cho et al., 2022) adopts a stochastic block model per attention head, sampling an expected $O(n)$ set of edges (token pairs) in a data-adaptive fashion, with gradients propagated through the discrete mask using a straight-through estimator. The method achieves state-of-the-art performance for both sequence tasks and natural language understanding benchmarks with significant computational savings.

AdaSplash (Gonçalves et al., 17 Feb 2025) realizes efficient, adaptive sparse attention by combining the expressive $\alpha$-entmax family (which generalizes softmax/entmax and automatically induces sparsity per input) with a hardware-efficient Halley–bisection solver and Triton kernels. Dynamic skipping at the block level attains speed and memory comparable to FlashAttention-2 for sequence lengths up to $8$K.

4. Hardware and Kernel Innovations

FlashAttention-inspired algorithms have driven extensive innovation at the kernel and hardware level:

• Efficient tiling, tiling-based recomputation, and block masking eliminate memory constraints as sequence length grows (Dao et al., 2022, Wang et al., 2 Oct 2024).
• FLASH-D (Alexandridis et al., 20 May 2025) replaces explicit softmax division and max subtraction in the recursion with a sigmoid-based update, reducing hardware area and power by over $20\%$ compared to state-of-the-art parallel accelerators, without compromising on kernel properties or accuracy.
• SystolicAttention (Lin et al., 15 Jul 2025) fuses all FlashAttention operations—including softmax and non-matrix transformations—directly within a single systolic array by augmenting each PE with comparators and Split units for piecewise exponential approximation. This eliminates data transfer to vector cores and achieves $1.77\times$-$4.83\times$ higher utilization versus AWS NeuronCore-v2 and TPUv5e.
• FPGA/ASIC-friendly sparse attention operators leveraging low-precision quantized Q/K vectors and Top-k preselection (with exact full-precision computation of only nontrivial entries) allow kernel and energy efficiency of $80\times$ over CPU and $2.6\times$ over GPU (Peng et al., 2022).

5. Structured and Adaptive Sparsification: Methods and Implications

Structured sparsification strategies seek to balance expressiveness, sample efficiency, and computational scaling:

• Combiner (Ren et al., 2021) achieves full attention with sub-quadratic cost by factorizing $p(j|i)$ in attention into structured partitions. Direct and pooled abstraction terms allow each token to reach all others, while fixed/log/axial partition patterns yield costs of $O(L \log L)$ or $O(L \sqrt{L})$.
• SPION (Yoon et al., 2023) uses convolutional and flood-filling patterns to determine block-sparse layouts layer-wise, dynamically adapting the sparsity pattern in each transformer layer. This adaption leads to up to $3.08\times$ speedup over previous sparse transformers and strong benchmark performance.
• SBM-Transformer (Cho et al., 2022) and VSA (Zhang et al., 19 May 2025) dynamically learn or predict sparsity patterns that adapt to each sequence. VSA, in particular, deploys a coarse-to-fine selection (mean-pool cubes $\rightarrow$ top-k critical cubes $\rightarrow$ block-sparse fine attention) in a design maintaining $85\%$ FlashAttention MFU with $8\times$ reduction in attention FLOPs, achieving up to $6\times$ attention and $1.7\times$ end-to-end speedup in large video DiTs.

The Spark Transformer (You et al., 7 Jun 2025) enforces top-$k$ activation sparsity in both the FFN and attention blocks, using a statistical thresholding mechanism for hardware-friendly, predictable sparsity. Parameter reallocation creates an integrated, low-cost predictor to select which neurons and tokens should be active, yielding up to $2.5\times$ FLOP reduction and substantial wall-time benefits without quality loss.

6. Stability, Expressivity, and Learning-Theoretic Considerations

Stability and expressivity remain central to sparsification strategies:

• FlashAttention incurs an order of magnitude higher numeric deviation in BF16 vs. baseline dense attention during the forward pass due to tile-rescaling (Golden et al., 5 May 2024). However, this deviation induces at most $2$–$5\times$ smaller weight divergence during training than switching between FP32 and FP16, demonstrating that system-level optimizations remain stable under practical training regimes.
• Universal approximation properties have been theoretically validated for both structured sparsifiers (e.g., BigBird, Combiner) and data-adaptive alternatives such as SBM-Transformer (Cho et al., 2022), provided certain connectivity and self-loop conditions are met.
• Chain-of-thought (CoT)–induced sparsity leads to interpretable, nearly one-hot attention patterns (Wen et al., 7 Oct 2024). CoT decomposes computation into steps with minimal dependency—each step attends to only a small, specific subset of prior tokens—leading to polynomial sample complexity in settings where dense attention suffers from exponential inefficiency.

7. Generalization, Domains, and Future Directions

Sparse transformer algorithms increasingly support multi-modal and multi-granularity sparsity patterns:

• FlashOmni (Qiao et al., 29 Sep 2025) introduces sparse "symbols" for both feature caching and block-sparse skipping, enabling a unified, dynamically decoded sparse attention kernel that can execute diverse strategies on a single efficient engine. Near-linear speedup (matching sparsity ratio) and up to $3.8\times$ acceleration are demonstrated.
• SFi-Former (Li et al., 29 Apr 2025) employs $\ell_1$-regularized network flow energy minimization to induce sparsity in graph transformer attention, outperforming dense and alternative sparse graph transformers on long-range benchmarks with reduced overfitting.
• Continued research targets adaptive mask representations, further reductions in scattering non-contiguous memory access overhead, generic hardware-agnostic kernels, and combinations of sparsity with other scaling strategies (e.g., MoE, low-rank, quantized models).

The present landscape of algorithms—from FlashAttention and graph-based implementations to trainable, block-, and adaptive sparse schemes—demonstrates the ongoing, multi-faceted effort to scale transformer models to unprecedented context lengths and domains while preserving accuracy and theoretical capacity. The convergence of algorithmic, hardware, and data-driven advances underpins the practical future of sparse transformers.