---
title: Chunked and Blockwise Attention
url: https://www.emergentmind.com/topics/chunked-and-blockwise-attention
type: topic
---

# Chunked and Blockwise Attention

Chunked and Blockwise Attention refers to a family of attention mechanisms in deep learning models—typically Transformer architectures—where the input sequence is partitioned into contiguous blocks or chunks, and attention is performed either locally within each chunk or hierarchically/compositionally across such blocks. This strategy is motivated by the need to scale attention and memory to very long sequences, improve efficiency, and facilitate streaming or online processing while still capturing essential local and global dependencies.

## 1. Formal Definitions and Variants

Chunked (or blockwise) attention partitions a sequence $X=[x_1, x_2, ..., x_L] \in \mathbb{R}^{L \times d}$ into $M = \lceil L / C \rceil$ non-overlapping blocks of size $C$, where $C$ is the chunk size. For each chunk $X^{(m)}$ (indices $[(m-1)C+1, mC]$), queries, keys, and values are computed and attention is restricted within the same chunk:
\[
A^{(m)} = \mathrm{softmax} \left( Q^{(m)} [K^{(m)}]^T / \sqrt{d_k} \right), \qquad Y^{(m)} = A^{(m)} V^{(m)}
\]
Global output is constructed via concatenation:
\[
Y = \mathrm{concat}(Y^{(1)}, ..., Y^{(M)}), \qquad Y \in \mathbb{R}^{L \times d}
\]
Variants augment local attention with additional mechanisms: overlapping blocks with strided or sliding windows [2107.09428], cross-chunk attention by shifting key/value blocks [2312.07305], and hierarchical, multi-stage chunking for progressive context expansion [2112.15087].

## 2. Mathematical Formulations and Computational Complexity

- **Memory and Computation:** For standard self-attention over sequence length $L$, complexity is $\mathcal{O}(L^2 d)$ in both compute and memory due to the dense $L\times L$ attention matrix. Chunked/blockwise attention reduces this to $\mathcal{O}(L C d)$ (with $C \ll L$), as only $C\times C$ attention is performed per chunk [2312.05605][1911.02972].
- **Hierarchical/Multistage:** Multistage chunking (as in ChunkFormer) performs multiple passes, each with growing chunk size $k_1 < k_2 < \cdots < k_N$, increasing the effective receptive field in a staged fashion:
  \[
  \text{Total cost} \approx \mathcal{O}(L k_N d N)
  \]
  where $N$ is the number of stages [2112.15087].
- **Block-Sparse and Hybrid:** Advanced blockwise schemes combine dense intra-block attention with block-sparse cross-block connections for global context at reduced cost, sometimes guided by principled selection criteria [2601.22379].

## 3. Applications and Model Designs

Chunked/blockwise attention appears in diverse domains:
- **Long-Document Language Modeling:** BlockBERT uses block-sparse patterns, sometimes with head-dependent permutations to mix local and global contexts [1911.02972]. ChunkLLM adopts chunk boundary detection and compression for efficient inference over 120k-token inputs [2510.02361].
- **Streaming and Online ASR:** Blockwise encoders and decoders enable low-latency, streamable speech recognition via strict chunked attention windows [2309.08436][2107.09428][2005.00205]. MoChA and its multi-head variant dynamically determine chunk boundaries for monotonic alignment [2005.00205].
- **Hierarchical Time Series or Sequential Data:** ChunkFormer demonstrates accelerated and stabilized training on long time series, learning local and global seasonality/hierarchies [2112.15087]. TCNCA integrates chunked attention with dilated convolutions for linear-in-length sequence processing [2312.05605].
- **Efficient Large-Context Transformers:** Blockwise Parallel Transformers and Ring Attention push context lengths into millions of tokens by distributing blockwise attention across devices for near-linear scaling [2305.19370][2310.01889].
- **Memory-Augmented Architectures:** Models such as the memory-augmented chunked Transformer use gated FIFO recurrent memory to bridge chunk boundaries [2507.00453]. SPLA combines block-sparse selection with residual linear attention to compress and retain “long tail” context [2601.22379].
- **Generative Diffusion and Video:** Blockwise (chunked) attention underlies efficient attention caching and reuse in diffusion models, where block-internal and block-external attention are fused for throughput gains [2602.05305].
- **Multimodal and Social Signal Modeling:** Blockwise masking can be tailored for causal, multimodal, and agent-wise chunking, as in cross-modal social signal prediction [2501.13416].

## 4. Receptive Field and Information Flow

Chunked attention is inherently local, but expressivity is extended by various strategies:
- **Cross-Chunk Communication:** Multi-stage chunking, memory augmentation, or cross-block sparse connections allow global information aggregation [2112.15087][2507.00453][1911.02972][2507.04416].
- **Shifting and Dilated Patterns:** Shifted Cross Chunk Attention (SCCA) and Shifted Dilated Attention use systematic key/value shifts or dilations across heads or layers. This dramatically accelerates receptive-field growth, approximating global attention at linear cost [2312.07305].
- **Hybrid Approaches:** Models such as RAT interleave recurrent intra-chunk processing with attention across chunk summaries, combining efficiency with recoverability of distant dependencies [2507.04416].

## 5. Empirical Results and Trade-offs

Empirical studies demonstrate:
- **Memory and Throughput:** Memory usage drops from $\mathcal{O}(L^2)$ to $\mathcal{O}(L C)$ [2312.05605][2305.19370]. In practical systems, per-layer activation memory is reduced by $4\times$ or more; context lengths can extend to millions of tokens [2305.19370][2310.01889].
- **Modeling Efficacy:** Chunked/blockwise attention can match or exceed the accuracy of full attention on long-sequence tasks, provided global communication is not overly restricted. For example, on language modeling and QA tasks, Blockwise-2x86 or multi-stage chunking recovers or exceeds RoBERTa/vanilla Transformer accuracy while saving 20–36% of memory [1911.02972][2112.15087].
- **Efficiency vs. Contextual Resolution:** Chunk size $C$ is central: smaller $C$ enhances locality and speed but truncates dependencies. Large $C$ recovers more global context, at the cost of increased compute per chunk, and may bring back quadratic cost if not carefully chosen [2312.05605][2112.15087].
- **Streaming and Latency:** In online ASR, blockwise attention achieves latencies as low as $\sim$140 ms at minor WER penalty compared to full-sequence models. Overlapping blocks and dynamic mapping mitigate chunk boundary artifacts [2107.09428][2309.08436].
- **Accuracy vs. Latency Trade-offs:** For chunk-based transducers, increasing chunk size improves BLEU and WER but adds latency and inference cost [2602.24245].

| Method/Model          | Task/Domain             | Empirical Highlights                                  |
|---------------------- |------------------------|------------------------------------------------------|
| ChunkFormer           | Long time series        | Macro F1 +1–3%; memory $\mathcal{O}(L k_N)$          |
| BlockBERT             | Question answering      | $18.7$–$36.1$% less memory, $12$–$25.1$% faster      |
| Ring Attention        | LLM, RL                | 4M-token context, $\sim$1/P scaling                  |
| ChunkLLM              | LLM inference          | 4.48× speedup, 98.6% accuracy retained               |
| CHAT                  | Streaming ASR/ST       | Mem −46.2%, train 1.36× faster, +6.3% WER rel. gain  |
| TCNCA                 | Language modeling      | BPC 1.01 (lower), 1.37× speedup, up to 7× faster TCN |
| RAT                   | Long-context LM        | 7–9× speedup, same perplexity as standard attention  |
| FlashBlock            | Diffusion/video/text   | 1.44× throughput, 1.6× attention time reduction      |

## 6. Limitations and Extensions

- **Locality Bottleneck:** Pure chunked attention, without cross-chunk links, cannot propagate information globally within a single layer and must rely on multi-layer or additional recurrent/global paths [2312.07305][2507.00453].
- **Block Boundary Effects:** Fixed block boundaries may disrupt dependencies at segment edges unless overlapping, shifting, or dynamic mapping strategies are used [2107.09428][2312.07305].
- **Parameterization Choices:** Permutation patterns, hierarchical chunking schedules, and chunk/block size are highly scenario- and resource-dependent.
- **Plug-and-Play Integration:** Some schemes (e.g., SCCA, ChunkLLM) are designed to integrate with efficient kernel implementations like FlashAttention and parameter-efficient fine-tuning frameworks (LoRA) [2510.02361][2312.07305].
- **Extensions:** Blockwise attention ideas are extended to multimodal, temporal, and agent-wise scenarios by introducing sub-blocks for modality, participant, or segment [2501.13416].

## 7. Context and Significance

The development of chunked and blockwise attention mechanisms is central to unlocking efficient, scalable, and streaming machine learning systems for long sequences—a prevailing challenge in language modeling, speech recognition, video, and beyond. By trading dense global attention for well-structured local and compositional context, these approaches offer practical and theoretically grounded speedups, massive memory savings, and substantial flexibility. They are now fundamental primitives in the growing repertoire of efficient large-context neural architectures [2112.15087][2305.19370][2310.01889][2312.07305][2602.24245][2507.04416].

Source: https://www.emergentmind.com/topics/chunked-and-blockwise-attention