Chunked and Blockwise Attention

• Chunked and blockwise attention is a method that partitions long sequences into contiguous blocks to perform efficient local self-attention.
• This approach reduces memory and compute complexity from quadratic to linear by focusing processing within smaller, manageable segments.
• Enhanced variants, including overlapping, shifted, and hierarchical block designs, enable the capture of global dependencies while optimizing speed and resource usage.

Chunked and Blockwise Attention refers to a family of attention mechanisms in deep learning models—typically Transformer architectures—where the input sequence is partitioned into contiguous blocks or chunks, and attention is performed either locally within each chunk or hierarchically/compositionally across such blocks. This strategy is motivated by the need to scale attention and memory to very long sequences, improve efficiency, and facilitate streaming or online processing while still capturing essential local and global dependencies.

1. Formal Definitions and Variants

Chunked (or blockwise) attention partitions a sequence $X=[x_1, x_2, ..., x_L] \in \mathbb{R}^{L \times d}$ into $M = \lceil L / C \rceil$ non-overlapping blocks of size $C$, where $C$ is the chunk size. For each chunk $X^{(m)}$ (indices $[(m-1)C+1, mC]$), queries, keys, and values are computed and attention is restricted within the same chunk: $$A^{(m)} = \mathrm{softmax} \left( Q^{(m)} [K^{(m)}]^T / \sqrt{d_k} \right), \qquad Y^{(m)} = A^{(m)} V^{(m)}$$ Global output is constructed via concatenation: $$Y = \mathrm{concat}(Y^{(1)}, ..., Y^{(M)}), \qquad Y \in \mathbb{R}^{L \times d}$$ Variants augment local attention with additional mechanisms: overlapping blocks with strided or sliding windows (Wang et al., 2021), cross-chunk attention by shifting key/value blocks (Guo, 2023), and hierarchical, multi-stage chunking for progressive context expansion (Ju et al., 2021).

2. Mathematical Formulations and Computational Complexity

• Memory and Computation: For standard self-attention over sequence length $L$, complexity is $\mathcal{O}(L^2 d)$ in both compute and memory due to the dense $M = \lceil L / C \rceil$0 attention matrix. Chunked/blockwise attention reduces this to $M = \lceil L / C \rceil$1 (with $M = \lceil L / C \rceil$2), as only $M = \lceil L / C \rceil$3 attention is performed per chunk (Terzic et al., 2023, Qiu et al., 2019).
• Hierarchical/Multistage: Multistage chunking (as in ChunkFormer) performs multiple passes, each with growing chunk size $M = \lceil L / C \rceil$4, increasing the effective receptive field in a staged fashion:

$M = \lceil L / C \rceil$5

where $M = \lceil L / C \rceil$6 is the number of stages (Ju et al., 2021).

• Block-Sparse and Hybrid: Advanced blockwise schemes combine dense intra-block attention with block-sparse cross-block connections for global context at reduced cost, sometimes guided by principled selection criteria (Wang et al., 29 Jan 2026).

3. Applications and Model Designs

Chunked/blockwise attention appears in diverse domains:

• Long-Document Language Modeling: BlockBERT uses block-sparse patterns, sometimes with head-dependent permutations to mix local and global contexts (Qiu et al., 2019). ChunkLLM adopts chunk boundary detection and compression for efficient inference over 120k-token inputs (Ouyang et al., 28 Sep 2025).
• Streaming and Online ASR: Blockwise encoders and decoders enable low-latency, streamable speech recognition via strict chunked attention windows (Zeineldeen et al., 2023, Wang et al., 2021, Liu et al., 2020). MoChA and its multi-head variant dynamically determine chunk boundaries for monotonic alignment (Liu et al., 2020).
• Hierarchical Time Series or Sequential Data: ChunkFormer demonstrates accelerated and stabilized training on long time series, learning local and global seasonality/hierarchies (Ju et al., 2021). TCNCA integrates chunked attention with dilated convolutions for linear-in-length sequence processing (Terzic et al., 2023).
• Efficient Large-Context Transformers: Blockwise Parallel Transformers and Ring Attention push context lengths into millions of tokens by distributing blockwise attention across devices for near-linear scaling (Liu et al., 2023, Liu et al., 2023).
• Memory-Augmented Architectures: Models such as the memory-augmented chunked Transformer use gated FIFO recurrent memory to bridge chunk boundaries (Kashyap, 1 Jul 2025). SPLA combines block-sparse selection with residual linear attention to compress and retain “long tail” context (Wang et al., 29 Jan 2026).
• Generative Diffusion and Video: Blockwise (chunked) attention underlies efficient attention caching and reuse in diffusion models, where block-internal and block-external attention are fused for throughput gains (Chen et al., 5 Feb 2026).
• Multimodal and Social Signal Modeling: Blockwise masking can be tailored for causal, multimodal, and agent-wise chunking, as in cross-modal social signal prediction (Tang et al., 23 Jan 2025).

4. Receptive Field and Information Flow

Chunked attention is inherently local, but expressivity is extended by various strategies:

• Cross-Chunk Communication: Multi-stage chunking, memory augmentation, or cross-block sparse connections allow global information aggregation (Ju et al., 2021, Kashyap, 1 Jul 2025, Qiu et al., 2019, Wei et al., 6 Jul 2025).
• Shifting and Dilated Patterns: Shifted Cross Chunk Attention (SCCA) and Shifted Dilated Attention use systematic key/value shifts or dilations across heads or layers. This dramatically accelerates receptive-field growth, approximating global attention at linear cost (Guo, 2023).
• Hybrid Approaches: Models such as RAT interleave recurrent intra-chunk processing with attention across chunk summaries, combining efficiency with recoverability of distant dependencies (Wei et al., 6 Jul 2025).

5. Empirical Results and Trade-offs

Empirical studies demonstrate:

• Memory and Throughput: Memory usage drops from $M = \lceil L / C \rceil$7 to $M = \lceil L / C \rceil$8 (Terzic et al., 2023, Liu et al., 2023). In practical systems, per-layer activation memory is reduced by $M = \lceil L / C \rceil$9 or more; context lengths can extend to millions of tokens (Liu et al., 2023, Liu et al., 2023).
• Modeling Efficacy: Chunked/blockwise attention can match or exceed the accuracy of full attention on long-sequence tasks, provided global communication is not overly restricted. For example, on language modeling and QA tasks, Blockwise-2x86 or multi-stage chunking recovers or exceeds RoBERTa/vanilla Transformer accuracy while saving 20–36% of memory (Qiu et al., 2019, Ju et al., 2021).
• Efficiency vs. Contextual Resolution: Chunk size $C$0 is central: smaller $C$1 enhances locality and speed but truncates dependencies. Large $C$2 recovers more global context, at the cost of increased compute per chunk, and may bring back quadratic cost if not carefully chosen (Terzic et al., 2023, Ju et al., 2021).
• Streaming and Latency: In online ASR, blockwise attention achieves latencies as low as $C$3140 ms at minor WER penalty compared to full-sequence models. Overlapping blocks and dynamic mapping mitigate chunk boundary artifacts (Wang et al., 2021, Zeineldeen et al., 2023).
• Accuracy vs. Latency Trade-offs: For chunk-based transducers, increasing chunk size improves BLEU and WER but adds latency and inference cost (Xu et al., 27 Feb 2026).

Method/Model	Task/Domain	Empirical Highlights
ChunkFormer	Long time series	Macro F1 +1–3%; memory $C$4
BlockBERT	Question answering	$C$5–$C$6% less memory, $C$7–$C$8% faster
Ring Attention	LLM, RL	4M-token context, $C$91/P scaling
ChunkLLM	LLM inference	4.48× speedup, 98.6% accuracy retained
CHAT	Streaming ASR/ST	Mem −46.2%, train 1.36× faster, +6.3% WER rel. gain
TCNCA	Language modeling	BPC 1.01 (lower), 1.37× speedup, up to 7× faster TCN
RAT	Long-context LM	7–9× speedup, same perplexity as standard attention
FlashBlock	Diffusion/video/text	1.44× throughput, 1.6× attention time reduction

6. Limitations and Extensions

• Locality Bottleneck: Pure chunked attention, without cross-chunk links, cannot propagate information globally within a single layer and must rely on multi-layer or additional recurrent/global paths (Guo, 2023, Kashyap, 1 Jul 2025).
• Block Boundary Effects: Fixed block boundaries may disrupt dependencies at segment edges unless overlapping, shifting, or dynamic mapping strategies are used (Wang et al., 2021, Guo, 2023).
• Parameterization Choices: Permutation patterns, hierarchical chunking schedules, and chunk/block size are highly scenario- and resource-dependent.
• Plug-and-Play Integration: Some schemes (e.g., SCCA, ChunkLLM) are designed to integrate with efficient kernel implementations like FlashAttention and parameter-efficient fine-tuning frameworks (LoRA) (Ouyang et al., 28 Sep 2025, Guo, 2023).
• Extensions: Blockwise attention ideas are extended to multimodal, temporal, and agent-wise scenarios by introducing sub-blocks for modality, participant, or segment (Tang et al., 23 Jan 2025).

7. Context and Significance

The development of chunked and blockwise attention mechanisms is central to unlocking efficient, scalable, and streaming machine learning systems for long sequences—a prevailing challenge in language modeling, speech recognition, video, and beyond. By trading dense global attention for well-structured local and compositional context, these approaches offer practical and theoretically grounded speedups, massive memory savings, and substantial flexibility. They are now fundamental primitives in the growing repertoire of efficient large-context neural architectures (Ju et al., 2021, Liu et al., 2023, Liu et al., 2023, Guo, 2023, Xu et al., 27 Feb 2026, Wei et al., 6 Jul 2025).