Chunked Attention: Methods & Trade-Offs

• Chunked attention is an attention mechanism that segments token sequences into blocks, clusters, or head groups to reduce quadratic computation while preserving local context.
• It employs strategies such as fixed contiguous chunking, adaptive learned routing, and cache partitioning to restore inter-chunk communication and maintain long-range dependencies.
• Applications span long-context LLM inference, time-series forecasting, and speech recognition, achieving notable speed-ups and memory efficiency improvements.

Chunked attention is a class of attention mechanisms and execution strategies that partitions attention work into smaller units—most often contiguous token blocks, but also attention heads, semantic clusters, or KV-cache segments—in order to reduce the quadratic cost of dense attention, preserve local structure, enable streaming, or improve communication and memory behavior during inference (Ju et al., 2021, Yuan et al., 2 Jul 2026, Bai et al., 2024). Across long time-series forecasting, diffusion language modeling, long-context LLM inference, speech recognition, and systems kernels, the shared principle is to replace a monolithic attention computation with chunk-local or chunk-indexed computation, then recover broader context through re-chunking, recurrence, summaries, routing, or cache orchestration (Zhu et al., 15 May 2026, Mao et al., 22 Apr 2026).

1. Scope and taxonomy

The term does not refer to a single algorithm. In the literature summarized here, chunked attention includes sequence-chunked or blockwise attention, head-chunked overlap schemes, learned semantic chunking, and serving-time cache chunking. HCMS explicitly distinguishes two common variants: sequence-chunked attention, which partitions token positions into blocks, and head-chunked attention, which partitions multi-head attention along the head dimension for independent scheduling (Yuan et al., 2 Jul 2026). Other works enlarge this taxonomy by defining chunks through learned semantic assignments, gist summaries, or runtime cache units rather than fixed positions (Zhu et al., 15 May 2026, Mao et al., 22 Apr 2026).

Chunk axis	Representative mechanism	Example
Sequence positions	Attention restricted to contiguous token windows or blocks	ChunkFormer (Ju et al., 2021)
Attention heads	Independent scheduling of head groups	HCMS (Yuan et al., 2 Jul 2026)
Semantic or learned groups	Tokens routed into learned chunks or summarized by learned chunk tokens	DCDM (Zhu et al., 15 May 2026)

A further systems-oriented usage appears in long-context serving, where chunks are not merely modeling units but also memory-management units. Prefix-aware KV chunks in a prefix tree, query-block/KV-block unions, and chunked state-eviction loops all treat chunking as a way to organize cache reuse and runtime access patterns rather than solely as an inductive bias in the attention mask (Ye et al., 2024, Song et al., 16 May 2026, Bai et al., 2024).

This breadth matters because the same phrase can denote materially different objectives. In ChunkFormer, chunking emphasizes local seasonality and memory efficiency for long time series; in HCMS, it enables communication–computation overlap; in SSA and ChunkLLM, it becomes a learned routing interface for long-context inference (Ju et al., 2021, Yuan et al., 2 Jul 2026, Mao et al., 22 Apr 2026, Ouyang et al., 28 Sep 2025).

2. Core mathematical patterns

A common starting point is standard multi-head attention,

$$A^{(h)}=\mathrm{softmax}\!\left(\frac{Q^{(h)}(K^{(h)})^\top}{\sqrt{d_k}}+B\right)V^{(h)},$$

after which chunked attention modifies the admissible keys, values, or execution order rather than abandoning dot-product attention altogether (Lu et al., 2024). In sequence-chunked models, the canonical modification is a block-diagonal or block-local mask. ChunkFormer makes this explicit at stage $s$ by restricting attention to contiguous subsets so that

$$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$

with later stages using larger chunk sizes to widen the receptive field while preserving the original sequence length through concatenation (Ju et al., 2021).

A second recurring pattern is chunk-causal ordering. In DCDM, hard cluster identities $c_\ell$ define semantic chunks

$\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$

and the inference-time chunk-causal mask

$$\mathbf{M}^{\text{chunk}}_{\ell,m}=\mathbb{I}[c_m \le c_\ell]$$

permits bidirectional attention within a chunk and one-way conditioning from later chunks to earlier chunks. This converts fixed positional block factorization into autoregression over learned semantic chunks, which the paper describes as a strict generalization of block discrete diffusion (Zhu et al., 15 May 2026).

A third pattern replaces token-level global access with chunk summaries. RAT forms per-chunk summary pairs

$$s_g^K:=\tilde K_{g,c},\qquad s_g^V:=\tilde V_{g,c},$$

then lets each token attend over all preceding chunk summaries plus its own chunk prefix summary, reducing the effective attention span from $N$ tokens to $G=N/c$ chunks (Wei et al., 6 Jul 2025). In serving-time sparse execution, the same reduction appears at the mask level: CompactAttention converts per-$(Q\text{-block},KV\text{-block})$ sparsity into per-group KV tables via

$s$0

followed by in-place dense attention over the selected block union (Song et al., 16 May 2026).

These constructions show that chunked attention is less a single masking rule than a family of constrained-access operators. The common structural move is to compress the admissible attention domain from all past tokens to a smaller chunk-indexed set, while introducing an auxiliary mechanism that restores useful cross-chunk information.

3. Fixed contiguous chunking and hybrid sequence models

The most direct form of chunked attention partitions a sequence into fixed contiguous windows and computes attention independently inside each window. ChunkFormer does so in a multi-stage encoder for long time series: a length-$s$1 sequence is partitioned into chunks of size $s$2 at stage $s$3, attention is computed only within each chunk, and later stages use larger chunk sizes so that information propagates from local fluctuations and seasonality to broader temporal structure without changing the total sequence length (Ju et al., 2021). The paper reports that 2–3 stages were sufficient to reach state-of-the-art performance on the reported datasets, that space complexity is reduced from $s$4 to $s$5 with $s$6, and that ChunkFormer consistently improves Macro F1 over regular Transformer, LSTM, and LogSparseFormer across sequence lengths $s$7, with Macro F1 stability of about $s$8 across lengths on CDN versus about $s$9 for LSTM (Ju et al., 2021).

TCNCA couples fixed non-overlapping chunked attention with a temporal convolutional network that supplies cross-window mixing before attention. Attention remains local to a chunk, but the preceding dilated TCN has an exponentially growing receptive field, which the paper presents as the mechanism that alleviates chunk-boundary isolation (Terzic et al., 2023). Under fixed chunk size $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$0, the model scales as $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$1, reaches 1.01 BPC on EnWik8, and is reported as $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$2 and $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$3 faster than MEGA in the forward and backward pass during training, with an average $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$4 inference speed-up on Long Range Arena at similar accuracy (Terzic et al., 2023).

RAT occupies a middle point between recurrent and attention architectures. It partitions the input into contiguous chunks, applies a gated linear recurrence within each chunk, and performs softmax attention across chunk-level summaries of preceding chunks plus the current prefix summary (Wei et al., 6 Jul 2025). With chunk size 16, the paper reports a $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$5 improvement in training speed for 100K sequence length and $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$6 in generation at the 4K position while maintaining similar performance to standard attention, and it further states that a hybrid RAT–SWA architecture consistently enhances performance while improving inference speed and reducing cache memory usage (Wei et al., 6 Jul 2025).

Taken together, these designs indicate a recurring separation of roles. Exact local computation is preserved inside chunks, while long-range transport is delegated to progressively larger chunks, dilated convolutions, or chunk summaries. This suggests that fixed chunking is most effective when it is paired with a secondary mechanism that explicitly compensates for the loss of direct cross-chunk token access.

4. Learned and adaptive chunk routing

A distinct line of work replaces fixed positional chunking with learned or query-adaptive chunk formation. DCDM argues that fixed positional blocks in block diffusion waste structure by splitting semantically coherent tokens and grouping unrelated ones. Its Chunking Attention layer clusters hidden states into $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$7 groups using learnable subspaces $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$8, produces hard chunk identities by $$A^{(s)}=\mathrm{blkdiag}\big(A^{(s)}_1,\ldots,A^{(s)}_{m_s}\big),$$9, and uses those identities to define a chunk-causal mask for the denoiser (Zhu et al., 15 May 2026). At 0.5B parameters, the reported suite-average accuracy improves from 30.96 for MDLM and 32.92 for BDLM to 33.68 for DCDM; at 1.5B, the averages are 36.14, 38.09, and 39.01 respectively. The same paper reports a non-monotonic ablation over $c_\ell$0 with an interior optimum around $c_\ell$1, and states that subspace dimension $c_\ell$2 stabilizes clustering whereas $c_\ell$3 collapses (Zhu et al., 15 May 2026).

LongHeads keeps chunks contiguous but makes their selection dynamic and head-specific. The sequence is split into fixed-size chunks, per-chunk representations are computed with in-chunk FlashAttention, and each head selects top-$c_\ell$4 chunks by correlation

$c_\ell$5

while always including the first and last chunk (Lu et al., 2024). Because selected chunks are remapped into a contiguous in-distribution window, the method is training-free and compatible with relative positional encoding. The paper reports 100% accuracy at 128k on passkey retrieval, LongBench average 30.14 at 16k on LLaMA-2-7B Base, and Cover Rate near 100% at lengths up to 16k, declining to 82% at 32k (Lu et al., 2024).

SSA uses learned gist tokens as chunk summaries. Continued pretraining interleaves one gist token after each chunk and applies a mask that forces later tokens to access the chunk only through its gist token; at inference, the current query scores only gist tokens, selects top-$c_\ell$6 chunks, and selectively unfolds the raw tokens of those chunks (Mao et al., 22 Apr 2026). On LongBench, the paper reports SSA averages of 46.20 at 8× compression, 45.39 at 16×, and 44.07 at 32×, consistently above ActivationBeacon and UniGist under the same compression ratios. In retrieval-augmented generation with Llama-3.2-1B, SSA at 8× compression is reported at 33.68, exceeding Full at 27.99 and Full-PT at 27.14 by over 5.7 points; the hierarchical H-SSA variant is described as achieving log-linear decoding complexity while maintaining or improving accuracy up to 32× compression (Mao et al., 22 Apr 2026).

ChunkLLM learns semantic chunk boundaries and chunk relevance with lightweight adapters while keeping the backbone frozen. A bottom-layer Chunk Adapter predicts chunk boundaries, and per-layer Q- and K-Adapters distill token-level attention into chunk-level scores used for key-chunk selection; selection is triggered only when the current token is detected as a chunk boundary, following the paper’s Intra-Chunk Attention Consistency observation (Ouyang et al., 28 Sep 2025). The reported outcome is 98.64% retention of long-context benchmark performance with 48.58% KV-cache retention, plus a maximum speedup of 4.48× on 120K-token inference; the boundary detector reaches precision 98.31, recall 95.54, and F1 96.91 on a 4K FineWeb-Edu test set (Ouyang et al., 28 Sep 2025).

Across these works, chunking becomes a routing problem rather than merely a locality prior. The chunk itself is no longer only a fixed span of tokens; it can be a learned semantic group, a gist-compressed unit, or a boundary-conditioned block whose relevance is query-dependent.

5. Streaming speech and transducer formulations

Speech recognition adopted chunked attention early because full-sequence attention is intrinsically non-streamable. In chunked attention-based encoder–decoder ASR, the input is divided into fixed-size chunks, attention is restricted to the current chunk, and a special end-of-chunk symbol advances the chunk pointer. The chunked AED formulation is described as equivalent to a transducer over chunk indices, with EOC playing the role of a chunk-level blank (Zeineldeen et al., 2023). On long-form TED-LIUM-v2 trials formed by concatenating up to 20 segments, the paper reports that chunked AED maintains WER of about 7.0–7.1% while the global AED deteriorates sharply, reaching 62.4% WER at 20 concatenated segments (Zeineldeen et al., 2023).

Chunked attention in Conformer Transducer encoders uses fixed, non-overlapping blocks with bidirectional attention inside a block and causal dependence across blocks. For Librispeech, $c_\ell$7 encoder frames gives average lookahead

$c_\ell$8

which the paper translates to about 0.48 s and uses to compare chunked attention fairly against autoregressive attention with lookahead (Weninger et al., 2022). Under that matched average lookahead, chunked attention yields 7.9% / 23.4% WER on Librispeech test-clean / test-other versus 8.4% / 24.8% for autoregressive attention with lookahead, and the paper summarizes this as about 5% relative WER improvement on Librispeech and 4% on a medical conversations task (Weninger et al., 2022).

Transformer Transducer work on variable attention masking likewise treats chunked masking as locally non-causal within short frame chunks and causal across chunk boundaries. Under similar latency budgets, chunked masking achieves a better accuracy–latency trade-off than fixed masking, and the reported partial result word latency drops from 835 ms for fixed masking to 453 ms for chunked masking, or from 700 ms to 409 ms with FastEmit (Swietojanski et al., 2022). The same work shows that training with variable masks yields a single configurable model that supports both streaming and second-pass acoustic rescoring, with up to 8% relative WER improvement in the rescoring scenario (Swietojanski et al., 2022).

CHAT extends RNN-T by replacing the single-frame joiner with cross-attention over the current chunk of encoder frames, while preserving streaming monotonicity at chunk resolution (Xu et al., 27 Feb 2026). The paper reports up to 46.2% reduction in peak training memory, up to 1.36× faster training, and up to 1.69× faster inference, together with up to 6.3% relative WER reduction for speech recognition and up to 18.0% BLEU improvement for speech translation. In the older monotonic chunkwise line, MTH-MoChA combines monotonic boundary selection with chunkwise soft attention and multiple heads; on AISHELL-1, the full system with pooling and MWER improves test CER from 8.96% for MoChA to 7.68%, and on 18000 hours of in-car speech it achieves 7.28% CER (Liu et al., 2020).

The speech literature therefore uses chunked attention in a particularly literal sense: chunking is the mechanism that transforms attention from an offline alignment procedure into an online one. Monotonic boundary updates, EOC symbols, and chunk-synchronous transducer lattices all operationalize chunking as a latency constraint.

6. Systems, serving, and kernel-level orchestration

In systems work, chunked attention often refers to decomposition for scheduling rather than to a new probability model. HCMS partitions the $c_\ell$9 attention heads into $\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$0 chunks and exploits the independence property

$\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$1

to pipeline input all-to-all, attention compute, and output all-to-all across two CUDA streams without modifying FlashAttention, SDPA, or NCCL kernels (Yuan et al., 2 Jul 2026). The method is reported as numerically equivalent to the baseline, with 10%–17.5% speedup over Ulysses and 5%–14.5% over Ring Attention at 31K–56K tokens, plus 6.8% end-to-end acceleration on Wan2.2; the paper recommends enabling it when the communication ratio $\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$2 exceeds 20% (Yuan et al., 2 Jul 2026).

Prefix-aware serving introduces another notion of chunking: the KV cache itself is broken into prefix-shareable chunks. ChunkAttention slices monolithic K/V tensors into 64-token chunks, organizes them in a CPU-maintained prefix tree, and uses a two-phase partition algorithm that first batches queries against shared chunks and then completes per-sequence reduction with online softmax (Ye et al., 2024). Because shared system prompts are stored once and read once in the chunk-first phase, the method remains exact and numerically stable while delivering 3.2–4.8× speedups over the baseline kernel for shared prompt lengths of 1024–4096 tokens; one reported end-to-end example reduces peak KV memory from 21.09 GB to 3.40 GB (Ye et al., 2024).

Chunked prefill creates a different bottleneck: short query chunks must attend to a growing KV cache, which weakens sparse-kernel efficiency. CompactAttention addresses this by treating 2D block-sparse masks as KV-selection signals rather than execution plans, lowering them to GQA-aware per-group block tables through Q-block union and intra-group union, then running zero-copy paged dense attention over the selected blocks (Song et al., 16 May 2026). On LLaMA-3.1-8B-Instruct, the paper reports up to 2.72× attention speedup and 1.96× end-to-end speedup at 128K context, with RULER accuracy remaining close to dense attention. StreamIndex targets an adjacent bottleneck in Compressed Sparse Attention: the lightning indexer’s materialized score tensor of shape $\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$3 (Jaber et al., 4 May 2026). Its chunked partition-merge top-$\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$4 driver never materializes the full intermediate, extending the V4-shaped indexer from OOM at $\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$5 to $\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$6 with 6.21 GB peak HBM, which the paper describes as a 32× regime extension, while mean recall rounds to 1.0000 across the reported sweeps (Jaber et al., 4 May 2026).

At larger system scale, MOCAP applies chunking to prefill-only LLM inference on wafer-scale chips. It partitions long inputs into chunks, then uses Memory-Balanced KV Reallocation to redistribute KV cache across pipeline stages and Latency-Balanced Chunk Partitioning to offset the fact that later chunks are more expensive because attention scales with prefix length (Wang et al., 22 Jun 2026). Against GPipe, the paper reports 76.4% lower end-to-end latency and 3.24× higher throughput on average, plus up to 1.31× longer supported sequence length relative to Terapipe. CItruS, by contrast, uses chunking as a memory-bounded eviction loop for decoder-only Transformers: the document is processed chunk by chunk, attention to cached states defines per-layer importance scores, and instruction-aware scoring decides which states persist across chunk boundaries (Bai et al., 2024). Under equal memory budgets, the paper reports that CItruS variants outperform Streaming LLM, TOVA, RoCo, H2O, and Standard CSE on reading comprehension, few-shot tasks, and TriviaQA while preserving perplexity, and that on passkey retrieval up to 1M tokens CItruS retrieves all passkeys on Llama 2 7B and Mistral 7B (Bai et al., 2024).

This systems literature generalizes chunked attention beyond masking. The chunk becomes the unit of pipeline balance, cache sharing, page-table construction, or streaming top-$\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$7 reduction, but the governing idea remains decomposition into independently manageable subproblems.

7. Trade-offs, sensitivities, and recurring limitations

Chunked attention methods are highly sensitive to how chunk boundaries and schedules are chosen. ChunkFormer explicitly notes that reduced accuracy can occur with random chunk-size choices and warns that if stage schedules are poorly configured, information may not propagate sufficiently across distant positions; TCNCA notes that non-overlapping windows create boundary effects that must be mitigated by a sufficiently large TCN receptive field; RAT states that overly large chunk size risks local over-compression, whereas overly small chunk size approaches full attention cost (Ju et al., 2021, Terzic et al., 2023, Wei et al., 6 Jul 2025). These are not incidental engineering details: they are direct consequences of replacing all-to-all token access with structured partial access.

Learned chunking introduces additional failure modes. DCDM reports that without load balancing, hard routing can collapse and starve clusters, and that too few or too many clusters can under-partition or over-fragment the sequence (Zhu et al., 15 May 2026). LongHeads warns that if a crucial sentence straddles two chunks, selection can miss or split critical content, and that accessible length is bounded by the product of selected chunks, heads, and layers (Lu et al., 2024). ChunkLLM notes that cross-chunk dependencies may be missed if the relevant chunk falls outside the selected top-$\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$8, and that false positives or false negatives in boundary detection can fragment or fuse semantic chunks improperly (Ouyang et al., 28 Sep 2025). SSA likewise reports that extreme compression can cause missed details when a gist token fails to encode crucial fine-grained information or a fact spans a chunk boundary (Mao et al., 22 Apr 2026).

Systems-oriented chunking has its own overhead regimes. HCMS measures gains below 1% when $\mathcal{B}_k=\{\ell\in\{1,\dots,L\}:c_\ell=k\},$9 and shows that overly large chunk count increases per-chunk overhead $$\mathbf{M}^{\text{chunk}}_{\ell,m}=\mathbb{I}[c_m \le c_\ell]$$0 enough to degrade throughput (Yuan et al., 2 Jul 2026). CompactAttention cannot recover blocks missed by the initial selector, and its unions reduce sparsity even when zero-copy paged execution makes that loss worthwhile (Song et al., 16 May 2026). MOCAP reports that its gains decline as sequence length grows because KV reallocation overhead increases, and CItruS notes that instruction-aware eviction depends on the quality and timing of the instruction: if the instruction is known only after reading, the final eviction pass still helps, but earlier chunk-level eviction cannot be steered by task-specific relevance (Wang et al., 22 Jun 2026, Bai et al., 2024).

A plausible implication is that chunked attention is best understood as a design space rather than a settled primitive. The decisive question is not whether attention is chunked, but along which axis it is chunked, what information is allowed to survive chunk boundaries, and which auxiliary mechanism—re-chunking, summaries, recurrence, monotonic alignment, cache sharing, or routing—restores the context that chunking deliberately suppresses.