---
title: Constant Memory Attention Block
url: https://www.emergentmind.com/topics/constant-memory-attention-block
type: topic
---

# Constant Memory Attention Block

A Constant Memory Attention Block (CMAB) is an attention mechanism designed to enable large-scale context aggregation with provably constant auxiliary memory requirements, independent of the number of input tokens or context elements. Originally introduced in the domain of meta-learning for neural processes, CMABs and related constant-memory block attention structures have since expanded to diverse architectures and practical settings, including temporal point processes, efficient video generation, block-wise long-sequence language modeling, and photonic LLM accelerators. The unifying characteristic is the ability to compress arbitrarily large input sets into a fixed-size latent, memory, or state representation that supports constant-memory forward computation and O(1)-time streaming/dynamic updates.

## 1. Formal Definitions and Structural Properties

CMABs operate by factoring attention across (potentially large) input sets or sequences using fixed-size learned embeddings, latent slots, or external memory modules. In the canonical form [2306.12599, 2305.14567], the block compresses N input tokens $x_1, \ldots, x_N$ into $L_B$ “bottleneck” queries (learned vectors) through cross-attention:
\[
C_B = \text{CA}(\text{BEMB}, \text{INPUT})
\]
with $\text{CA}$ denoting cross-attention. This is followed by self-attention over the compressed $C_B$, then cross-attention from a fixed $L_I\times d$ (input latent) set to $C_B$ and another self-attention, yielding the block’s output:
\[
\mathrm{CMAB}(\mathrm{IEMB}, \mathrm{INPUT}) = \mathrm{SA}( \mathrm{CA}(\mathrm{IEMB}, \mathrm{SA}( \mathrm{CA}(\mathrm{BEMB}, \mathrm{INPUT}))))\,.
\]
The memory cost beyond the raw input storage is dominated by fixed objects of sizes $L_B$, $L_I$ ($O(1)$ in N), and chunked processing keeps all intermediate matrices bounded [2306.12599]. Streaming updates are supported by keeping per-bottleneck accumulators for softmax normalizers and value-weighted sums, updatable in O(1) with each new datum.

Alternative structural paradigms include block-wise autoregressive linear attention with constant-memory key-value (KV) summaries [2509.24695], fixed-size external learnable memory “packing/unpacking” [2404.00798], and block self-attention hierarchies [1804.00857].

## 2. Computational Complexity and Memory Analysis

CMABs enforce strict upper bounds on peak workspace. For the canonical set attention block [2306.12599]:
- Forward pass: $O(N L_B)$ time and $O(L_B)$ active memory; all cross-attention with the context is executed in chunks or via rolled accumulators.
- Streaming update: Each new input requires only $O(L_B)$ work and storage, as only accumulators are updated.
- For model stacks (e.g., in neural processes), memory does not grow with the size of context or target sets, as all further attentions are applied to constant-size latents.

Block-wise linear attention structures achieve true constant-memory scaling by maintaining only block-wise compressed summaries. In SANA-Video, cumulative kernelized key-value matrices $S_{\text{cache}}\in\mathbb R^{D\times D}$ and $Z_{\text{cache}}\in\mathbb R^{D\times 1}$ are sufficient; per-block updates and computations remain $O(D^2)$ regardless of total sequence/token count [2509.24695].

Memory-augmented approaches as in ConvLuna restrict the token-to-memory interaction to $O(L M)$ (for $L$ inputs, $M$ memory slots), and $M$ is fixed; thus peak attention matrix storage, as well as the total synthesized feature map, are constant in $L$ [2404.00798]. Block self-attention, as in Bi-BloSAN, splits into intra-block and inter-block attention, confining high-memory operations within blocks of size $B$ or across $N/B$ blocks; memory becomes $O(B^2 + (N/B)^2)$, minimized for $B\approx(2N)^{1/3}$ [1804.00857].

## 3. Algorithms and Streaming Update Mechanisms

The core computational primitive is an accumulator-based approach to attention. For the set-to-bottleneck cross-attention,
\[
M_j = \frac{E_j}{C_j} \quad\text{where} \quad E_j = \sum_{i=1}^N \exp(s_{i,j}) v_i,\quad C_j = \sum_{i=1}^N \exp(s_{i,j})
\]
and $s_{i,j} = b_j \cdot k_i$. On streaming updates, arrival of $k_{N+1},v_{N+1}$ allows immediate computation of the update for every $b_j$:
\[
C'_j = C_j + \exp(s_{N+1,j}),\qquad E'_j = E_j + \exp(s_{N+1,j}) v_{N+1}
\]
and correspondingly the output $M'_j$ can be updated with simple arithmetic [2306.12599, 2305.14567]. These formulas can be batched or stabilized in log space with no loss of O(1)-per-step guarantee.

For blockwise linear attention, constant-memory is enforced via kernel tricks: all of history up to the current block is compressed in $S_{\text{cache}}$ and $Z_{\text{cache}}$, each block processes local tokens, then merges its summary into the cache [2509.24695]. Lightning Attention [2405.17381] employs intra-block softmax attention (masking remains $O(B^2)$) and inter-block kernel-accumulated updates, guaranteeing no cumulative summation buffer ever scales with $n$.

Photonic block-selection as in PRISM [2603.21576] eschews memory-bound electronic searching for O(1) “broadcast-and-weight” inner-product computation: all block similarities to a query are evaluated simultaneously in hardware, so (electronic) memory usage for KV selection is strictly bounded regardless of total context length.

## 4. Applications and Benchmarks

### Meta-learning and Neural Processes  
CMABs have been integrated into Neural Processes (specifically in Constant Memory Attentive Neural Processes, CMANPs), Temporal Point Processes (CMHP), and general set-based regression, producing memory usage flat versus $N$ and outperforming or matching prior approaches on image completion (CelebA, EMNIST) and event prediction tasks [2306.12599, 2305.14567]. For example, on CelebA128, CMANP achieved 5.55±0.01 log-likelihood with $\sim$1MB GPU memory, while all quadratic/linear-memory baselines OOM at this scale.

### Video Generation  
In SANA-Video, block linear diffusion with a constant-memory KV cache enables efficient autoregressive video synthesis with minute-scale temporal horizons, O(1) memory, and real-time throughput on commodity RTX 5090 GPUs [2509.24695]. Compared to state-of-the-art models (e.g., Wan 2.1/SkyReel-V2-1.3B), SANA-Video achieves 16× lower latency and maintains competitive generation quality.

### Language Modeling and Blockwise LLM  
Lightning Attention [2405.17381] demonstrates constant SRAM memory for all block-wise linear attention at any sequence length, with measured GPU memory at n=64K fixed at 10GB (compared to 160GB for FlashAttention softmax), with little loss in token-level throughput or downstream perplexity. Photonic KV block-selection architectures [2603.21576] reduce memory traffic by >16× and energy by 3–4 orders of magnitude relative to conventional GPU implementations at large context lengths (up to 128K tokens).

### Block Attention for Efficient Sequence Models  
Bi-BloSAN [1804.00857] shows high-accuracy, memory-efficient sequence encoding: intra-block and inter-block self-attention constructions sustain task performance on standard NLP benchmarks with drastically reduced memory and computation compared to full self-attention.

Empirical results across these domains consistently demonstrate that constant-memory block attention mechanisms enable tractable scaling to large input sets, outperform or match higher-memory baselines, and are particularly well-suited for resource-constrained or latency-critical deployments [2306.12599, 2509.24695, 2405.17381].

## 5. Design Choices, Limitations, and Trade-offs

Key design choices include bottleneck/slot count (in CMAB), block size (in blockwise structures), and the filtering operator for input compression (kernel pooling or convolution in ConvLuna [2404.00798]). Increasing bottleneck size improves representational power but increases per-block memory cost. Filtering (e.g., max-pool, Conv1D) before memory writes prevents memory collapse and keeps prototype utilization—even $M=1$ outperforms a full Transformer on some long-range tasks [2404.00798].

Streaming updates rely on numerical stability: log-sum-exp machinery avoids catastrophic overflow, but in very long runs, drift can accumulate, so periodic batch recomputation may be necessary [2306.12599]. CMABs, like all compressed-set techniques, can lose fine-grained pairwise dependencies—hierarchical or composite mixtures could potentially alleviate this at fixed cost.

Hardware constraints for photonic selection (PRISM) include limited microring (MRR) count and wavelength-division multiplexing channel bandwidth, though practical multi-chip and time-multiplexed configurations suffice for current model sizes [2603.21576].

Limitations include:
- Not all global dependencies or subtle input interactions are preserved in the compressive representation; tasks requiring full fidelity may see information loss.
- For blockwise methods, block size and stride impact both local/global trade-off and hardware efficiency.
- Pure residual memory updates risk slot collapse or under-utilization; exploration of gated or selective update mechanisms is ongoing [2404.00798].

## 6. Broader Trends, Variants, and Outlook

Constant memory attention blocks have generalized rapidly from meta-learning to efficient sequence modeling, long-context language model inference, video generation, and hardware accelerators. Distinct instantiations include:
- Cross-attention with fixed-size latent queries (CMAB) [2306.12599]
- Hierarchical block self-attention (Bi-BloSAN) [1804.00857]
- Kernel-based block linear attention with constant-memory accumulators (SANA-Video, Lightning Attention) [2509.24695, 2405.17381]
- External learnable shared memory with filtering and packing-unpacking (ConvLuna) [2404.00798]
- Photonic block-similarity engines for KV selection (PRISM) [2603.21576]

Open directions include adaptive bottleneck sizing, integrating sparse/low-rank/factorized attention with constant-memory affordances, improving memory update rules (dynamic, gated, selective), and extending current block attention mechanisms to cross-modal and generative scenarios beyond classification and regression [2306.12599, 2509.24695, 2404.00798].

By compressing all history into fixed-size statistics and modularizing computation and updates blockwise or via external memory, constant-memory attention blocks enable scalable, efficient, and streaming-capable architectures for long-context and on-device applications, while sustaining competitive empirical performance across modalities and paradigms.

Source: https://www.emergentmind.com/topics/constant-memory-attention-block