Compressed Attention Techniques

Updated 3 July 2026

Compressed Attention (CA) is a family of methods that compress or re-encode token representations to reduce the quadratic time and memory costs in standard attention mechanisms.
CA methods employ techniques like token-wise summarization, quantization, and latent-based compression to achieve significant memory and speed improvements across language, vision, and multimodal tasks.
These approaches enable efficient training and inference for long-context models by strategically reducing the key-value cache size and using hardware-aware optimizations with minimal accuracy loss.

Compressed Attention (CA) encompasses a broad family of techniques for reducing the computational and memory complexity of attention mechanisms by replacing global dense computation with operations over a compressed or reduced representation. These methods are fundamentally motivated by the quadratic scaling of standard attention in sequence length and are applicable in language, vision, and multimodal modeling. The following sections provide a technical synthesis, taxonomy, key algorithms, and empirical findings from the current literature.

1. Core Principles and Motivation

Standard self-attention computes global dependencies between all input tokens, incurring $O(N^2 d)$ time and memory complexity ( $N$ = sequence length, $d$ = hidden dimension). This quadratic scaling becomes prohibitive for long contexts and large batches, especially in online or streaming applications, as well as high-resolution visual domains. Compressed Attention (CA) approaches address this by compressing, selecting, or re-encoding the key-value (KV) cache and/or input tokens so that attention computations are performed over a much smaller set—either via structured summarization, learned mapping, lossy/lossless condensation, or algorithmic sparsification. These reductions may target (a) asymptotic speed/memory, (b) hardware-bounded deployments, or (c) enabling qualitatively new regimes (e.g., million-token context, sublinear attention memory) (Kim et al., 2023, Wang et al., 1 Dec 2025, Sun et al., 15 Jun 2025, Figliolia et al., 6 Oct 2025, Yang et al., 26 Jul 2025, Vegasena, 18 Apr 2026, Wang et al., 2024, Prakash et al., 7 Nov 2025, Jaber et al., 4 May 2026, Becker et al., 20 Mar 2025, Schröder et al., 10 Feb 2026, Wen et al., 21 Sep 2025, Li et al., 2021).

2. Algorithmic Approaches to Compression

The literature presents several principal strategies for attention compression:

a) Token-wise Summarization and Memory Approaches

Compressed Context Memory (CCM): At each step, a growing context is summarized into a fixed-size per-layer “memory” using special compression tokens. The memory is updated by concatenating or recursively merging new representations, with all downstream attention directed only to memory plus current input, yielding 5–8x memory savings (Kim et al., 2023).
Compress & Attend Transformer (CAT): Input is partitioned into chunks, each chunk compressed via a learnable operator (e.g., transformer encoder), with chunkwise representations forming the context for subsequent attention (Prakash et al., 7 Nov 2025).
LoMA (Lossless Compressed Memory Attention): Supports perfect, lossless KV compression by training the model to periodically distill a long context into a smaller segment using special memory tokens and learned attention masks. Empirically achieves up to 8× cache reduction with near 100% recall (Wang et al., 2024).

b) Descriptor- and Latent-based Compression

Compressed Descriptor Attention (CDA): In visual models, spatial tokens from frames are bilinearly downsampled to create descriptors; global attention is replaced by cross-attention between tokens and descriptors. This approach enables $O(N^2 / r^2)$ cost for a downsampling factor $r$ (Wang et al., 1 Dec 2025).
GTA (Grouped-Head Latent Attention): Reduces KV memory and attention computation by sharing keys/values and/or attention maps across groups of heads, and by parameterizing values in a lower-dimensional latent space decoded via nonlinear transforms (Sun et al., 15 Jun 2025).
Compressed Convolutional Attention (CCA/CCGQA): Projects Q/K/V into a shared compressed latent (via 1D and grouped convolutions), conducting all attention in this space and mapping back to the output. CCGQA combines this with head-sharing, realizing ≥8× cache reduction (Figliolia et al., 6 Oct 2025).

c) Quantization and Hardware-Driven Compression

Open-TQ-Metal / HCAttention: Compresses the KV cache via low-bit quantization (e.g., int4 per-group asymmetric quantization) and implements attention-by-decompressing-on-the-fly within GPU kernels, avoiding FP16/FP32 cache instantiation. Fused computation and online softmax achieve 40–48× speedup while enabling extremely large contexts (≥128K tokens) on consumer hardware (Vegasena, 18 Apr 2026, Yang et al., 26 Jul 2025).
Dynamic Eviction and Offloading: Retains only the most-attended-to keys/values and offloads less frequently used values to CPU, with dynamic thresholds set per layer. This enables up to 8× cache compression with sub-1% accuracy loss for long LLM contexts (Yang et al., 26 Jul 2025).

d) Sparsification and Coreset Selection

WildCat: Uses a randomly pivoted Cholesky factorization to select a small, optimally weighted “coreset” of keys/values, approximating attention with super-polynomial decay in error at near-linear cost. Optimal Nyström-style weighting is applied, with theoretical and empirical bounds (Schröder et al., 10 Feb 2026).
StreamIndex (Compressed Sparse Attention): Replaces dense indexer-score tensors with chunked, streaming top- $k$ computation for compressed sparse attention. Eliminates intermediate materialization, achieving extreme memory efficiency and scaling to million-token sequences without loss (Jaber et al., 4 May 2026).

e) Analytical and Unified Compression

Contract-and-Broadcast Self-Attention (CBSA): Emerges from a maximal coding rate reduction objective. Contracts all tokens to a small set of subspace representatives, broadcasts these back, and interprets attention as a gradient update on a coding-rate functional. Special cases include softmax attention, linear/agent attention, and channel attention (Wen et al., 21 Sep 2025).

f) Layer Fusion and Compressed Decoder Design

Compressed Attention Network (CAN): Fuses standard decoder sublayers (self-attn, cross-attn, feedforward) into a single joint sublayer by algebraic manipulation, exploiting high similarity between adjacent activations. Achieves O(t²d+t sd+6td²) → O((t+s)td+4td²) complexity for t,target; s,source length, maintaining accuracy (Li et al., 2021).

g) Domain-specific Adaptations

EDiT / Linear Compressed Attention: Applies multi-layer convolutions to locally modulate queries and spatially compress keys/values for image-based diffusion transformers. Linear kernelized attention follows, yielding linear cost in spatial size (Becker et al., 20 Mar 2025).
AMPA-Net: In deep compressed sensing, integrates three attention forms (initialization, spatial, channel) into an unrolled optimization-inspired network, boosting PSNR by up to 1 dB at negligible additional compute (Li et al., 2020).

3. Training, Inference, and Implementation Considerations

Parallelized and Efficient Training

Many compression schemes (e.g., CCM, LoMA) introduce custom attention masks and structured input sequences to support parallel computation of recursively compressed summaries within a single forward pass. This enables training efficiency: for example, CCM enables 7× faster training than prior recurrent compressor baselines, while LoMA's lossless compression is enforced via repetition loss, backpropagating gradients into compressed tokens (Kim et al., 2023, Wang et al., 2024).

Inference Protocols

Most CA methods permit incremental, streaming inference. CCM updates a fixed-size memory at each timestep, CAT and LoMA alternate compression and generation, quantized-domain approaches perform attention calculation in compressed space, and CCA/CCGQA maintain the compressed latent or grouped cache in both train and inference phases (Wang et al., 1 Dec 2025, Figliolia et al., 6 Oct 2025, Prakash et al., 7 Nov 2025, Vegasena, 18 Apr 2026). Methods such as HCAttention require hardware-aware orchestration of offloading and dynamic eviction logic (Yang et al., 26 Jul 2025).

Complexity Profiles

A representative summary is provided below:

Method	Memory	Attention FLOPs	Compression Ratio
Full context	$O(N d)$	$O(N^2 d)$	1× (baseline)
CCM-concat	$O(t + l_i)$	$O(t l_i + l_i^2)$	5×–8× (Kim et al., 2023)
CDA (FlashVGGT)	$N$ 0	$N$ 1	10–16× (Wang et al., 1 Dec 2025)
GTA	$N$ 2	$N$ 3	1.4–3.3× (Sun et al., 15 Jun 2025)
CCA/CCGQA	$N$ 4	$N$ 5	4–8× (Figliolia et al., 6 Oct 2025)
Quantized (int4)	$N$ 6	$N$ 7	3.2× (Vegasena, 18 Apr 2026)
LoMA	$N$ 8	$N$ 9	up to 8× (Wang et al., 2024)
WildCat	$d$ 0	$d$ 1	variable (Schröder et al., 10 Feb 2026)

This table highlights the diversity of tradeoffs in design, with specialized hardware-aware reductions (e.g., quantized CA) and domain-adapted methods (e.g., EDiT's convolutional image compression).

4. Empirical Results, Benchmarks, and Comparisons

Language Modeling and Long-context Inference

CCM attains MetaICL accuracy of 70.0% (5× memory reduction, CCM-concat) versus 70.8% (full), with throughput increasing from 5.3 to 24.4–69.9 samples/s at batch size 300–950 (Kim et al., 2023).
LoMA reduces KV-cache by 4–8×, matching in-context recall to within 0.1% for $d$ 2, and accelerates autoregressive decoding by up to 75% (Wang et al., 2024).
CAT models trained across chunk sizes achieve 1.4–3.2× throughput and 2.2–9.5× lower memory than dense transformers, with improved in-context recall for small chunk sizes (Prakash et al., 7 Nov 2025).
GTA achieves up to 70% cache reduction, 62.5% theoretical compute saving, and maintains <1–2 point loss on standard extractive benchmarks compared to Grouped Query Attention (Sun et al., 15 Jun 2025).
HCAttention processes up to 4 million tokens, preserving full-attention accuracy with only 25% of the GPU KV cache, and graceful degradation (<1 point) down to 12.5% cache (Yang et al., 26 Jul 2025).
Open-TQ-Metal enables Llama 3.1 70B to run at 128K context on 64GB Mac with a 48× attention speedup and 3.2× memory reduction (Vegasena, 18 Apr 2026).

Vision and Multimodal Contexts

FlashVGGT's CDA achieves 90% speedup (~10–15×) and <5% loss in Chamfer distance for 3D multi-view geometry, scaling beyond 3,000 frames (Wang et al., 1 Dec 2025).
CBSA matches or exceeds ViT-Base in ImageNet-1K accuracy with lower FLOPs and memory, and demonstrates robust interpretability properties (Wen et al., 21 Sep 2025).
EDiT (LCA) yields ≈2.5× faster diffusion image synthesis at 2048×2048 pixels with ≤1 FID point degradation (Becker et al., 20 Mar 2025).
AMPA-Net outperforms previous CS reconstruction networks by up to 1 dB PSNR with negligible compute increase (Li et al., 2020).

Sparse and Coreset Attention

WildCat offers the first super-polynomial decay error guarantee, empirical 3–10× speedup on real benchmarks, and state-of-the-art memory vs. fidelity in cache compression tasks (Schröder et al., 10 Feb 2026).
StreamIndex enables V4-Flash–style CSA to scale to S=1 million tokens at 6.21 GB HBM (vs. 256 GB+ for dense), with bit-exact top-k recall at lower S (Jaber et al., 4 May 2026).

Decoder and Efficient Architectural Fusion

CAN achieves 2.8× speedups on machine translation with BLEU within 0.3 points versus a standard strong baseline, halving the effective kernel pipeline depth and memory (Li et al., 2021).

5. Tradeoffs, Limitations, and Future Directions

CA introduces fundamental tradeoffs between accuracy, speed, and memory:

Aggressive compression may induce small but measurable degradation in accuracy or recall, notably at very high ratios (e.g., ≥16×).
For approaches relying on quantization, specific scaling coefficients or model architectures are determinative: e.g., per-group int4 robust across scale, but angular (PolarQuant) schemes fail for attentionscale=1 (Vegasena, 18 Apr 2026).
Some methods (e.g., LoMA, CCM) guarantee losslessness only up to moderate compression ( $d$ 3); higher compression introduces empirical loss in recall or increased error (Wang et al., 2024).
Hybrid architectures involving both compressed attention and specialized hardware pipelines (e.g., offload, streaming) demand careful synchronization and latency-aware implementation (Yang et al., 26 Jul 2025, Jaber et al., 4 May 2026).
Unified objectives (e.g., CBSA's coding-rate) offer interpretability and a path toward principled layer-wise compression, but further research is warranted in initialization, early-layer decompression, and extension to broader function classes (Wen et al., 21 Sep 2025).

Future work investigates:

Layer-wise or adaptive compression schedules.
Integration of non-linear transforms onto compressed latent spaces.
Exploiting joint compression across multiple attention heads and block structures.
Hardware/OS-level fusion and extension to multimodal models with even larger memory footprints.
Theoretical understanding of expressivity under aggressive compression—i.e., formalizing the exact tradeoffs and loss boundaries.

6. Theoretical Guarantees and Interpretability

CA techniques vary in their formal properties:

WildCat provides the first super-polynomial ( $d$ 4) approximation guarantee for attention with near-linear runtime (Schröder et al., 10 Feb 2026).
CBSA emerges from a maximal coding-rate reduction, exposing the attention mechanism as a natural compression operator and unifying quadratic, linear, and subspace-specific variants under one analytical framework (Wen et al., 21 Sep 2025).
Lossless CA (LoMA): For moderate $d$ 5, token-level recall reaches $d$ 6 with empirically verified zero cross-entropy on repetition zones (Wang et al., 2024).

Other methods rely primarily on empirical error and resource curves, with ablation studies revealing Pareto frontiers between accuracy and efficiency (e.g., CDA's $d$ 7 vs. inference time vs. Chamfer-Distance curve; GTA's FLOPs vs. cache vs. error, etc.) (Wang et al., 1 Dec 2025, Sun et al., 15 Jun 2025).

7. Synthesis and Taxonomy

Compressed Attention constitutes a paradigm—rather than a single algorithmic instance—encompassing procedures which:

Select, compress, or reweight subsets of tokens, KV pairs, or representations for scalable and efficient attention.
Operate via specialized memory structures, attention masks, quantization, clustering/sampling, or hybrid hardware logic.
Demonstrate domain- and modality-adaptation: from autoregressive LMs and translation to high-resolution image synthesis, compressed sensing, and online streaming models.

This paradigm continues to drive advances in practical long-context inference, interpretable model design, hardware-efficient deployment, and theoretical understanding of attention model capacity and generalization.