Latent-Condensed Attention (LCA)

• Latent-Condensed Attention (LCA) is a family of mechanisms that reduces key-value cache growth and computational cost by compressing token embeddings into low-dimensional latent states.
• It features two complementary formulations—alternating sparse attention (ASA) and latent condensation within Multi-Head Latent Attention—balancing sliding-window and global selective attention across layers.
• Empirical evaluations demonstrate that LCA achieves significant KV-cache reduction, enhanced inference speed, and competitive performance on long-context benchmarks.

Latent-Condensed Attention (LCA) denotes a family of attention mechanisms for efficient long-context modeling that combine latent-state compression with structured sparsification of the attention context. In recent work, the term has been used in two closely related senses: as a reformulation of Alternating Sparse Attention in the context of Native Sparse Attention optimization, and as a latent-space condensation mechanism inside Multi-head Latent Attention (MLA). In both cases, the central objective is to reduce the growth of key-value (KV) cache and the cost of long-range attention while preserving competitive performance on reasoning and long-context benchmarks (Hu et al., 2 Nov 2025, You et al., 14 Apr 2026).

1. Nomenclature and scope

A source of terminological ambiguity is that LCA is associated with two distinct but related constructions. In “Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies,” the paper’s mechanism is named ASA, but the technical summary explicitly states that “ASA” in the paper corresponds to “LCA” in that exposition. There, LCA is a natively sparse attention mechanism that combines latent attention with layerwise local-global alternation. In “Latent-Condensed Transformer for Efficient Long Context Modeling,” LCA is the explicit name of a condensation procedure applied directly in MLA’s latent space (Hu et al., 2 Nov 2025, You et al., 14 Apr 2026).

Formulation	Core mechanism	Stated effect
LCA as ASA	Alternating sliding-window layers and compressed + selective layers, with MLA/GLA	50 % KV-cache reduction versus NSA
LCA in latent-condensed transformer	Query-aware pooling of semantic latents and anchor selection for positional keys	up to 2.5$\times$ prefilling speedup and 90% KV cache reduction at 128K context

The first formulation is defined relative to three baselines. Full attention attends to all past keys and values, with $O(T^2d)$ compute and $O(Td)$ memory per layer. Standard sparse attention as exemplified by NSA executes sliding-window, compressed, and selective branches in every layer, still caching full-dimensional keys and values. By contrast, the ASA-style LCA splits those branches across layers and replaces conventional grouped-query treatment with latent-state variants. The second formulation is defined relative to MLA: it retains MLA’s low-dimensional latent cache, but condenses distant history before attention is computed, thereby reducing both computation and cache footprint.

2. Layerwise LCA in native sparse attention

In the ASA-style formulation, LCA combines two orthogonal design principles. The first is Latent Attention: instead of storing full-dimension key/value vectors for every past token, the mechanism first compresses the entire sequence of token embeddings into a small set of latent states. During inference it only needs to cache those latent states, yielding memory costs comparable to Multi-Query Attention, while during training each head retains its own projections and thus the expressiveness of full Multi-Head Attention. The second is Local-Global Alternation: rather than computing both local and global sparse branches in every layer, odd-numbered layers perform sliding-window attention and even-numbered layers perform compression plus top-$K$ selective attention (Hu et al., 2 Nov 2025).

This alternation is explicit. Layer 1 applies sliding-window attention with Multi-head Latent Attention (MLA); Layer 2 applies compressed and selective attention with Group-head Latent Attention (GLA); the pattern then repeats depthwise. The summary describes this as a stratification that reduces KV-cache size by 50 % versus NSA and yields better propagation of long-range dependencies. The same source states that alternating between local (sliding-window) and global (compression, selective) attention across layers, rather than using fixed patterns, enables more effective propagation of long-range dependencies and substantially boosts performance on long-sequence tasks.

The branch specialization is asymmetric. The sliding-window branch is enhanced with MLA, whereas the compression and selective branches adopt GLA. This division reflects the differing structural requirements of the branches: MLA preserves head-specific expressiveness in local attention, while GLA aligns with sparse selective attention’s sharing of key/value blocks across heads. A plausible implication is that the design is intended to prevent redundant execution of all sparse pathways at every layer while preserving a path for both short-range and long-range signal transport.

3. Mathematical structure of latent attention and sparse branches

The ASA-style formulation is grounded in a latent reparameterization of attention. For standard grouped attention, with queries $Q\in\mathbb R^{T\times H\times d_k}$, keys $K\in\mathbb R^{T\times (H/G)\times d_k}$, values $V\in\mathbb R^{T\times (H/G)\times d_v}$, and group size $G$, the reference form is

$$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$

In MLA, each token is projected to a latent state $c_t\in\mathbb R^{d_c}$ with $O(T^2d)$0:

$O(T^2d)$1

and over the prefix,

$O(T^2d)$2

The full-time-step output can then be written as

$O(T^2d)$3

The key point stated in the source is that during training each head has its own $O(T^2d)$4 so expressiveness equals MHA, while during inference only the latent states and projection weights need to be cached (Hu et al., 2 Nov 2025).

GLA introduces grouped sharing over latent-derived keys and values. If $O(T^2d)$5 is the total number of query heads, $O(T^2d)$6 the group size, and $O(T^2d)$7 the number of key/value projections, then for head $O(T^2d)$8:

$O(T^2d)$9

$O(Td)$0

Global layers combine compressed and selective branches. The past is partitioned into blocks of size $O(Td)$1, each block is compressed into a single block-key and block-value,

$O(Td)$2

and top-$O(Td)$3 selective attention chooses $O(Td)$4. With gating scalars $O(Td)$5 and $O(Td)$6,

$O(Td)$7

Local layers instead perform sliding-window attention over the most recent $O(Td)$8 tokens,

$O(Td)$9

This formulation makes the alternation explicit: local layers realize short-range attention directly, while global layers route long-range access through compressed screening plus selective expansion.

4. Latent-space condensation within MLA

In the latent-condensed transformer formulation, LCA is inserted as a drop-in replacement for the standard attention sublayer within a Transformer that already uses MLA. Each token representation $K$0 is down-projected to latent space as

$K$1

with $K$2. During inference, MLA caches only $K$3 and positional keys $K$4, reconstructs full keys and values by up-projections, and performs dense self-attention over all $K$5 positions. LCA inserts a latent-space condensation step before attention computation (You et al., 14 Apr 2026).

The representation is explicitly disentangled into semantic latent vectors and positional keys. Distant history is partitioned into $K$6 groups, while a local window of the most recent $K$7 tokens is retained in full fidelity. If

$K$8

and $K$9, then each group $Q\in\mathbb R^{T\times H\times d_k}$0 produces two representatives: a semantic representative $Q\in\mathbb R^{T\times H\times d_k}$1 and a positional anchor $Q\in\mathbb R^{T\times H\times d_k}$2.

Semantic condensation is query-aware. A summary query is formed from the last $Q\in\mathbb R^{T\times H\times d_k}$3 positions:

$Q\in\mathbb R^{T\times H\times d_k}$4

Each element of a group is scored by

$Q\in\mathbb R^{T\times H\times d_k}$5

and normalized to weights

$Q\in\mathbb R^{T\times H\times d_k}$6

The semantic representative is the weighted convex combination

$Q\in\mathbb R^{T\times H\times d_k}$7

Positional condensation uses anchor selection rather than interpolation. Because the residual positional keys $Q\in\mathbb R^{T\times H\times d_k}$8 are produced by Rotary Position Embedding (RoPE), which is described as a nonlinear function of absolute position, the method selects

$Q\in\mathbb R^{T\times H\times d_k}$9

The full representative key and value are then reconstructed by MLA’s up-projection weights:

$K\in\mathbb R^{T\times (H/G)\times d_k}$0

The final fused context is

$K\in\mathbb R^{T\times (H/G)\times d_k}$1

$K\in\mathbb R^{T\times (H/G)\times d_k}$2

and standard attention is computed over this reduced set. The paper states that this jointly reduces both computational cost and KV cache without adding parameters, and that the design is architecture-agnostic and readily extends to other attention mechanisms such as GQA.

5. Approximation guarantees and computational scaling

The latent-condensed transformer formulation provides an explicit approximation guarantee. Proposition 1 states that, for latent vectors $K\in\mathbb R^{T\times (H/G)\times d_k}$3 and any probability vector $K\in\mathbb R^{T\times (H/G)\times d_k}$4, the expected squared error

$K\in\mathbb R^{T\times (H/G)\times d_k}$5

is uniquely minimized by

$K\in\mathbb R^{T\times (H/G)\times d_k}$6

Theorem 1 then gives a uniform error bound. If $K\in\mathbb R^{T\times (H/G)\times d_k}$7, $K\in\mathbb R^{T\times (H/G)\times d_k}$8, and each distant-history group representative satisfies

$K\in\mathbb R^{T\times (H/G)\times d_k}$9

then

$V\in\mathbb R^{T\times (H/G)\times d_v}$0

The source emphasizes that the bound depends only on $V\in\mathbb R^{T\times (H/G)\times d_v}$1 and not on sequence length $V\in\mathbb R^{T\times (H/G)\times d_v}$2 (You et al., 14 Apr 2026).

The two LCA formulations target complexity from different directions. In the latent-condensed transformer formulation, the complexity summary is as follows:

Method	Compute Cost	KV-Cache Size
Vanilla Self-Attention	$V\in\mathbb R^{T\times (H/G)\times d_v}$3	$V\in\mathbb R^{T\times (H/G)\times d_v}$4
MLA (no condensation)	$V\in\mathbb R^{T\times (H/G)\times d_v}$5	$V\in\mathbb R^{T\times (H/G)\times d_v}$6
MLA + LCA	$V\in\mathbb R^{T\times (H/G)\times d_v}$7	$V\in\mathbb R^{T\times (H/G)\times d_v}$8

If $V\in\mathbb R^{T\times (H/G)\times d_v}$9 is chosen so that $G$0, the compute becomes $G$1 and cache $G$2. The paper states that in practice $G$3 gives $G$4, a significant reduction (You et al., 14 Apr 2026).

In the ASA-style formulation, memory per time step is described as follows: full attention has KV-cache cost $G$5; NSA still caches one full-dimension key/value per head per token and therefore also has $G$6; LCA caches only the latent states $G$7, with $G$8, and in practice $G$9 such as $$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$0 versus $$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$1, so KV-cache cost is approximately 50 % of NSA. For compute, full attention is $$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$2, NSA incurs sliding, compression, and selective costs within each layer, and LCA pays only the local cost in half of the layers and only the global cost in the other half. The same source reports that optimized CUDA kernels achieve a $$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$3 % forward speedup and $$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$4 % backwards speedup, for sequence lengths up to 32K, versus NSA’s default kernel, with negligible quality loss (Hu et al., 2 Nov 2025).

6. Empirical behavior, limitations, and open directions

The ASA-style formulation was evaluated on LLaMA-style models from 340M to 1.3B parameters, trained on 15B and 100B tokens. On common-sense reasoning, the average over LAMBADA, PIQA, HellaSwag, WinoGrande, ARC-easy/challenge, and BoolQ was 43.24 for GQA, 43.80 for NSA, and 44.06 for LCA at 340M parameters; at 1.3B parameters the corresponding scores were 51.45, 52.96, and 53.10. On long-context understanding, averaged over 14 tasks from LongBench, the 340M scores were 10.75 for GQA, 11.02 for NSA, and 12.67 for LCA; at 1.3B they were 16.49, 16.78, and 18.25. On in-context retrieval, at 340M and 8K context on the hardest task S-NIAH-3, GQA scored 33.0 %, NSA 11.6 %, and LCA 52.6 %; at 1.3B and 8K context, GQA scored 64.4 %, NSA 65.0 %, and LCA 62.0 %. The source summarizes these results by stating that LCA matches or exceeds both full-attention and NSA baselines, especially improving retrieval at very long contexts (Hu et al., 2 Nov 2025).

Setting	Baselines	LCA
Common-sense reasoning, 340M	GQA 43.24; NSA 43.80	44.06
Common-sense reasoning, 1.3B	GQA 51.45; NSA 52.96	53.10
LongBench avg, 340M	GQA 10.75; NSA 11.02	12.67
LongBench avg, 1.3B	GQA 16.49; NSA 16.78	18.25

The latent-condensed transformer formulation was evaluated using DeepSeek-V2-Lite (16B, MLA) fine-tuned with LCA for 1,000 steps on 64K SlimPajama, with group size $$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$5 and window $$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$6. At 128K context on H200 GPUs, prefill latency was reduced by 2.5$$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$7 and the KV-cache footprint was cut by 90% from 10 GB to 1 GB. On LongBench-E at 64K context, the MLA baseline average was 29.51 and the LCA average was 29.09, with 1.8$$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$8 speedup. On RULER from 4K to 128K contexts, the 128K score changed from 23.96 for MLA to 24.38 for LCA, with 2.5$$\mathrm{Attn}(q_t,K,V)=\mathbin\|_{h=1}^H \mathrm{Softmax}\!\left(\frac{q_t^h K^{h\top}}{\sqrt{d_k}}\right)V^h.$$9 speedup. On short-context tasks, MMLU changed from 57.12 to 57.04, GSM8K from 41.47 to 41.17, and MBPP from 54.09 to 53.31. In the ablations, weighted pooling plus max-select achieved 29.09 on LongBench-E average, whereas other combinations ranged from 26 to 28; smaller group size $c_t\in\mathbb R^{d_c}$0 improved accuracy at the cost of latency; larger window size $c_t\in\mathbb R^{d_c}$1 yielded slight accuracy gains at higher cost; and among 8, 16, and 32 summary queries, 16 was described as a good tradeoff (You et al., 14 Apr 2026).

Setting	Baseline	LCA
128K context on H200 GPUs	MLA	2.5$c_t\in\mathbb R^{d_c}$2 prefilling speedup; 90% KV cache reduction
LongBench-E, 64K	29.51	29.09
RULER, 128K	23.96	24.38

The limitations stated for the ASA-style formulation are explicit. The 1:1 alternation ratio may not be optimal for all tasks, and adaptive or learned schedules remain to be explored. The sliding-window branch could be replaced by newer linear or Fourier-based local mechanisms. The experiments also use fixed block size $c_t\in\mathbb R^{d_c}$3 and top-$c_t\in\mathbb R^{d_c}$4, and dynamic block sizing or learned compression are identified as possible extensions. In the latent-condensed transformer formulation, the ablations show a direct trade-off between group size, window size, latency, and accuracy. This suggests that LCA should not be interpreted as a single fixed recipe, but rather as a design space in which the condensation granularity, local retention budget, and summary construction materially affect the efficiency-quality frontier.