Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators

Abstract: Long chain-of-thought reasoning and agentic tool-calling produce traces spanning tens of thousands of tokens, yet Transformer KV caches grow linearly with sequence length, creating a memory bottleneck on commodity hardware. State-space models offer constant-memory recurrence but suffer a memory cliff: retrieval accuracy collapses once the gap between a stored fact and its query exceeds the effective horizon of the recurrent state. We introduce Echo, a KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA); a drop-in replacement for attention layers that augments SSM blocks with a closed-form dynamical operator whose sufficient statistics are accumulated in constant memory with no KV cache. Echo fits a spectral linear system to the key and value history via kernel ridge regression and retrieves through a learned power-iterated filter, all from $O(r^{2})$ streaming state where $r$ is a small projection rank. On the Multi-Query Associative Recall benchmark, a pure Mamba-2 SSM fails to exceed chance accuracy (${\sim}3\%$) across all gap lengths and KV-pair counts, while at the 50M parameter scale SKA-augmented models achieve $100\%$ retrieval accuracy on every configuration tested, including distractor gaps of $4{,}096$ tokens with $32$ KV pairs. Across five additional transfer benchmarks including needle-in-a-haystack, tool-trace, and multi-hop retrieval, SKA consistently outperforms both pure SSM and SSM+Attention hybrids while maintaining constant inference memory. Ablations confirm that the spectral operator, not the prefix masking strategy, drives the retrieval gain.

• The paper introduces Spectral Koopman Attention (SKA) that recasts associative recall as kernel ridge regression to eliminate the growing KV cache.
• It achieves constant-memory, linear-time sequence processing with significantly improved long-context retrieval over traditional SSM and Transformer hybrids.
• Empirical results show 100% exact recall and robust performance on challenging benchmarks while reducing memory overhead in SSM-based architectures.

Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators

Motivation and Problem Definition

State Space Models (SSMs) such as Mamba and Mamba-2 have established strong efficiency for sequence modeling, offering linear-time inference and constant memory per decoding step. However, they compress the contextual sequence into a fixed-size recurrent state, which induces an exponential decay—termed the "memory cliff"—in the capacity to retrieve information as the gap between fact and query increases (2605.06997). Conversely, the Transformer’s effectiveness in long-range retrieval relies on the linear-cost KV cache, the memory requirement of which diverges with sequence length, precluding practical application in long-context settings (e.g., agentic tool-trace tasks or multi-turn reasoning).

Hybrid architectures that interleave SSM blocks with sparse attention layers (e.g., Nemotron-H, Jamba) reclaim transformer-like recall but do not remove the structural necessity of the growing KV cache. Several attempts at compressing or sparsifying the cache (e.g., token eviction, quantization, dynamic sparsification) ultimately trade off recall robustness. This work posits the possibility of a retrieval mechanism that is both constant-memory (no KV cache growth) and high-fidelity (matching or exceeding softmax attention on associative recall and transfer tasks).

Spectral Koopman Attention: Approach

Echo introduces Spectral Koopman Attention (SKA) as a constant-memory, content-addressed recall module compatible with SSM backbones. The central observation is that associative recall can be precisely recast as kernel ridge regression, where sufficient statistics (key-value and lagged-key covariances) are maintained as fixed-size running sums. This architecture dispenses with materialized attention matrices and grows in neither activation nor cache memory with sequence length.

SKA Architecture: Each SKA module uses independent learned projections (not shared with the SSM backbone) to produce keys, queries, and values. Key and query features are normalized by the per-sequence maximal $\ell_2$ norm, preserving salient signal magnitude and avoiding the amplification of distractor noise from per-token normalization.

Sufficient statistics are accumulated via additive updates:

• $G$: regularized key Gram matrix ($\mathbb{R}^{r \times r}$).
• $M$: lag-one key covariance.
• $C_v$: value–key covariance.

Training and inference regimes handle chunk-causal, prefix, and masked modes. During autoregressive generation, each SKA head stores only $O(r^2 + rP)$ floats (Gram, lag covariance, value-key covariance, last key, max-norm scalar), independent of sequence length.

Operator Estimation: The closed-form Koopman operators are:

• Transition: $A_w = M G^{-1}$ (spectrally whitened via Cholesky, stabilized with ridge regularization).
• Value: $B_v = C_v G^{-1}$ (direct solver).

Power Spectral Filter: To amplify persistent and suppress transient components, the retrieved output applies $K$ power iterations of the whitened Koopman operator. Modes with $|\lambda| \approx 1$ are strongly retained through $G$0, while those with $G$1 decay. This enables SKA to function analogously to "persistent memory," robustly storing content tokens while filtering out noise (or distractor-mass) accumulation.

Spectral Koopman MLP: Feedforward layers substitute SwiGLU with block-diagonal, spectrally-constrained rotations, achieving parameter reduction and norm preservation, guaranteeing robust gradient backpropagation and mitigating depth-wise bottlenecks.

Empirical Evaluation

Synthetic and Transfer Retrieval

On a battery of synthetic tasks targeting SSM failure regimes (zero-shot system prompt recall, tool trace, needle-in-a-haystack, two-hop retrieval, common word identification), SSM+SKA achieves superior mean accuracy across all settings relative to both pure SSM and SSM+attention hybrids. Notably, sequence length generalization is strictly flat: performance persists at up to $G$2 training length (65% at 4,096 tokens, versus SSM's collapse to 2%).

Figure 2: Mamba-3's failure to learn on the System Prompt CoT Retrieval benchmark, compared to SKA-augmented variants’ success.

Figure 4: Performance on a synthetic "Cot Experiment"—highlighting rapid convergence for SKA-augmented models.

MQAR Associative Recall Benchmark

At the 50M parameter scale, pure Mamba-2 fails on MQAR (∼3% chance accuracy on all configurations, regardless of gap length or distractor count). SSM+Attention and SSM+SKA resolve nearly all tasks, but SKA achieves 100% exact recall in every tested configuration—including the hardest regime with 4,096 distractor tokens and 32 key-value pairs. The memory cliff is thus shown as a fundamental limitation of multiplicative SSM recurrence, not a capacity artifact.

Figure 1: SKA maintains strong learning and accuracy even without action masks on the challenging Tool Trace Delayed Commit tasks.

Language Modeling (180M Scale)

Echo-180M (Mamba-2+SKA+KoopmanMLP) matches or outperforms all competing 180M-class SSM and Transformer baselines on five of six standard language understanding benchmarks (HellaSwag, PIQA, ARC, Winogrande, LAMBADA) while using only 10B tokens (~10% of standard training budgets). On LAMBADA (testing multi-sentence dependency), Echo-180M exceeds Mamba-370M, with half the parameters.

Ablation and Analysis

Ablation studies confirm that retrieval improvement derives from the spectral operator itself, not auxiliary mechanisms such as action masking. Removing the action mask, SKA still converges successfully, indicating robustness to distractor token injection—the power filter and spectral selection suffice to recover signal across unstructured noise.

SKA is shown as an explicit, closed-form regression over cached sequence statistics, providing a more stable and interpretable mechanism than the implicit optimization executed by softmax attention (which must be learned indirectly via the nonconvex loss landscape). Notably, fixed-statistic SKA is always linear in the query, whereas softmax attention is not—enabling both provable global convergence (matching the classical ridge solution) and theoretical limits for expressivity.

Practical and Theoretical Implications

• Memory-Efficient Long-Context Processing: Echo enables constant-memory, linear-time recall over unbounded context—removing the bottleneck of KV-cache growth in attention. For agentic applications or multi-modal reasoning pipelines, this unlocks long-horizon computation on commodity hardware.
• Sharp Structural Limits: The memory cliff in SSMs is structural, reflecting dynamics dictated by multiplicative recurrence. Attention hybrids eliminate it at increased cost, but SKA offers a principled alternative capturing persistent subspace content.
• Operator-Based Recall: Grounding retrieval in spectral operators (Koopman or lag-one covariance) opens new avenues for analyzing and extending content-addressed memory in neural sequence models. Explicit covariance updates enable interpretable mechanisms and provable generalization beyond SSMs’ exponential decay.
• Gradient Flow: Echo's backward pass eliminates per-layer rank compression beyond the LM head, in contrast to softmax attention's depthwise effective-rank attrition—potentially enhancing data efficiency, as observed in experiments.
• Limitations & Future Directions: Current validation remains at 180M parameter scale on 10B tokens; extension to billion-parameter and trillion-token settings is required. SKA’s content recall is optimal on synthetic and clean tasks; transfer to more noisy or ambiguous natural language workloads (e.g., RULER, BABILong) remains for future work. The $G$3 Cholesky bottleneck in SKA inference is currently amortized due to small $G$4 but could be further optimized with custom low-precision kernels.

  Figure 6: Demonstration of successful transfer in system prompt recall tasks with SKA-augmented architectures.

Conclusion

Echo establishes that SSMs supplemented with Spectral Koopman Attention can completely eliminate the memory cliff in content-addressed recall, matching or exceeding attention-based architectures at a fraction of the memory cost. Empirical evidence supports strong transfer, data efficiency, and architectural robustness compared to both pure SSM and hybrid SSM-attention baselines. Echo thus motivates a new perspective for long-range sequence modeling, wherein structured operator modules replace explicit key-value caching for persistent content retrieval—suggesting promising directions for scalable and memory-efficient architectural design in agentic and language processing pipelines.

Reference: "Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators" (2605.06997).