Artificial Hippocampus Networks for Efficient Long-Context Modeling (2510.07318v1)

Abstract: Long-sequence modeling faces a fundamental trade-off between the efficiency of compressive fixed-size memory in RNN-like models and the fidelity of lossless growing memory in attention-based Transformers. Inspired by the Multi-Store Model in cognitive science, we introduce a memory framework of artificial neural networks. Our method maintains a sliding window of the Transformer's KV cache as lossless short-term memory, while a learnable module termed Artificial Hippocampus Network (AHN) recurrently compresses out-of-window information into a fixed-size compact long-term memory. To validate this framework, we instantiate AHNs using modern RNN-like architectures, including Mamba2, DeltaNet, and Gated DeltaNet. Extensive experiments on long-context benchmarks LV-Eval and InfiniteBench demonstrate that AHN-augmented models consistently outperform sliding window baselines and achieve performance comparable or even superior to full-attention models, while substantially reducing computational and memory requirements. For instance, augmenting the Qwen2.5-3B-Instruct with AHNs reduces inference FLOPs by 40.5% and memory cache by 74.0%, while improving its average score on LV-Eval (128k sequence length) from 4.41 to 5.88. Code is available at: https://github.com/ByteDance-Seed/AHN.

• The paper presents Artificial Hippocampus Networks (AHNs) to balance efficiency and fidelity in long-context modeling by merging lossless short-term memory with RNN-like long-term compression.
• It integrates a sliding window over the Transformer's KV cache and a learnable module that compresses out-of-window information, reducing FLOPs by over 40% and memory usage by 74%.
• Experimental results demonstrate improved performance on LV-Eval and LongBench, showcasing robust generalization and practical deployment potential in ultra-long context tasks.

Artificial Hippocampus Networks for Efficient Long-Context Modeling

Introduction

The paper presents Artificial Hippocampus Networks (AHNs), a memory framework designed to address the efficiency-fidelity trade-off in long-context modeling for neural networks. AHNs are inspired by the Multi-Store Model (MSM) from cognitive science, which posits that the human brain maintains a limited-capacity, lossless short-term memory and consolidates it into compressed long-term memory via the hippocampus. In the context of neural architectures, AHNs maintain a sliding window of the Transformer's key-value (KV) cache as lossless short-term memory, while a learnable RNN-like module compresses out-of-window information into a fixed-size long-term memory. This approach enables efficient processing of long sequences by combining the fidelity of attention-based models with the efficiency of RNN-like memory compression.

Figure 1: Illustration of the model augmented with Artificial Hippocampus Networks (AHNs). The model uses attention sinks and a sliding window for lossless memory, and compresses out-of-window tokens into a compact memory representation via AHNs.

Methodology

Memory Framework

The AHN framework operates by maintaining two distinct memory stores:

• Lossless Short-Term Memory: Implemented as a sliding window over the Transformer's KV cache, preserving exact token-level information for recent inputs.
• Compressed Long-Term Memory: Out-of-window KV pairs are recurrently compressed into a fixed-size state using an AHN module, instantiated with modern RNN-like architectures (Mamba2, DeltaNet, GatedDeltaNet).

The integration is formalized as follows: for each token $t > W$ (where $W$ is the window size), the AHN updates its memory state $h_{t-W}$ by processing the KV pair $(k_{t-W}, v_{t-W})$ and the previous memory $h_{t-W-1}$:

$$h_{t-W} = \text{AHN}((k_{t-W}, v_{t-W}), h_{t-W-1})$$

The current query $q_t$ then accesses both the compressed memory $h_{t-W}$ and the lossless windowed KV cache to produce the output:

$$y_t = f(h_{t-W}, \{(k_{i}, v_{i})\}_{i=t-W+1}^{t}, q_t)$$

AHN Instantiation

AHNs are instantiated using efficient linear recurrent architectures. For example, AHN-GDN (GatedDeltaNet) updates its memory via a gated delta rule, and the output is modulated by a gating function and projected linearly. The outputs from AHN and attention are summed to produce the final token representation.

Training Strategy

A parameter-efficient self-distillation scheme is employed: the base LLM's weights are frozen, and only the AHN parameters are trained to mimic the output distribution of the full-attention teacher model via KL divergence. This enables rapid adaptation of AHNs to compress long-range context without retraining the entire model.

Figure 2: AHNs transform lossless memory into fixed-size compressed representations, reducing computational and memory costs while improving long-context performance.

Experimental Results

Efficiency and Performance

AHN-augmented models were evaluated on LV-Eval, InfiniteBench, and LongBench, using Qwen2.5-Instruct (3B, 7B, 14B) as the base. Key findings include:

• Computational Complexity: AHNs reduce FLOPs by over 40% and memory cache by 74% compared to full attention, achieving linear complexity in sequence length.
• Performance: On LV-Eval (128k tokens), AHN-augmented Qwen2.5-3B-Instruct improved average score from 4.41 to 5.88, outperforming both sliding window and compressive transformer baselines.
• Generalization: AHNs maintain strong performance across varying window sizes, demonstrating robust context generalization.

  Figure 3: AHN modules demonstrate strong context generalization capacity on LongBench.

Benchmark Comparisons

On ultra-long-context tasks, AHN-augmented models consistently surpassed sliding window and compressive transformer baselines, and in several cases matched or exceeded full attention performance, despite using a fraction of the memory and compute. On LongBench tasks with average sequence lengths exceeding 8k tokens, AHN variants (Mamba2, DN, GDN) achieved superior accuracy, confirming the effectiveness of recurrent compression.

Ablation Studies

• Training Objective: Self-distillation (KL loss) yielded higher generalization and accuracy than next-token prediction (CE loss), due to denser learning signals.
• Window Randomization: Training with randomized window sizes improved generalization to unseen context lengths.

Gradient Probing

Gradient visualization revealed that AHNs preferentially compress semantically critical tokens (e.g., mathematical symbols, numbers) while discarding less relevant information, validating their targeted compression capability.

Complexity Analysis and Scaling

Integrating AHNs into Transformer models yields constant memory cache size and linear computational complexity per token for long sequences. This enables practical deployment in resource-constrained environments and streaming applications. The additional parameter overhead is minimal (0.2–0.4%), making AHNs suitable for large-scale models.

Figure 4: Complexity analysis of Qwen2.5-3B-Instruct with and without AHNs, showing linear scaling in FLOPs and constant memory usage for AHN-augmented models.

Limitations and Future Directions

While AHNs offer substantial efficiency gains, the fixed-size compressed memory is inherently lossy, which can impair exact recall on certain tasks. Performance is also bounded by the base model's capacity due to the parameter-efficient training regime. Future research may explore hybrid memory management strategies, full-parameter training, and integration with retrieval-augmented mechanisms to further enhance recall and generalization.

Practical Implications

AHNs enable efficient long-context modeling for LLMs, making them suitable for lifelong learning, streaming data processing, and deployment on edge devices. The framework is flexible and can be instantiated with various RNN-like architectures, facilitating further innovation in memory-efficient neural modeling.

Conclusion

Artificial Hippocampus Networks provide a principled and practical solution to the efficiency-fidelity trade-off in long-context neural modeling. By combining lossless short-term memory with learnable compressed long-term memory, AHNs enable Transformer models to process ultra-long sequences with constant resource requirements and competitive performance. This work lays the foundation for future advances in memory-augmented neural architectures and efficient long-context reasoning.