Hippocampus-DETR: An Explicit Memory Object Detection Framework Based on Hippocampus Modeling

Abstract: This paper addresses the lack of explicit memory mechanisms in current object detection models and proposes Hippocampus-DETR, a novel detection framework based on biological hippocampal memory modeling. This framework integrates a hippocampal memory network module, HipNet, into the DETR architecture and systematically simulates the anatomical structure and functional organization of hippocampal subregions, including the entorhinal cortex, dentate gyrus, CA3, CA1, and subiculum. Through this design, Hippocampus-DETR realizes pattern separation, pattern completion, importance filtering, and information integration of visual encoding features. During training, different memory submodules are optimized using a layer-wise training strategy, ultimately forming a memory system with memory retrieval and completion capabilities. Experimental results demonstrate that Hippocampus-DETR achieves higher detection accuracy than current mainstream models. More importantly, models equipped with this framework also exhibit excellent generalization ability and data efficiency in tasks such as few-shot image classification, multimodal feature construction, and image restoration. Subsequent experiments further validate the functional necessity and internal interpretability of each memory submodule. This study not only provides a novel object detection framework, but also offers a feasible technical pathway for integrating neurocognitive mechanisms with deep learning models, highlighting its significant value in improving model learning efficiency and task robustness. The project is available at https://github.com/2186cloud/hipnet.

• The paper introduces HipNet, a hippocampus-inspired memory module that significantly improves object detection under occlusion and few-shot conditions.
• It employs biologically faithful models of hippocampal subregions to achieve effective pattern separation, completion, and importance filtering, as shown by controlled ablation studies.
• The framework also supports multimodal learning and image restoration, offering a versatile approach for robust and interpretable AI systems.

Hippocampus-DETR: A Hippocampus-Inspired Memory Mechanism for Object Detection

Motivation and Background

Traditional object detection frameworks, including DETR and its derivatives, have made substantial progress in efficiency and accuracy through architectural and training innovations. Despite these advances, these models lack an explicit memory mechanism comparable with neurocognitive processes in the human brain, specifically the hippocampal system. Such a deficit becomes especially limiting in challenging scenarios, such as severe occlusion or few-shot learning, where the ability to recall prior visual states or leverage previous experience is critical. The integration of memory in detection architectures, while explored via various ad hoc memory modules, has largely neglected the anatomical and functional depth of hippocampal modeling.

Hippocampal Modeling in HipNet

Hippocampus-DETR introduces HipNet, a memory module explicitly modeled after hippocampal subregions and their connections: entorhinal cortex (EC), dentate gyrus (DG), CA3, CA1, and the subiculum. Anatomical fidelity is prioritized; EC is subdivided into EC2, EC3, and EC5, with projection pathways mirroring biological transmission. The model leverages the following functional assignments:

• EC (Sensory Relay): Receives inputs from the perceptual integration component (RT-DETR tokens) and normalizes/removes empty features (EC2, EC3), then integrates memory and sensory features (EC5).
• DG (Pattern Separation): Implements a SOM network to produce sparse, orthogonal feature representations, laying the groundwork for efficient pattern completion downstream.
• CA3 (Pattern Completion): Modeled as a modern Hopfield network, CA3 retrieves complete visual and label features from cues generated by DG and EC2—addressing the need for partial-to-whole recall.
• CA1 (Importance Filtering): Uses a CNN to compare current sensory and recalled memory features, identifying features important for task discrimination via channel-wise cosine similarity.
• Subiculum (Information Integration): Employs a memory attention mechanism; features are selectively output as either memory or sensory information depending on their congruence, simulating memory distortion and contextual adaptation.

Modules are trained in a layer-wise fashion; DG and CA3 are trained with single-shot class prototypes, CA1 with remainder samples, and other components with the full set. Memory assistance is conditionally activated in inference—only when RT-DETR's output is ambiguous and HipNet's is confident.

Numerical Performance and Comparative Analysis

On the MS COCO dataset, Hippocampus-DETR shows improved detection accuracy, outperforming both YOLO and DETR variants with comparable parameter and computation footprints, even in regimes where memory effects are diluted by large-scale training data. Notably, HipNet is modular and transferable, capable of augmenting other object detection backbones.

In few-shot image classification (FashionMNIST, MNIST, CIFAR-10), Hippocampus-DETR variants (e.g., ResNet18-HipNet) consistently achieve superior accuracy compared to their conventional counterparts. HipNet is naturally suited for these tasks given its architecture: class prototypes are memorized without reliance on base set pretraining. The model is further robust under noisy/few-shot conditions, suggesting deeper associative links between memory features and present inputs, beyond simple similarity matching.

Extensions to Multimodal Learning and Image Restoration

HipNet supports semantic and visual feature association, facilitating interpretable word vector construction and text-to-image tasks. By activating memory imagery through cue similarity, the model infers new category word vectors aligned with image semantics and can generate images via a two-stage approach: semantic feature generation (CA3) followed by deconvolution. This decoupled scheme enhances flexibility and interpretability.

For image restoration (completion, denoising, super-resolution), HipNet leverages its memory module to recall missing or clean representations, eschewing the need for bespoke restoration architectures. Memory features dominate outputs under occlusion or blur, with gating logic modifiable for task specificity.

Ablation and Interpretability

Ablation studies on few-shot classification emphasize the necessity of DG, CA3, and CA1:

• Removing DG impairs pattern separation, degrading the discriminability of CA3 outputs—consistent with biological findings on spatial pattern separation deficits.
• Removing CA3 eliminates memory, reducing accuracy especially in stringent few-shot settings.
• CA1 ablation diminishes the model’s ability to identify discriminative memory features, leading to learning impairments akin to hippocampal lesion evidence.

Feature extractors employing regional convolution (vs. global) produce more orthogonal channels, facilitating effective downstream pattern separation.

Interpretability is a highlight: winning neuron positions in DG correlate with category similarity, and CA3 outputs exhibit confusion mainly at the subordinate-category level. Top-1 and Top-2 accuracy measurements confirm holistic memory recall capability at broader category levels.

Implications and Future Directions

The integration of explicit, neuroanatomically informed memory mechanisms addresses fundamental limitations in current object detection and classification frameworks. Hippocampus-DETR demonstrates superior robustness, few-shot efficacy, interpretability, and multimodal generalization by simulating key hippocampal processes in deep neural models. The approach serves as a technical pathway for harmonizing neural computation with cognitive neuroscience, opening prospects for biologically plausible, adaptive AI systems.

Potential avenues for future research include:

• Extending Hippocampus-DETR to lifelong learning and continual adaptation scenarios, leveraging its modular explicit memory.
• Scaling anatomical modeling to additional brain regions for other cognitive tasks (e.g., reasoning, planning).
• Exploring neuromorphic implementations for greater energy efficiency and architectural scalability.

Conclusion

Hippocampus-DETR establishes an explicit, biologically motivated memory mechanism in object detection architectures, achieving state-of-the-art performance on standard and few-shot visual benchmarks, as well as demonstrable gains in interpretability, multimodal adaptation, and image restoration tasks. The functional necessity and internal structure of each hippocampal submodule are validated through controlled ablation, underscoring the practical and theoretical value of hippocampal modeling for robust, efficient, and cognitively inspired artificial intelligence (2606.27831).