Relating transformers to models and neural representations of the hippocampal formation (2112.04035v2)

Abstract: Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension.

• The paper demonstrates that Transformers with recurrent positional encodings can simulate hippocampal spatial representations, including place and grid cell activities.
• The paper mathematically links self-attention mechanisms in Transformers with neural firing patterns in hippocampal models, outperforming traditional approaches.
• The paper reveals potential advancements in AI by enhancing spatial cognition and memory retrieval, thus bridging computational neuroscience and machine learning.

Relating Transformers to Hippocampal Models and Neural Representations

The paper "Relating transformers to models and neural representations of the hippocampal formation" presents an intriguing paper into the parallels between Transformer neural networks and hippocampal models from the field of neuroscience, specifically those concerning the hippocampal formation's structure and function. The authors, Whittington et al., propose and demonstrate that Transformers equipped with recurrent positional encodings can replicate spatial representations in the hippocampus, which include neural processes involved in place and grid cell activity.

Key Findings and Claims

1. Transformers and Hippocampal Representations: The authors show that Transformer networks, when adapted with recurrent positional encodings, can simulate the spatial representations usually attributed to the hippocampal formation. This is noteworthy as it establishes a computational connection between artificial neural network architectures and biological models of the brain.
2. Mathematical Relationship: A significant portion of the paper explores the mathematical similarities between the operations of Transformers and those of hippocampal models. There is a discussion on how the patterns of neural firing in the hippocampus, previously modeled through bespoke networks like the Tolman-Eichenbaum Machine (TEM), share computational parallels with self-attention mechanisms in Transformers.
3. Performance Gains: The paper illustrates that Transformers with the discussed modifications outperform traditional hippocampal computational models (e.g., TEM), particularly in tasks requiring spatial cognition and memory retrieval. These tasks demand the internalization and generalization of spatial rules and relations, much like those managed by the biological hippocampal circuits.
4. Path Integration and Position Encoding: The research underscores how recurrent positional encodings in Transformers serve a similar role to path integration in hippocampal models. The adaptable nature of these encodings allows for a flexible representation of different spatial and potentially abstract cognitive tasks.
5. Place Cells and Memory Neurons: Through this computational model, the authors provide a novel perspective on place cells by proposing that memory neurons in the Transformer's architecture mimic the behavior of hippocampal place cells, thus offering insights into the mechanisms of hippocampal memory indexing.

Implications and Future Directions

• Theoretical Insights: By showcasing the operational parallels between Transformers and the neural models of the hippocampus, the research enriches the understanding of both artificial and biological neural networks. It poses significant theoretical implications for computational neuroscience, especially in understanding neural representations and memory schemas.
• Advancements in AI: The findings suggest that leveraging principles from hippocampal computation models in Transformers could lead to significant advancements in AI, particularly in enhancing memory-related tasks and complex cognitive processing, such as spatial navigation, language comprehension, and relational reasoning.
• Broader Cognitive Applications: While the primary focus is spatial cognition, the analogies drawn may extend to other domains of cognition serviced by the cortex and hippocampal systems, stimulating further research into cross-disciplinary applications of AI and cognitive neuroscience.

The paper calls for further investigation into how these neural computations can extend beyond spatial navigation tasks into broader cognitive domains. Future research could explore how transformers can mimic other sophisticated cognitive functions attributed to different neural circuits, potentially bridging gaps in both AI and neuroscience models.