Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture (2310.03052v3)

Abstract: Making neural networks remember over the long term has been a longstanding issue. Although several external memory techniques have been introduced, most focus on retaining recent information in the short term. Regardless of its importance, information tends to be fatefully forgotten over time. We present Memoria, a memory system for artificial neural networks, drawing inspiration from humans and applying various neuroscientific and psychological theories. The experimental results prove the effectiveness of Memoria in the diverse tasks of sorting, LLMing, and classification, surpassing conventional techniques. Engram analysis reveals that Memoria exhibits the primacy, recency, and temporal contiguity effects which are characteristics of human memory.

• The paper introduces Memoria, a novel memory network that integrates Hebbian theory to overcome Transformers' limitations in handling long input sequences.
• It employs a multi-level memory system—working, short-term, and long-term memory—to selectively store and recall information more effectively.
• Experimental results show enhanced accuracy and efficiency in tasks like language modeling and text classification, bridging cognitive theory and AI.

Memoria: Enhancing Transformers with a Hebbian Memory Architecture

Introduction to Memoria

Recent research introduces Memoria, a novel memory network embedding Hebbian theory principles—a foundational concept in explaining human memory dynamics—into neural network architectures. This advancement targets the inherent limitations of Transformers, particularly their struggle with processing long input sequences due to computational complexity and their inability to selectively remember informative parts of the input. Memoria innovatively integrates a multi-level memory system, categorizing information into working, short-term, and long-term memory, enabling more human-like sequential data processing by neural networks.

Hebbian Theory and Memory Processing

The Hebbian theory, proposing the idea that neurons that fire together wire together, serves as the theoretical backbone of Memoria. This concept translates into a computational model where associations between pieces of information (engrams) strengthen with repeated simultaneous activation. Memoria mirrors this process, adjusting the connection weights between engrams across multiple memory levels, thereby fostering robust and stable memory formation akin to human memory processes.

Enhancements Over Existing Models

Memoria's proposition is grounded in its ability to outperform current Transformer-based methodologies in tasks requiring attention to long-term dependencies, such as sorting, LLMing, and long-text classification. The paper presents compelling evidence where Memoria shows significant improvements over existing models, achieving higher accuracy and efficiency in processing long-sequence data. By adopting Hebbian learning principles, Memoria not only addresses the computational constraints of conventional Transformers but also introduces a more nuanced approach to data retention and recall.

Practical Implications and Theoretical Contributions

The introduction of Memoria brings forth both practical and theoretical advancements. Practically, its integration with popular Transformer models like BERT and GPT extends the applicability of these models to tasks previously constrained by sequence length limitations. Theoretically, Memoria enriches the understanding of applying cognitive theories, such as Hebbian theory, within machine learning frameworks, bridging a gap between artificial intelligence and neural science. This dual contribution potentiates further exploration into memory-augmented neural networks, opening pathways for more sophisticated and human-like AI systems.

Future Perspectives in AI Research

Looking forward, the development of Memoria proposes several avenues for future research. One promising direction is exploring the continuous structure of memory, inspired by the Levels of Processing theory, to develop models that mimic the human memory system even more closely. Additionally, incorporating forgetting mechanisms, such as the interference theory, could enhance the realism and efficiency of these models. As AI continues to evolve, integrating such nuanced human-like processes into machine learning models will be paramount in achieving true artificial general intelligence.

Conclusion

Memoria marks a significant step forward in the quest to imbue neural networks with human-like memory processing capabilities. By leveraging Hebbian theory to implement a hierarchical memory system, this research not only overcomes existing limitations of Transformer models but also paves the way for future advancements in AI. The practical achievements of Memoria, coupled with its theoretical implications, underscore the potential of cognitive principles in enhancing machine learning algorithms, heralding a new era of AI research that closely mirrors human cognition.