Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
36 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
37 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
4 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
2000 character limit reached

Survey on Memory-Augmented Neural Networks: Cognitive Insights to AI Applications (2312.06141v2)

Published 11 Dec 2023 in cs.AI

Abstract: This paper explores Memory-Augmented Neural Networks (MANNs), delving into how they blend human-like memory processes into AI. It covers different memory types, like sensory, short-term, and long-term memory, linking psychological theories with AI applications. The study investigates advanced architectures such as Hopfield Networks, Neural Turing Machines, Correlation Matrix Memories, Memformer, and Neural Attention Memory, explaining how they work and where they excel. It dives into real-world uses of MANNs across Natural Language Processing, Computer Vision, Multimodal Learning, and Retrieval Models, showing how memory boosters enhance accuracy, efficiency, and reliability in AI tasks. Overall, this survey provides a comprehensive view of MANNs, offering insights for future research in memory-based AI systems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. J J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554–2558, 1982.
  2. Neural turing machines, 2014.
  3. Memformer: A memory-augmented transformer for sequence modeling, 2022.
  4. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020.
  5. Improving language models by retrieving from trillions of tokens, 2022.
  6. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.
  7. Retrieval augmented classification for long-tail visual recognition, 2022.
  8. Semi-parametric neural image synthesis, 2022.
  9. Murag: Multimodal retrieval-augmented generator for open question answering over images and text, 2022.
  10. Introduction to psychology. BCcampus, 2014.
  11. Human memory: A proposed system and its control processes. In Psychology of learning and motivation, volume 2, pages 89–195. Elsevier, 1968.
  12. Alan Baddeley. Working memory, thought, and action, volume 45. OuP Oxford, 2007.
  13. George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):81, 1956.
  14. Working memory. volume 8 of Psychology of Learning and Motivation, pages 47–89. Academic Press, 1974.
  15. Nelson Cowan. What are the differences between long-term, short-term, and working memory? Progress in brain research, 169:323–338, 2008.
  16. Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
  17. Frank and Benington. Primed to sleep: The dynamics of synaptic plasticity across sleep-wake cycles. Frontiers, 2013.
  18. Offline neuronal activity and synaptic plasticity during sleep and memory consolidation. Neuroscience Research, 2022.
  19. Long short-term memory. 9(8), 1997.
  20. Attention is all you need, 2023.
  21. Teuvo Kohonen. Correlation matrix memories. IEEE Transactions on Computers, C-21(4):353–359, 1972.
  22. Neural attention memory, 2023.
  23. Long short-term memory. Neural Comput., 9(8):1735–1780, nov 1997.
  24. Augmented language models: a survey, 2023.
  25. One question answering model for many languages with cross-lingual dense passage retrieval. Advances in Neural Information Processing Systems, 34:7547–7560, 2021.
  26. Retrieval-augmented multimodal language modeling, 2023.
  27. Realm: Retrieval-augmented language model pre-training, 2020.
  28. Dense passage retrieval for open-domain question answering, 2020.
  29. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, 2019.
  30. Improving retrieval augmented neural machine translation by controlling source and fuzzy-match interactions. In EACL 2023, 2023.
  31. Chain-of-note: Enhancing robustness in retrieval-augmented language models, 2023.
  32. Language models are few-shot learners, 2020.
  33. Jurassic-1: Technical details and evaluation. Technical report, AI21 Labs, August 2021.
  34. Atlas: Few-shot learning with retrieval augmented language models, 2022.
  35. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083, 2023.
  36. Learning customized visual models with retrieval-augmented knowledge, 2023.
  37. Learning transferable visual models from natural language supervision, 2021.
  38. Knn-diffusion: Image generation via large-scale retrieval, 2022.
  39. Retrieval-augmented diffusion models. Advances in Neural Information Processing Systems, 35:15309–15324, 2022.
  40. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  41. Retrieval augmented visual question answering with outside knowledge. arXiv preprint arXiv:2210.03809, 2022.
  42. Cm3: A causal masked multimodal model of the internet, 2022.
  43. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23369–23379, 2023.
  44. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612–17625, 2022.
  45. Task-aware retrieval with instructions, 2022.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now