Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Neuro-mimetic Task-free Unsupervised Online Learning with Continual Self-Organizing Maps (2402.12465v1)

Published 19 Feb 2024 in cs.LG and cs.NE

Abstract: An intelligent system capable of continual learning is one that can process and extract knowledge from potentially infinitely long streams of pattern vectors. The major challenge that makes crafting such a system difficult is known as catastrophic forgetting - an agent, such as one based on artificial neural networks (ANNs), struggles to retain previously acquired knowledge when learning from new samples. Furthermore, ensuring that knowledge is preserved for previous tasks becomes more challenging when input is not supplemented with task boundary information. Although forgetting in the context of ANNs has been studied extensively, there still exists far less work investigating it in terms of unsupervised architectures such as the venerable self-organizing map (SOM), a neural model often used in clustering and dimensionality reduction. While the internal mechanisms of SOMs could, in principle, yield sparse representations that improve memory retention, we observe that, when a fixed-size SOM processes continuous data streams, it experiences concept drift. In light of this, we propose a generalization of the SOM, the continual SOM (CSOM), which is capable of online unsupervised learning under a low memory budget. Our results, on benchmarks including MNIST, Kuzushiji-MNIST, and Fashion-MNIST, show almost a two times increase in accuracy, and CIFAR-10 demonstrates a state-of-the-art result when tested on (online) unsupervised class incremental learning setting.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Kernel-based online machine learning and support vector reduction. In The European Symposium on Artificial Neural Networks (2008).
  2. Task-free continual learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018), 11246–11255.
  3. {EEC}: Learning to encode and regenerate images for continual learning. In International Conference on Learning Representations (2021).
  4. Self-organizing maps as substitutes for k-means clustering. In Computational Science–ICCS 2005: 5th International Conference, Atlanta, GA, USA, May 22-25, 2005, Proceedings, Part III 5 (2005), Springer, pp. 476–483.
  5. Continual learning with self-organizing maps. arXiv preprint arXiv:1904.09330 (2019).
  6. Online learning from data streams with varying feature spaces. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence (2019), AAAI’19/IAAI’19/EAAI’19, AAAI Press.
  7. Application of self-organizing map (som) and k-means clustering algorithms for portraying geochemical anomaly patterns in moalleman district, ne iran. Journal of Geochemical Exploration 233 (2022), 106923.
  8. Class-incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 5 (2023), 5497–5512.
  9. Burton, T. Stability by fixed point theory or liapunov’s theory: A comparison. Fixed point theory 4, 1 (2003), 15–32.
  10. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning 10, 3 (2016), 1–145.
  11. Choe, Y. Hebbian Learning. Springer New York, New York, NY, 2013, pp. 1–5.
  12. Deep learning for classical japanese literature. ArXiv abs/1812.01718 (2018).
  13. Using prior knowledge in a nnpda to learn context-free languages. Advances in neural information processing systems 5 (1992).
  14. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX (Berlin, Heidelberg, 2020), Springer-Verlag, p. 86–102.
  15. Forgy, E. W. Cluster analysis of multivariate data: efficiency versus interpretability of classifications. biometrics 21 (1965), 768–769.
  16. French, R. M. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3, 4 (1999), 128–135.
  17. Fritzke, B. A growing neural gas network learns topologies. In Proceedings of the 7th International Conference on Neural Information Processing Systems (Cambridge, MA, USA, 1994), NIPS’94, MIT Press, p. 625–632.
  18. Biologically inspired incremental learning for high-dimensional spaces. In 2015 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob) (2015), pp. 269–275.
  19. Self-organizing maps and generalization: an algorithmic description of numerosity and variability effects. arXiv preprint arXiv:1802.09442 (2018).
  20. Neural turing machines. arXiv preprint arXiv:1410.5401 (2014).
  21. Gray, R. Vector quantization. IEEE Assp Magazine 1, 2 (1984), 4–29.
  22. Stability and equilibrium states. Communications in Mathematical Physics 38 (1974), 173–193.
  23. Hartono, P. Competitive Learning. Springer US, Boston, MA, 2012, pp. 671–677.
  24. Memory efficient experience replay for streaming learning. In 2019 International Conference on Robotics and Automation (ICRA) (2019), pp. 9769–9776.
  25. The organization of behavior, 1949.
  26. Selective experience replay for lifelong learning. In AAAI (2018).
  27. Inferring algorithmic patterns with stack-augmented recurrent nets. Advances in neural information processing systems 28 (2015).
  28. Improving self-organizing maps with unsupervised feature extraction. In International Conference on Neural Information Processing (2020), Springer, pp. 474–486.
  29. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences 114, 13 (2017), 3521–3526.
  30. Kohonen, T. Self-organized formation of topologically correct feature maps. Biological cybernetics 43, 1 (1982), 59–69.
  31. Kopczyński, D. C.-K. E. Non-euclidean self-organizing maps. arXiv preprint arXiv:2109.11769 (2021).
  32. Convolutional deep belief networks on cifar-10. Unpublished manuscript 40, 7 (2010), 1–9.
  33. Gradient-based learning applied to document recognition. Proc. IEEE 86 (1998), 2278–2324.
  34. Variational data-free knowledge distillation for continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 10 (2023), 12618–12634.
  35. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 12 (2018), 2935–2947.
  36. Edge computing for autonomous driving: Opportunities and challenges. Proceedings of the IEEE 107, 8 (2019), 1697–1716.
  37. Lloyd, S. Least squares quantization in pcm. IEEE transactions on information theory 28, 2 (1982), 129–137.
  38. Gradient episodic memory for continual learning. Advances in neural information processing systems 30 (2017), 6467–6476.
  39. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Red Hook, NY, USA, 2017), NIPS’17, Curran Associates Inc., p. 6470–6479.
  40. Minimally supervised learning using topological projections in self-organizing maps. arXiv preprint arXiv:2401.06923 (2024).
  41. A neural state pushdown automata. IEEE Transactions on Artificial Intelligence 1, 3 (2020), 193–205.
  42. Martinetz, T. Competitive hebbian learning rule forms perfectly topology preserving maps. In International conference on artificial neural networks (1993), Springer, pp. 427–434.
  43. Class-incremental learning: Survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 5 (2023), 5513–5533.
  44. Catastrophic interference in connectionist networks: The sequential learning problem. The psychology of learning and motivation 24, 109 (1989), 92.
  45. The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology 4 (2013), 504.
  46. Continual learning of recurrent neural networks by locally aligning distributed representations. IEEE transactions on neural networks and learning systems 31, 10 (2020), 4267–4278.
  47. Lifelong neural predictive coding: Learning cumulatively online without forgetting. Advances in Neural Information Processing Systems 35 (2022), 5867–5881.
  48. Ororbia, A. G. Continual competitive memory: A neural system for online task-free lifelong learning. arXiv preprint arXiv:2106.13300 (2021).
  49. Ororbia, A. G. Brain-inspired machine intelligence: A survey of neurobiologically-plausible credit assignment. arXiv preprint arXiv:2312.09257 (2023).
  50. Continual learning, fast and slow. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 1 (2024), 134–149.
  51. Dendritic self-organizing maps for continual learning. arXiv preprint arXiv:2110.13611 (2021).
  52. Ratcliff, R. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review 97, 2 (1990), 285.
  53. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), pp. 5533–5542.
  54. Randomized self-organizing map. Neural Computation 33, 8 (2021), 2241–2273.
  55. Parallel distributed processing: Explorations in the microstructure of cognition, Vol. 1: Foundations. MIT press, 1986.
  56. Continual learning with deep generative replay. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Red Hook, NY, USA, 2017), NIPS’17, Curran Associates Inc., p. 2994–3003.
  57. Compete to compute. Advances in neural information processing systems 26 (2013).
  58. A provably stable neural network turing machine with finite precision and time. Information Sciences 658 (2024), 120034.
  59. Lifelong robot learning. Robotics and autonomous systems 15, 1-2 (1995), 25–46.
  60. Scaling the growing neural gas for visual cluster analysis. Big Data Research 26 (2021), 100254.
  61. Clustering of the self-organizing map. IEEE Transactions on neural networks 11, 3 (2000), 586–600.
  62. Distributionally robust memory evolution with generalized divergence for continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 12 (2023), 14337–14352.
  63. Welford, B. Note on a method for calculating corrected sums of squares and products. Technometrics 4, 3 (1962), 419–420.
  64. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019).
  65. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. ArXiv abs/1708.07747 (2017).
  66. Adaptive progressive continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 10 (2022), 6715–6728.
  67. Dynamic support network for few-shot class incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 3 (2023), 2945–2951.
  68. Dynamic self-supervised teacher-student network learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 5 (2023), 5731–5748.
  69. Mitigating forgetting in online continual learning with neuron calibration. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual (2021), M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, Eds., pp. 10260–10272.
  70. Scale: Online self-supervised lifelong learning without prior knowledge. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2022), 2484–2495.
  71. Pycil: a python toolbox for class-incremental learning. SCIENCE CHINA Information Sciences 66, 9 (2023), 197101–.
  72. A model or 603 exemplars: Towards memory-efficient class-incremental learning. In The Eleventh International Conference on Learning Representations (2023).
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets