Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Training neural networks with structured noise improves classification and generalization (2302.13417v6)

Published 26 Feb 2023 in cond-mat.dis-nn and cs.LG

Abstract: The beneficial role of noise-injection in learning is a consolidated concept in the field of artificial neural networks, suggesting that even biological systems might take advantage of similar mechanisms to optimize their performance. The training-with-noise algorithm proposed by Gardner and collaborators is an emblematic example of a noise-injection procedure in recurrent networks, which can be used to model biological neural systems. We show how adding structure to noisy training data can substantially improve the algorithm performance, allowing the network to approach perfect retrieval of the memories and wide basins of attraction, even in the scenario of maximal injected noise. We also prove that the so-called Hebbian Unlearning rule coincides with the training-with-noise algorithm when noise is maximal and data are stable fixed points of the network dynamics.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. D.J. Amit. Modeling Brain Function. Cambridge University Press, 1989.
  2. Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications, volume 9 of World Scientific Lecture Notes in Physics. World Scientific, 1986.
  3. C. Shorten and T. M. Khoshgoftaar. A survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6(1):1, 2019.
  4. Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness. In Advances in Neural Information Processing Systems, volume 33, pages 14435–14447. Curran Associates, Inc., 2020.
  5. A. Achille and S. Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40(12):2897–2905, 2018.
  6. ‘Unlearning’ has a stabilizing effect in collective memories. Nature, 304(5922):158, 1983.
  7. Training with noise and the storage of correlated patterns in a neural network model. Journal of Physics A: Mathematical and General, 22(12):2019, 1989.
  8. K.Y.M. Wong and D. Sherrington. Neural networks optimally trained with noisy data. Physical Review E, 47(6):4465, 1993.
  9. Storing Infinite Numbers of Patterns in a Spin-Glass Model of Neural Networks. Physical Review Letters, 55(14):1530, 1985.
  10. E. Gardner. Structure of metastable states in the Hopfield model. Journal of Physics A: Mathematical and General, 19(16):L1047, 1986.
  11. K.Y.M. Wong and D. Sherrington. Optimally adapted attractor neural networks in the presence of noise. Journal of Physics A: Mathematical and General, 23(20):4659, 1990.
  12. D.O. Hebb. The Organization of Behavior : A Neuropsychological Theory. John Wiley and Sons, 1949.
  13. E. Gardner. The space of interactions in neural network models. Journal of Physics A: Mathematical and General, 21(1):257, 1988.
  14. The phase space of interactions in neural networks with definite symmetry. Journal of Physics A: Mathematical and General, 22(12):1995, 1989.
  15. M. Minsky and P. Seymour. Perceptrons: an introduction to computational geometry. MIT Press, 1969.
  16. A. Battista and R. Monasson. Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks. Physical Review Letters, 124(4):048302, 2020.
  17. Optimal Information Storage and the Distribution of Synaptic Weights: Perceptron versus Purkinje Cell. Neuron, 43(5):745–757, 2004.
  18. N. Brunel. Is cortical connectivity optimized for storing information? Nature Neuroscience, 19(5):749–755, 2016.
  19. Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks. The Journal of Chemical Physics, 156(10):104107, 2022.
  20. B. Schölkopf and A.J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. The MIT Press, 2018.
  21. J.J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554, 1982.
  22. F. Crick and G. Mitchison. The function of dream sleep. Nature, 304(5922):111, 1983.
  23. Increasing the efficiency of a neural network through unlearning. Physica A: Statistical Mechanics and its Applications, 163(1):386, 1990.
  24. Unlearning and Its Relevance to REM Sleep: Decorrelating Correlated Data. In J.G. Taylor, C.L.T. Mannion, J.G. Taylor, E.R. Caianiello, R.M.J. Cotterill, and J.W. Clark, editors, Neural Network Dynamics, page 30. Springer London, London, 1992.
  25. Unlearning regularization in Boltzmann Machines. arXiv:2311.09418, 2024.
  26. Eigenvector dreaming. Journal of Statistical Mechanics: Theory and Experiment, 2024(1):013302, 2024.
  27. Y. Le Cun. Learning Process in an Asymmetric Threshold Network. In Disordered Systems and Biological Organization, page 233. Springer, Berlin, Heidelberg, 1986.
  28. B.M. Forrest. Content-addressability and learning in neural networks. Journal of Physics A: Mathematical and General, 21(1):245, 1988.
  29. M. Newman and G. Barkema. Monte Carlo Methods in Statistical Physics. Oxford University Press, 1999.
  30. D. Sherrington and S. Kirkpatrick. Solvable Model of a Spin-Glass. Physical Review Letters, 35(26):1792, 1975.
  31. Domains of attraction in neural networks. Journal de Physique, 49(10):1657, 1988.
  32. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929, 2014.
  33. Biologically inspired sleep algorithm for increased generalization and adversarial robustness in deep neural networks. International Conference on Learning Representations, 2019.
  34. D. Saad and S. Solla. Learning with Noise and Regularizers in Multilayer Neural Networks. In Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996.
  35. B. Schottky and U. Krey. Phase transitions in the generalization behaviour of multilayer perceptrons: II. The influence of noise. Journal of Physics A: Mathematical and General, 30(24):8541, 1997.
  36. Electrophysiological markers of memory consolidation in the human brain when memories are reactivated during sleep. Proceedings of the National Academy of Sciences, 119(44):e2123430119, 2022.
  37. Hippocampo-cortical coupling mediates memory consolidation during sleep. Nature Neuroscience, 19(7):959, 2016.
  38. G. Girardeau and V. Lopes-dos Santos. Brain neural patterns and the memory function of sleep. Science, 374(6567):560–564, 2021.
  39. G. Hinton. The Forward-Forward Algorithm: Some Preliminary Investigations. arXiv:2212.13345, 2022.
  40. J. Hinton and T.J. Sejnowski. Unsupervised Learning: Foundations of Neural Computation. The MIT Press, 1999.
  41. How deep convolutional neural networks lose spatial information with training. Machine Learning: Science and Technology, 4(4):045026, 2023.
  42. L. Bonnasse-Gahot and J.P. Nadal. Categorical Perception: A Groundwork for Deep Learning. Neural Computation, 34(2):437, 2022.
  43. S. Diamond and S. Boyd. CVXPY: A Python-Embedded Modeling Language for Convex Optimization. Journal of Machine Learning Research, 17:1, 2016.
  44. Statistical mechanics of Hopfield-like neural networks with modified interactions. Journal of Physics A: Mathematical and General, 24(10):2419–2429, 1991.
  45. Free energy landscapes, dynamics and the edge of chaos in mean-field models of spin glasses. Physical Review B, 74(18):184411, 2006.
  46. Optimization by simulated annealing. Science, 220(4598):671, 1983.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Marco Benedetti (8 papers)
  2. Enrico Ventura (10 papers)
Citations (6)
View on Semantic Scholar