Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
121 tokens/sec
GPT-4o
9 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Binary perceptrons capacity via fully lifted random duality theory (2312.00073v1)

Published 29 Nov 2023 in math.PR, cond-mat.dis-nn, cs.IT, math.IT, math.ST, stat.ML, and stat.TH

Abstract: We study the statistical capacity of the classical binary perceptrons with general thresholds $\kappa$. After recognizing the connection between the capacity and the bilinearly indexed (bli) random processes, we utilize a recent progress in studying such processes to characterize the capacity. In particular, we rely on \emph{fully lifted} random duality theory (fl RDT) established in \cite{Stojnicflrdt23} to create a general framework for studying the perceptrons' capacities. Successful underlying numerical evaluations are required for the framework (and ultimately the entire fl RDT machinery) to become fully practically operational. We present results obtained in that directions and uncover that the capacity characterizations are achieved on the second (first non-trivial) level of \emph{stationarized} full lifting. The obtained results \emph{exactly} match the replica symmetry breaking predictions obtained through statistical physics replica methods in \cite{KraMez89}. Most notably, for the famous zero-threshold scenario, $\kappa=0$, we uncover the well known $\alpha\approx0.8330786$ scaled capacity.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Binary perceptron: efficient algorithms can find solutions in a rare well-connected cluster. 2021. available online at http://arxiv.org/abs/2111.03084.
  2. Proof of the contiguity conjecture and lognormal limit for the symmetric perceptron. 2021. available online at http://arxiv.org/abs/2102.13069.
  3. A. E. Alaoui and M. Sellke. Algorithmic pure states for the negative spherical perceptron. 2020. available online at http://arxiv.org/abs/2010.15811.
  4. Discrepancy minimization via a self-balancing walk. In Proc. 53rd STOC, ACM, pages 14–20, 2021.
  5. Storage capacity in symmetric binary perceptrons. J. Phys. A, 52(29):294003, 2019.
  6. P. Baldi and S. Venkatesh. Number od stable points for spin-glasses and neural networks of higher orders. Phys. Rev. Letters, 58(9):913–916, Mar. 1987.
  7. Gardner formula for Ising perceptron models at small densities. Proceedings of Thirty Fifth Conference on Learning Theory, PMLR, 178:1787–1911, 2022.
  8. S. H. Cameron. Tech-report 60-600. Proceedings of the bionics symposium, pages 197–212, 1960. Wright air development division, Dayton, Ohio.
  9. S. Chatterjee. A generalization of the Lindenberg principle. The Annals of Probability, 34(6):2061–2076.
  10. T. Cover. Geomretrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers, (EC-14):326–334, 1965.
  11. J. Ding and N. Sun. Capacity lower bound for the Ising perceptron. STOC 2019: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pages 816–827, 2019.
  12. Jamming in multilayer supervised learning models. Phys. Rev. Lett., 123(16):160602, 2019.
  13. S. Franz and G. Parisi. The simplest model of jamming. Journal of Physics A: Mathematical and Theoretical, 49(14):145001, 2016.
  14. Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems. SciPost Physics, 2:019, 2017.
  15. Critical jammed phase of the linear perceptron. Phys. Rev. Lett., 123(11):115702, 2019.
  16. Surfing on minima of isostatic landscapes: avalanches and unjamming transition. SciPost Physics, 9:12, 2020.
  17. Algorithms and barriers in the symmetric binary perceptron model. 2022. available online at http://arxiv.org/abs/2203.15667.
  18. E. Gardner. The space of interactions in neural networks models. J. Phys. A: Math. Gen., 21:257–270, 1988.
  19. E. Gardner and B. Derrida. Optimal storage properties of neural networks models. J. Phys. A: Math. Gen., 21:271–284, 1988.
  20. H. Gutfreund and Y. Stein. Capacity of neural networks with discrete synaptic couplings. J. Physics A: Math. Gen, 23:2613, 1990.
  21. R. D. Joseph. The number of orthants in n𝑛nitalic_n-space instersected by an s𝑠sitalic_s-dimensional subspace. Tech. memo 8, project PARA, 1960. Cornel aeronautical lab., Buffalo, N.Y.
  22. Covering cubes by random half cubes with applications to biniary neural networks. Journal of Computer and System Sciences, 56:223–252, 1998.
  23. W. Krauth and M. Mezard. Storage capacity of memory networks with binary couplings. J. Phys. France, 50:3057–3066, 1989.
  24. J. W. Lindeberg. Eine neue herleitung des exponentialgesetzes in der wahrscheinlichkeitsrechnung. Math. Z., 15:211–225, 1922.
  25. S. Nakajima and N. Sun. Sharp threshold sequence and universality for Ising perceptron models. Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 638–674, 2023.
  26. R. O.Winder. Threshold logic. Ph. D. dissertation, Princetoin University, 1962.
  27. W. Perkins and C. Xu. Frozen 1-RSB structure of the symmetric Ising perceptron. STOC 2021: Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1579–1588, 2021.
  28. L. Schlafli. Gesammelte Mathematische AbhandLungen I. Basel, Switzerland: Verlag Birkhauser, 1950.
  29. M. Shcherbina and B. Tirozzi. On the volume of the intrersection of a sphere with random half spaces. C. R. Acad. Sci. Paris. Ser I, (334):803–806, 2002.
  30. M. Shcherbina and B. Tirozzi. Rigorous solution of the Gardner problem. Comm. on Math. Physics, (234):383–422, 2003.
  31. M. Stojnic. Box constrained ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT optimization in random linear systems – asymptotics. available online at http://arxiv.org/abs/1612.06835.
  32. M. Stojnic. Box constrained ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT optimization in random linear systems – finite dimensions. available online at http://arxiv.org/abs/1612.06839.
  33. M. Stojnic. Various thresholds for ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization in compressed sensing. available online at http://arxiv.org/abs/0907.3666.
  34. M. Stojnic. A simple performance analysis of ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization in compressed sensing. ICASSP, International Conference on Acoustics, Signal and Speech Processing, April 2009.
  35. M. Stojnic. ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT optimization and its various thresholds in compressed sensing. ICASSP, IEEE International Conference on Acoustics, Signal and Speech Processing, pages 3910–3913, 14-19 March 2010. Dallas, TX.
  36. M. Stojnic. Another look at the Gardner problem. 2013. available online at http://arxiv.org/abs/1306.3979.
  37. M. Stojnic. Discrete perceptrons. 2013. available online at http://arxiv.org/abs/1303.4375.
  38. M. Stojnic. Lifting ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization strong and sectional thresholds. 2013. available online at http://arxiv.org/abs/1306.3770.
  39. M. Stojnic. Lifting/lowering Hopfield models ground state energies. 2013. available online at http://arxiv.org/abs/1306.3975.
  40. M. Stojnic. Negative spherical perceptron. 2013. available online at http://arxiv.org/abs/1306.3980.
  41. M. Stojnic. Regularly random duality. 2013. available online at http://arxiv.org/abs/1303.7295.
  42. M. Stojnic. Spherical perceptron as a storage memory with limited errors. 2013. available online at http://arxiv.org/abs/1306.3809.
  43. M. Stojnic. Bilinearly indexed random processes – stationarization of fully lifted interpolation. 2023. available online at arxiv.
  44. M. Stojnic. Fully lifted interpolating comparisons of bilinearly indexed random processes. 2023. available online at arxiv.
  45. M. Stojnic. Fully lifted random duality theory. 2023. available online at arxiv.
  46. M. Stojnic. Studying Hopfield models via fully lifted random duality theory. 2023. available online at arxiv.
  47. M. Talagrand. Intersecting random half cubes. Random Structures Algorithms, 15(3-4):436–449, 1999.
  48. M. Talagrand. The Generic Chaining. Springer-Verlag, 2005.
  49. M. Talagrand. Mean field models and spin glasse: Volume II. A series of modern surveys in mathematics 55, Springer-Verlag, Berlin Heidelberg, 2011.
  50. M. Talagrand. Mean field models and spin glasses: Volume I. A series of modern surveys in mathematics 54, Springer-Verlag, Berlin Heidelberg, 2011.
  51. S. Venkatesh. Epsilon capacity of neural networks. Proc. Conf. on Neural Networks for Computing, Snowbird, UT, 1986.
  52. J. G. Wendel. A problem in geometric probability. Mathematica Scandinavica, 1:109–111, 1962.
  53. J. G. Wendel. A problem in geometric probablity. Mathematics Scandinavia, 11:109–111, 1962.
  54. R. O. Winder. Single stage threshold logic. Switching circuit theory and logical design, pages 321–332, Sep. 1961. AIEE Special publications S-134.
  55. C. Xu. Sharp threshold for the Ising perceptron model. Ann. Probab., 43(5):2399–2415, 2021.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now