Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Fl RDT based ultimate lowering of the negative spherical perceptron capacity (2312.16531v1)

Published 27 Dec 2023 in stat.ML, cond-mat.dis-nn, cs.IT, cs.LG, math.IT, and math.PR

Abstract: We consider the classical \emph{spherical} perceptrons and study their capacities. The famous zero-threshold case was solved in the sixties of the last century (see, \cite{Wendel62,Winder,Cover65}) through the high-dimensional combinatorial considerations. The general threshold, $\kappa$, case though turned out to be much harder and stayed out of reach for the following several decades. A substantial progress was then made in \cite{SchTir02} and \cite{StojnicGardGen13} where the \emph{positive} threshold ($\kappa\geq 0$) scenario was finally fully settled. While the negative counterpart ($\kappa\leq 0$) remained out of reach, \cite{StojnicGardGen13} did show that the random duality theory (RDT) is still powerful enough to provide excellent upper bounds. Moreover, in \cite{StojnicGardSphNeg13}, a \emph{partially lifted} RDT variant was considered and it was shown that the upper bounds of \cite{StojnicGardGen13} can be lowered. After recent breakthroughs in studying bilinearly indexed (bli) random processes in \cite{Stojnicsflgscompyx23,Stojnicnflgscompyx23}, \emph{fully lifted} random duality theory (fl RDT) was developed in \cite{Stojnicflrdt23}. We here first show that the \emph{negative spherical perceptrons} can be fitted into the frame of the fl RDT and then employ the whole fl RDT machinery to characterize the capacity. To be fully practically operational, the fl RDT requires a substantial numerical work. We, however, uncover remarkable closed form analytical relations among key lifting parameters. Such a discovery enables performing the needed numerical calculations to obtain concrete capacity values. We also observe that an excellent convergence (with the relative improvement $\sim 0.1\%$) is achieved already on the third (second non-trivial) level of the \emph{stationarized} full lifting.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Binary perceptron: efficient algorithms can find solutions in a rare well-connected cluster. 2021. available online at http://arxiv.org/abs/2111.03084.
  2. Proof of the contiguity conjecture and lognormal limit for the symmetric perceptron. 2021. available online at http://arxiv.org/abs/2102.13069.
  3. A. E. Alaoui and M. Sellke. Algorithmic pure states for the negative spherical perceptron. 2020. available online at http://arxiv.org/abs/2010.15811.
  4. Discrepancy minimization via a self-balancing walk. In Proc. 53rd STOC, ACM, pages 14–20, 2021.
  5. Storage capacity in symmetric binary perceptrons. J. Phys. A, 52(29):294003, 2019.
  6. Gardner formula for Ising perceptron models at small densities. Proceedings of Thirty Fifth Conference on Learning Theory, PMLR, 178:1787–1911, 2022.
  7. S. Chatterjee. A generalization of the Lindenberg principle. The Annals of Probability, 34(6):2061–2076.
  8. T. Cover. Geomretrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers, (EC-14):326–334, 1965.
  9. J. Ding and N. Sun. Capacity lower bound for the Ising perceptron. STOC 2019: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pages 816–827, 2019.
  10. S. Franz and G. Parisi. The simplest model of jamming. Journal of Physics A: Mathematical and Theoretical, 49(14):145001, 2016.
  11. Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems. SciPost Physics, 2:019, 2017.
  12. Critical jammed phase of the linear perceptron. Phys. Rev. Lett., 123(11):115702, 2019.
  13. Surfing on minima of isostatic landscapes: avalanches and unjamming transition. SciPost Physics, 9:12, 2020.
  14. Algorithms and barriers in the symmetric binary perceptron model. 2022. available online at http://arxiv.org/abs/2203.15667.
  15. E. Gardner. The space of interactions in neural networks models. J. Phys. A: Math. Gen., 21:257–270, 1988.
  16. E. Gardner and B. Derrida. Optimal storage properties of neural networks models. J. Phys. A: Math. Gen., 21:271–284, 1988.
  17. W. Krauth and M. Mezard. Storage capacity of memory networks with binary couplings. J. Phys. France, 50:3057–3066, 1989.
  18. J. W. Lindeberg. Eine neue herleitung des exponentialgesetzes in der wahrscheinlichkeitsrechnung. Math. Z., 15:211–225, 1922.
  19. Tractability from overparametrization: The example of the negative perceptron. 2021. available online at http://arxiv.org/abs/2110.15824.
  20. S. Nakajima and N. Sun. Sharp threshold sequence and universality for Ising perceptron models. Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 638–674, 2023.
  21. W. Perkins and C. Xu. Frozen 1-RSB structure of the symmetric Ising perceptron. STOC 2021: Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 1579–1588, 2021.
  22. M. Shcherbina and B. Tirozzi. On the volume of the intrersection of a sphere with random half spaces. C. R. Acad. Sci. Paris. Ser I, (334):803–806, 2002.
  23. M. Shcherbina and B. Tirozzi. Rigorous solution of the Gardner problem. Comm. on Math. Physics, (234):383–422, 2003.
  24. M. Stojnic. Box constrained ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT optimization in random linear systems – asymptotics. available online at http://arxiv.org/abs/1612.06835.
  25. M. Stojnic. Box constrained ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT optimization in random linear systems – finite dimensions. available online at http://arxiv.org/abs/1612.06839.
  26. M. Stojnic. Various thresholds for ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization in compressed sensing. available online at http://arxiv.org/abs/0907.3666.
  27. M. Stojnic. A simple performance analysis of ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization in compressed sensing. ICASSP, International Conference on Acoustics, Signal and Speech Processing, April 2009.
  28. M. Stojnic. ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT optimization and its various thresholds in compressed sensing. ICASSP, IEEE International Conference on Acoustics, Signal and Speech Processing, pages 3910–3913, 14-19 March 2010. Dallas, TX.
  29. M. Stojnic. Another look at the Gardner problem. 2013. available online at http://arxiv.org/abs/1306.3979.
  30. M. Stojnic. Discrete perceptrons. 2013. available online at http://arxiv.org/abs/1303.4375.
  31. M. Stojnic. Lifting ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization strong and sectional thresholds. 2013. available online at http://arxiv.org/abs/1306.3770.
  32. M. Stojnic. Lifting/lowering Hopfield models ground state energies. 2013. available online at http://arxiv.org/abs/1306.3975.
  33. M. Stojnic. Negative spherical perceptron. 2013. available online at http://arxiv.org/abs/1306.3980.
  34. M. Stojnic. Regularly random duality. 2013. available online at http://arxiv.org/abs/1303.7295.
  35. M. Stojnic. Spherical perceptron as a storage memory with limited errors. 2013. available online at http://arxiv.org/abs/1306.3809.
  36. M. Stojnic. Bilinearly indexed random processes – stationarization of fully lifted interpolation. 2023. available online at arxiv.
  37. M. Stojnic. Binary perceptrons capacity via fully lifted random duality theory. 2023. available online at arxiv.
  38. M. Stojnic. Fully lifted interpolating comparisons of bilinearly indexed random processes. 2023. available online at arxiv.
  39. M. Stojnic. Fully lifted random duality theory. 2023. available online at arxiv.
  40. M. Stojnic. Studying Hopfield models via fully lifted random duality theory. 2023. available online at arxiv.
  41. M. Talagrand. Intersecting random half cubes. Random Structures Algorithms, 15(3-4):436–449, 1999.
  42. M. Talagrand. The Generic Chaining. Springer-Verlag, 2005.
  43. M. Talagrand. Mean field models and spin glasse: Volume II. A series of modern surveys in mathematics 55, Springer-Verlag, Berlin Heidelberg, 2011.
  44. M. Talagrand. Mean field models and spin glasses: Volume I. A series of modern surveys in mathematics 54, Springer-Verlag, Berlin Heidelberg, 2011.
  45. J. G. Wendel. A problem in geometric probability. Mathematica Scandinavica, 1:109–111, 1962.
  46. R. O. Winder. Threshold logic. Ph. D. dissertation, Princetoin University, 1962.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets