Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Fully lifted random duality theory (2312.00070v1)

Published 29 Nov 2023 in math.PR, cs.IT, math.IT, math.OC, math.ST, and stat.TH

Abstract: We study a generic class of \emph{random optimization problems} (rops) and their typical behavior. The foundational aspects of the random duality theory (RDT), associated with rops, were discussed in \cite{StojnicRegRndDlt10}, where it was shown that one can often infer rops' behavior even without actually solving them. Moreover, \cite{StojnicRegRndDlt10} uncovered that various quantities relevant to rops (including, for example, their typical objective values) can be determined (in a large dimensional context) even completely analytically. The key observation was that the \emph{strong deterministic duality} implies the, so-called, \emph{strong random duality} and therefore the full exactness of the analytical RDT characterizations. Here, we attack precisely those scenarios where the strong deterministic duality is not necessarily present and connect them to the recent progress made in studying bilinearly indexed (bli) random processes in \cite{Stojnicnflgscompyx23,Stojnicsflgscompyx23}. In particular, utilizing a fully lifted (fl) interpolating comparison mechanism introduced in \cite{Stojnicnflgscompyx23}, we establish corresponding \emph{fully lifted} RDT (fl RDT). We then rely on a stationarized fl interpolation realization introduced in \cite{Stojnicsflgscompyx23} to obtain complete \emph{statitionarized} fl RDT (sfl RDT). A few well known problems are then discussed as illustrations of a wide range of practical applications implied by the generality of the considered rops.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Phys. Rev. A, 32:1007, 1985.
  2. P. Baldi and S. Venkatesh. Number od stable points for spin-glasses and neural networks of higher orders. Phys. Rev. Letters, 58(9):913–916, Mar. 1987.
  3. The replica symmetric approximation of the analogical neural network. J. Stat. Physics, July 2010.
  4. Equlibrium statistical mechanics of bipartite spin systems. Journal of Physics A: Mathematical and Theoeretical, 44(245002), 2011.
  5. How glassy are neural networks. J. Stat. Mechanics: Thery and Experiment, July 2012.
  6. M. Bayati and A. Montanari. The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Transactions on Information Theory, 57(2), 2011.
  7. A. Ben-Tal and A. Nemirovski. Lectures on Modern Convex Optimization. Analysis, Algorithms, and Engineering Applications. Society for Industrial and Applied Mathematics, 2001.
  8. D. P. Bertsekas. Nonlinear Programming. Athena Scientific, second edition, 1999.
  9. D. Bertsimas and J. N. Tsitsiklis. Introduction to Linear Optimization. Athena Scientific, 1997.
  10. Gardner formula for Ising perceptron models at small densities. Proceedings of Thirty Fifth Conference on Learning Theory, PMLR, 178:1787–1911, 2022.
  11. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2003.
  12. Asymmetric Little spin glas model. Physical Review B, 46(9), September 1992.
  13. Eigenstates and limit cycles in the SK model. Journal of Physics A: Mathematical and General, 21:4201, 1988.
  14. S. H. Cameron. Tech-report 60-600. Proceedings of the bionics symposium, pages 197–212, 1960. Wright air development division, Dayton, Ohio.
  15. S. Chatterjee. A generalization of the Lindenberg principle. The Annals of Probability, 34(6):2061–2076.
  16. T. Cover. Geomretrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers, (EC-14):326–334, 1965.
  17. J. Ding and N. Sun. Capacity lower bound for the Ising perceptron. STOC 2019: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pages 816–827, 2019.
  18. D. Donoho. High-dimensional centrally symmetric polytopes with neighborlines proportional to dimension. Disc. Comput. Geometry, 35(4):617–652, 2006.
  19. Message-passing algorithms for compressed sensing. Proc. National Academy of Sciences, 106(45):18914–18919, Nov. 2009.
  20. Jamming in multilayer supervised learning models. Phys. Rev. Lett., 123(16):160602, 2019.
  21. S. Franz and G. Parisi. The simplest model of jamming. Journal of Physics A: Mathematical and Theoretical, 49(14):145001, 2016.
  22. Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems. SciPost Physics, 2:019, 2017.
  23. Critical jammed phase of the linear perceptron. Phys. Rev. Lett., 123(11):115702, 2019.
  24. Surfing on minima of isostatic landscapes: avalanches and unjamming transition. SciPost Physics, 9:12, 2020.
  25. E. Gardner. The space of interactions in neural networks models. J. Phys. A: Math. Gen., 21:257–270, 1988.
  26. E. Gardner and B. Derrida. Optimal storage properties of neural networks models. J. Phys. A: Math. Gen., 21:271–284, 1988.
  27. H. Gutfreund and Y. Stein. Capacity of neural networks with discrete synaptic couplings. J. Physics A: Math. Gen, 23:2613, 1990.
  28. D. O. Hebb. Organization of behavior. New York: Wiley, 1949.
  29. J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proc. Nat. Acad. Science, 79:2554, 1982.
  30. R. D. Joseph. The number of orthants in n𝑛nitalic_n-space instersected by an s𝑠sitalic_s-dimensional subspace. Tech. memo 8, project PARA, 1960. Cornel aeronautical lab., Buffalo, N.Y.
  31. Covering cubes by random half cubes with applications to biniary neural networks. Journal of Computer and System Sciences, 56:223–252, 1998.
  32. W. Krauth and M. Mezard. Storage capacity of memory networks with binary couplings. J. Phys. France, 50:3057–3066, 1989.
  33. H. W. Kuhn. Nonlinear programming. A historical view. In R. W. Cottle and C. E. Lemke, editors, Nonlinear Programming, volume 9 of SIAM-AMS Proceedings. American Mathematical Society, 1976.
  34. Nonlinear programming. In J. Neyman, editor, Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, pages 481–492. University of California Press, 1951.
  35. J. W. Lindeberg. Eine neue herleitung des exponentialgesetzes in der wahrscheinlichkeitsrechnung. Math. Z., 15:211–225, 1922.
  36. W. A. Little. The existence of persistent states in the brain. Math. Biosci., 19(1-2):101–120, 1974.
  37. D. G. Luenberger. Linear and Nonlinear Programming. Addison-Wesley, second edition, 1984.
  38. S. Nakajima and N. Sun. Sharp threshold sequence and universality for Ising perceptron models. Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 638–674, 2023.
  39. R. O.Winder. Threshold logic. Ph. D. dissertation, Princetoin University, 1962.
  40. Theory of Simple Glasses: Exact Solutions in Infinite Dimensions. Cambridge University Press, 2020.
  41. L. Pastur and A. Figotin. On the theory of disordered spin systems. Theory Math. Phys., 35(403-414), 1978.
  42. The replica-symmetric solution without the replica trick for the Hopfield model. Journal of Statistical Physics, 74(5/6), 1994.
  43. R. T. Rockafellar. Convex Analysis. Princeton University Press, 1970.
  44. L. Schlafli. Gesammelte Mathematische AbhandLungen I. Basel, Switzerland: Verlag Birkhauser, 1950.
  45. M. Shcherbina and B. Tirozzi. The free energy of a class of Hopfield models. Journal of Statistical Physics, 72(1/2), 1993.
  46. M. Shcherbina and B. Tirozzi. On the volume of the intrersection of a sphere with random half spaces. C. R. Acad. Sci. Paris. Ser I, (334):803–806, 2002.
  47. M. Shcherbina and B. Tirozzi. Rigorous solution of the Gardner problem. Comm. on Math. Physics, (234):383–422, 2003.
  48. M. Stojnic. Upper-bounding ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization weak thresholds. available online at http://arxiv.org/abs/1303.7289.
  49. M. Stojnic. Various thresholds for ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization in compressed sensing. available online at http://arxiv.org/abs/0907.3666.
  50. M. Stojnic. A simple performance analysis of ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization in compressed sensing. ICASSP, International Conference on Acoustics, Signal and Speech Processing, April 2009.
  51. M. Stojnic. Block-length dependent thresholds for ℓ2/ℓ1subscriptℓ2subscriptℓ1\ell_{2}/\ell_{1}roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT / roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization in block-sparse compressed sensing. ICASSP, IEEE International Conference on Acoustics, Signal and Speech Processing, pages 3918–3921, 14-19 March 2010. Dallas, TX.
  52. M. Stojnic. ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT optimization and its various thresholds in compressed sensing. ICASSP, IEEE International Conference on Acoustics, Signal and Speech Processing, pages 3910–3913, 14-19 March 2010. Dallas, TX.
  53. M. Stojnic. Another look at the Gardner problem. 2013. available online at http://arxiv.org/abs/1306.3979.
  54. M. Stojnic. Asymmetric Little model and its ground state energies. 2013. available online at http://arxiv.org/abs/1306.3978.
  55. M. Stojnic. Discrete perceptrons. 2013. available online at http://arxiv.org/abs/1303.4375.
  56. M. Stojnic. Lifting ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization strong and sectional thresholds. 2013. available online at http://arxiv.org/abs/1306.3770.
  57. M. Stojnic. Lifting/lowering Hopfield models ground state energies. 2013. available online at http://arxiv.org/abs/1306.3975.
  58. M. Stojnic. Negative spherical perceptron. 2013. available online at http://arxiv.org/abs/1306.3980.
  59. M. Stojnic. Regularly random duality. 2013. available online at http://arxiv.org/abs/1303.7295.
  60. M. Stojnic. Spherical perceptron as a storage memory with limited errors. 2013. available online at http://arxiv.org/abs/1306.3809.
  61. M. Stojnic. Bilinearly indexed random processes – stationarization of fully lifted interpolation. 2023. available online at arxiv.
  62. M. Stojnic. Fully lifted interpolating comparisons of bilinearly indexed random processes. 2023. available online at arxiv.
  63. M. Talagrand. Rigorous results for the Hopfield model with many patterns. Prob. theory and related fields, 110:177–276, 1998.
  64. M. Talagrand. Mean field models and spin glasse: Volume II. A series of modern surveys in mathematics 55, Springer-Verlag, Berlin Heidelberg, 2011.
  65. M. Talagrand. Mean field models and spin glasses: Volume I. A series of modern surveys in mathematics 54, Springer-Verlag, Berlin Heidelberg, 2011.
  66. S. Venkatesh. Epsilon capacity of neural networks. Proc. Conf. on Neural Networks for Computing, Snowbird, UT, 1986.
  67. J. G. Wendel. A problem in geometric probablity. Mathematics Scandinavia, 11:109–111, 1962.
  68. R. O. Winder. Single stage threshold logic. Switching circuit theory and logical design, pages 321–332, Sep. 1961. AIEE Special publications S-134.
  69. C. Xu. Sharp threshold for the Ising perceptron model. Ann. Probab., 43(5):2399–2415, 2021.
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now