Exact objectives of random linear programs and mean widths of random polyhedrons (2403.03637v1)
Abstract: We consider \emph{random linear programs} (rlps) as a subclass of \emph{random optimization problems} (rops) and study their typical behavior. Our particular focus is on appropriate linear objectives which connect the rlps to the mean widths of random polyhedrons/polytopes. Utilizing the powerful machinery of \emph{random duality theory} (RDT) \cite{StojnicRegRndDlt10}, we obtain, in a large dimensional context, the exact characterizations of the program's objectives. In particular, for any $\alpha=\lim_{n\rightarrow\infty}\frac{m}{n}\in(0,\infty)$, any unit vector $\mathbf{c}\in{\mathbb R}n$, any fixed $\mathbf{a}\in{\mathbb R}n$, and $A\in {\mathbb R}{m\times n}$ with iid standard normal entries, we have \begin{eqnarray*} \lim_{n\rightarrow\infty}{\mathbb P}{A} \left ( (1-\epsilon) \xi{opt}(\alpha;\mathbf{a}) \leq \min_{A\mathbf{x}\leq \mathbf{a}}\mathbf{c}T\mathbf{x} \leq (1+\epsilon) \xi_{opt}(\alpha;\mathbf{a}) \right ) \longrightarrow 1, \end{eqnarray*} where \begin{equation*} \xi_{opt}(\alpha;\mathbf{a}) \triangleq \min_{x>0} \sqrt{x2- x2 \lim_{n\rightarrow\infty} \frac{\sum_{i=1}{m} \left ( \frac{1}{2} \left (\left ( \frac{\mathbf{a}i}{x}\right )2 + 1\right ) \mbox{erfc}\left( \frac{\mathbf{a}_i}{x\sqrt{2}}\right ) - \frac{\mathbf{a}_i}{x} \frac{e{-\frac{\mathbf{a}_i2}{2x2}}}{\sqrt{2\pi}} \right ) }{n} }. \end{equation*} For example, for $\mathbf{a}=\mathbf{1}$, one uncovers \begin{equation*} \xi{opt}(\alpha) = \min_{x>0} \sqrt{x2- x2 \alpha \left ( \frac{1}{2} \left ( \frac{1}{x2} + 1\right ) \mbox{erfc} \left ( \frac{1}{x\sqrt{2}}\right ) - \frac{1}{x} \frac{e{-\frac{1}{2x2}}}{\sqrt{2\pi}} \right ) }. \end{equation*} Moreover, $2 \xi_{opt}(\alpha)$ is precisely the concentrating point of the mean width of the polyhedron ${\mathbf{x}|A\mathbf{x} \leq \mathbf{1}}$.
- S. Agmon. The relaxation method for linear inequalities. Canadian Journal of Mathematics, 6:382–392, 1954.
- D. Alonso-Gutierrez and J. Prochno. On the Gaussian behavior of marginals and the mean width of random polytopes. Proc. Amer. Math. Soc, 143(2):821–832, 2015.
- M. M. Babbar. Distributions of solutions of a set of linear equations (with an application to linear programming). J. Amer. Statist. Assoc., 50(1):854–869, 1955.
- On the optimal objective value of random linear programs. 2023. available online at http://arxiv.org/abs/2401.17530.
- Properties of the geometry of solutions and capacity of multilayer neural networks with rectified linear unit activations. Phys. Rev. Lett., 123:170602, October 2019.
- P. Baldi and S. Venkatesh. Number od stable points for spin-glasses and neural networks of higher orders. Phys. Rev. Letters, 58(9):913–916, Mar. 1987.
- Broken symmetries in multilayered perceptrons. Phys. Rev. A, 45(6):4146, March 1992.
- E. M. Beale. On minimizing a convex function subject to linear inequalities. Journal of the Royal Statistical Society: Series B (Methodological), 17(2):173–184, 1955.
- A. Ben-Tal and A. Nemirovski. Lectures on Modern Convex Optimization. Analysis, Algorithms, and Engineering Applications. Society for Industrial and Applied Mathematics, 2001.
- D. P. Bertsekas. Nonlinear Programming. Athena Scientific, second edition, 1999.
- D. Bertsimas and J. N. Tsitsiklis. Introduction to Linear Optimization. Athena Scientific, 1997.
- Gardner formula for Ising perceptron models at small densities. Proceedings of Thirty Fifth Conference on Learning Theory, PMLR, 178:1787–1911, 2022.
- K.-H. Borgwardt. The average number of pivot steps required by the simplex-method is polynomial. Z. Oper. Res. Ser. A-B, 26(5):A157–A177, 1982.
- K.-H. Borgwardt. A sharp upper bound for the expected number of shadow vertices in lp-polyhedra under orthogonal projection on two-dimensional planes. Math. Oper. Res., 24(3):544–603, 1999.
- S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2003.
- K. Chandrasekaran and S. S. Vempala. Integer feasibility of random polytopes: random integer programs. In Proceedings of the 5th conference on Innovations in theoretical computer science, pages 449–458, 2014.
- S. Chatterjee. A generalization of the Lindenberg principle. The Annals of Probability, 34(6):2061–2076.
- T. Cover. Geomretrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE Transactions on Electronic Computers, (EC-14):326–334, 1965.
- D. Dadush and S. Huiberts. A friendly smoothed analysis of the simplex method. STOC 18-Proceedings of the 50th Annual ACM Symposium on Theory of Computing, pages 390–403, 2018.
- G. B. Dantzig. Linear programming under uncertainty. Management Science, 1:197–206, 1955.
- J. Ding and N. Sun. Capacity lower bound for the Ising perceptron. STOC 2019: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pages 816–827, 2019.
- D. Donoho and J. Tanner. Counting the face of randomly projected hypercubes and orthants, with application. Discrete and Computational Geometry, 43:522–541, 2010.
- J. Dupacova. Stochastic programming with incomplete information: a surrey of results on postoptimization and sensitivity analysis. Optimization, 18(4):507–532, 1987.
- J. Dupacova and R. Wets. Asymptotic behavior of statistical estimators and of optimal solutions of stochastic optimization problems. The Annals of Statistics, pages 1517–1549, 1988.
- Y. C. Eldar and D. Needell. Acceleration of randomized Kaczmarz method via the Johnson-Lindenstrauss lemma. Numer. Algorithms, 58(2):163–177, 2011.
- Storage capacity and learning algorithms for two-layer neural networks. Phys. Rev. A, 45(10):7590, May 1992.
- The allocation of aircraft to routes: An example of linear programming under uncertain demand. Management Science, 3(1):45–73, 1956.
- R. M. Durbin G. J. Mitchison. Bounds on the learning capacity of some multi-layer networks. Biological Cybernetics, 60:345–365, 1989.
- E. Gardner. The space of interactions in neural networks models. J. Phys. A: Math. Gen., 21:257–270, 1988.
- Asymptotic shape of the convex hull of isotropic log-concave random vectors. Adv. in Appl. Math., 75:116–143, 2016.
- E. D. Gluskin. Extremal properties of orthogonal parallelepipeds and their applications to the geometry of banach spaces. Mat. Sb. (N.S.), 136(178)(1):85–96, 1988.
- Y. Gordon. On Milman’s inequality and random subspaces which escape through a mesh in Rnsuperscript𝑅𝑛{R}^{n}italic_R start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT. Geometric Aspect of of functional analysis, Isr. Semin. 1986-87, Lect. Notes Math, 1317, 1988.
- H. Gutfreund and Y. Stein. Capacity of neural networks with discrete synaptic couplings. J. Physics A: Math. Gen, 23:2613, 1990.
- Upper and lower bounds on the smoothed complexity of the simplex method. STOC 23-Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 1904–1917, 2023.
- S Kaczmarz. Angenaherte auflosung von systemen linearer gleichungen. Bulletin international de lacademie polonaise des sciences et des lettres, 1937.
- P. Kall and J. Mayer. Stochastic linear programming, volume 7. Springer, 1979.
- Limit laws for empirical optimal solutions in random linear programs. Annals of Operations Research, 315(1):251–278, 2022.
- W. Krauth and M. Mezard. Storage capacity of memory networks with binary couplings. J. Phys. France, 50:3057–3066, 1989.
- H. W. Kuhn. Nonlinear programming. A historical view. In R. W. Cottle and C. E. Lemke, editors, Nonlinear Programming, volume 9 of SIAM-AMS Proceedings. American Mathematical Society, 1976.
- Nonlinear programming. In J. Neyman, editor, Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, pages 481–492. University of California Press, 1951.
- J. W. Lindeberg. Eine neue herleitung des exponentialgesetzes in der wahrscheinlichkeitsrechnung. Math. Z., 15:211–225, 1922.
- Smallest singular value of random matrices and geometry of random polytopes. Adv. Math., 195(2):491–523, 2005.
- Asymptotic confidence sets for random linear programs. 2023. available online at http://arxiv.org/abs/2302.12364.
- Y. Liu and C. Q. Gu. On greedy randomized block Kaczmarz method for consistent linear systems. Linear Algebra Appl., 616:178–200, 2021.
- D. G. Luenberger. Linear and Nonlinear Programming. Addison-Wesley, second edition, 1984.
- Sampling Kaczmarz-Motzkin method for linear feasibility problems: generalization and acceleration. Math. Program., 194(1-2):719–779, 2022.
- The relaxation method for linear inequalities. Canadian Journal of Mathematics, 6:393–404, 1954.
- S. Nakajima and N. Sun. Sharp threshold sequence and universality for Ising perceptron models. Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 638–674, 2023.
- D. Needell and J. A. Tropp. Paved with good intentions: analysis of a randomized block Kaczmarz method. Linear Algebra Appl., 441:199–221, 2014.
- A. Prekopa. On the probability distribution of the optimum of a random linear program. SIAM Journal on Control, 4(1):211–222, 1966.
- R. T. Rockafellar. Convex Analysis. Princeton University Press, 1970.
- A. Ruszczynski and A. Shapiro. Stochastic programming models. Handbooks in Operations Research and Management Science, pages 841–858, 1989.
- A. Shapiro. Asymptotic properties of statistical estimators in stochastic programming. The Annals of Statistics, pages 841–858, 1989.
- A. Shapiro. Asymptotic behavior of optimal solutions in stochastic programming. Mathematics of Operations Research, 18(4):829–845, 1993.
- M. Shcherbina and B. Tirozzi. Rigorous solution of the Gardner problem. Comm. on Math. Physics, (234):383–422, 2003.
- S. Smale. On the average number of steps of the simplex method of linear programming. Math. Programming, 27(3):241–262, 1983.
- Smoothed analysis of algorithms: why the simplex algorithm usually takes polynomial time. J. ACM, 51(3):385–468, 2004.
- M. Stojnic. Various thresholds for ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-optimization in compressed sensing. available online at http://arxiv.org/abs/0907.3666.
- M. Stojnic. ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT optimization and its various thresholds in compressed sensing. ICASSP, IEEE International Conference on Acoustics, Signal and Speech Processing, pages 3910–3913, 14-19 March 2010. Dallas, TX.
- M. Stojnic. Recovery thresholds for ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT optimization in binary compressed sensing. ISIT, IEEE International Symposium on Information Theory, pages 1593 – 1597, 13-18 June 2010. Austin, TX.
- M. Stojnic. Another look at the Gardner problem. 2013. available online at http://arxiv.org/abs/1306.3979.
- M. Stojnic. Discrete perceptrons. 2013. available online at http://arxiv.org/abs/1303.4375.
- M. Stojnic. Meshes that trap random subspaces. 2013. available online at http://arxiv.org/abs/1304.0003.
- M. Stojnic. Regularly random duality. 2013. available online at http://arxiv.org/abs/1303.7295.
- M. Stojnic. Generic and lifted probabilistic comparisons – max replaces minmax. 2016. available online at http://arxiv.org/abs/1612.08506.
- M. Stojnic. Bilinearly indexed random processes – stationarization of fully lifted interpolation. 2023. available online at http://arxiv.org/abs/2311.18097.
- M. Stojnic. Binary perceptrons capacity via fully lifted random duality theory. 2023. available online at http://arxiv.org/abs/2312.00073.
- M. Stojnic. Fl RDT based ultimate lowering of the negative spherical perceptron capacity. 2023. available online at http://arxiv.org/abs/2312.16531.
- M. Stojnic. Exact capacity of the wide hidden layer treelike neural networks with generic activations. 2024. available online at http://arxiv.org/abs/2402.05719.
- T. Strohmer and R. Vershynin. A randomized kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl., 15(2):262–278, 2004.
- M. Talagrand. Intersecting random half cubes. Random Structures Algorithms, 15(3-4):436–449, 1999.
- M. Talagrand. Mean field models and spin glasse: Volume II. A series of modern surveys in mathematics 55, Springer-Verlag, Berlin Heidelberg, 2011.
- M. Talagrand. Mean field models and spin glasses: Volume I. A series of modern surveys in mathematics 54, Springer-Verlag, Berlin Heidelberg, 2011.
- G. Tintner. A note on stochastic linear programming). Econometrica, 28(1):490–495, 1960.
- J. G. Wendel. A problem in geometric probability. Mathematica Scandinavica, 1:109–111, 1962.
- R. O. Winder. Single stage threshold logic. Switching circuit theory and logical design, pages 321–332, Sep. 1961. AIEE Special publications S-134.
- J. A. Zavatone-Veth and C. Pehlevan. Activation function dependence of the storage capacity of treelike neural networks. Phys. Rev. E, 103:L020301, February 2021.