Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Online estimation of the inverse of the Hessian for stochastic optimization with application to universal stochastic Newton algorithms (2401.10923v3)

Published 15 Jan 2024 in math.OC and stat.ML

Abstract: This paper addresses second-order stochastic optimization for estimating the minimizer of a convex function written as an expectation. A direct recursive estimation technique for the inverse Hessian matrix using a Robbins-Monro procedure is introduced. This approach enables to drastically reduces computational complexity. Above all, it allows to develop universal stochastic Newton methods and investigate the asymptotic efficiency of the proposed approach. This work so expands the application scope of secondorder algorithms in stochastic optimization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. Bach, F. (2014). Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression. The Journal of Machine Learning Research, 15(1):595–627.
  2. A stochastic gauss–newton algorithm for regularized semi-discrete optimal transport. Information and Inference: A Journal of the IMA, 12(1):390–447.
  3. Stochastic approximation algorithms for superquantiles estimation. arXiv preprint arXiv:2007.14659.
  4. An efficient stochastic newton algorithm for parameter estimation in logistic regressions. SIAM Journal on Control and Optimization, 58(1):348–367.
  5. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and electronics in agriculture, 24(3):131–151.
  6. On the asymptotic rate of convergence of stochastic newton algorithms and their weighted averaged versions. Computational Optimization and Applications, pages 1–52.
  7. Online estimation of the geometric median in hilbert spaces: Nonasymptotic confidence balls. The Annals of Statistics, 45(2):591–614.
  8. Efficient and fast estimation of the geometric median in Hilbert spaces with an averaged stochastic gradient algorithm. Bernoulli, 19(1):18–43.
  9. Fast estimation of the median covariation matrix with application to online robust principal components analysis. Test, 26(3):461–480.
  10. An efficient averaged stochastic gauss-newton algorithm for estimating parameters of non linear regressions models. arXiv preprint arXiv:2006.12920.
  11. On projected stochastic gradient descent algorithm with weighted averaging for least squares regression. IEEE Transactions on Automatic Control, 62(11):5974–5981.
  12. Non asymptotic controls on a recursive superquantile approximation.
  13. Duflo, M. (1990). Méthodes récursives aléatoires. (No Title).
  14. Fréchet, M. (1948). Les éléments aléatoires de nature quelconque dans un espace distancié. In Annales de l’institut Henri Poincaré, volume 10, pages 215–310.
  15. Optimal non-asymptotic bound of the ruppert-polyak averaging without strong convexity. arXiv preprint arXiv:1709.03342.
  16. Godichon-Baggioni, A. (2019). Online estimation of the asymptotic variance for averaged stochastic gradient algorithms. Journal of Statistical Planning and Inference, 203:1–19.
  17. Godichon-Baggioni, A. (2021). Convergence in quadratic mean of averaged stochastic gradient algorithms without strong convexity nor bounded gradient. arXiv preprint arXiv:2107.12058.
  18. Online stochastic newton methods for estimating the geometric median and applications. arXiv preprint arXiv:2304.00770.
  19. An averaged projected robbins-monro algorithm for estimating the parameters of a truncated spherical distribution.
  20. Recursive ridge regression using second-order stochastic algorithms.
  21. Non-asymptotic analysis of stochastic approximation algorithms for streaming data. ESAIM: Probability and Statistics, 27:482–514.
  22. Asymptotic optimality of conditioned stochastic gradient descent. arXiv preprint arXiv:2006.02745.
  23. A generalization of the averaging procedure: The use of two-time-scale algorithms. SIAM Journal on Control and Optimization, 49(4):1523–1543.
  24. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in neural information processing systems, 24.
  25. Pelletier, M. (1998). On the almost sure asymptotic behaviour of stochastic algorithms. Stochastic processes and their applications, 78(2):217–244.
  26. Pelletier, M. (2000). Asymptotic almost sure efficiency of averaged stochastic algorithms. SIAM Journal on Control and Optimization, 39(1):49–72.
  27. Pinelis, I. (1994). Optimum bounds for the distributions of martingales in banach spaces. The Annals of Probability, pages 1679–1706.
  28. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855.
  29. A stochastic approximation method. The annals of mathematical statistics, pages 400–407.
  30. A convergence theorem for non negative almost supermartingales and some applications. In Optimizing methods in statistics, pages 233–257. Elsevier.
  31. Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162:83–112.
  32. Towards stability and optimality in stochastic gradient descent. In Artificial Intelligence and Statistics, pages 1290–1298. PMLR.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now