2000 character limit reached
An Inexact Preconditioned Zeroth-order Proximal Method for Composite Optimization (2401.03565v1)
Published 7 Jan 2024 in math.OC
Abstract: In this paper, we consider the composite optimization problem, where the objective function integrates a continuously differentiable loss function with a nonsmooth regularization term. Moreover, only the function values for the differentiable part of the objective function are available. To efficiently solve this composite optimization problem, we propose a preconditioned zeroth-order proximal gradient method in which the gradients and preconditioners are estimated by finite-difference schemes based on the function values at the same trial points. We establish the global convergence and worst-case complexity for our proposed method. Numerical experiments exhibit the superiority of our developed method.
- Powell, M.J.: Developments of NEWUOA for minimization without derivatives. IMA Journal of Numerical Analysis 28(4), 649–664 (2008) Conn et al. [2009] Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-free Optimization. SIAM, Philadelphia (2009) Zhang [2012] Zhang, Z.: On derivative-free optimization methods. PhD Thesis, Graduate School of Chinese Academy of Sciences, University of Chinese Academy of Sciences (2012) Grapiglia et al. [2016] Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-free Optimization. SIAM, Philadelphia (2009) Zhang [2012] Zhang, Z.: On derivative-free optimization methods. PhD Thesis, Graduate School of Chinese Academy of Sciences, University of Chinese Academy of Sciences (2012) Grapiglia et al. [2016] Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Zhang, Z.: On derivative-free optimization methods. PhD Thesis, Graduate School of Chinese Academy of Sciences, University of Chinese Academy of Sciences (2012) Grapiglia et al. [2016] Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-free Optimization. SIAM, Philadelphia (2009) Zhang [2012] Zhang, Z.: On derivative-free optimization methods. PhD Thesis, Graduate School of Chinese Academy of Sciences, University of Chinese Academy of Sciences (2012) Grapiglia et al. [2016] Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Zhang, Z.: On derivative-free optimization methods. PhD Thesis, Graduate School of Chinese Academy of Sciences, University of Chinese Academy of Sciences (2012) Grapiglia et al. [2016] Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Zhang, Z.: On derivative-free optimization methods. PhD Thesis, Graduate School of Chinese Academy of Sciences, University of Chinese Academy of Sciences (2012) Grapiglia et al. [2016] Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Grapiglia, G.N., Yuan, J., Yuan, Y.-X.: A derivative-free trust-region algorithm for composite nonsmooth optimization. Computational and Applied Mathematics 35, 475–499 (2016) Larson et al. [2019] Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization methods. Acta Numerica 28, 287–404 (2019) Ciccazzo et al. [2015] Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Ciccazzo, A., Latorre, V., Liuzzi, G., Lucidi, S., Rinaldi, F.: Derivative-free robust optimization for circuit design. Journal of Optimization Theory and Applications 164, 842–861 (2015) Taskar et al. [2005] Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Taskar, B., Chatalbashev, V., Koller, D., Guestrin, C.: Learning structured prediction models: A large margin approach. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 896–903 (2005) Wild et al. [2015] Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Wild, S.M., Sarich, J., Schunck, N.: Derivative-free optimization for parameter estimation in computational nuclear physics. Journal of Physics G: Nuclear and Particle Physics 42(3), 034031 (2015) Shamir [2017] Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Shamir, O.: An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research 18(1), 1703–1713 (2017) Nakamura et al. [2017] Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Nakamura, N., Seepaul, J., Kadane, J.B., Reeja-Jayan, B.: Design for low-temperature microwave-assisted crystallization of ceramic thin films. Applied Stochastic Models in Business and Industry 33(3), 314–321 (2017) Papernot et al. [2017] Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against machine learning. In: Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506–519 (2017) Snoek et al. [2012] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012) Ragonneau and Zhang [2023] Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Ragonneau, T.M., Zhang, Z.: PDFO: A cross-platform package for Powell’s derivative-free optimization solver. arXiv:2302.13246 (2023) Mine and Fukushima [1981] Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. Journal of Optimization Theory and Applications 33, 9–23 (1981) Ghadimi et al. [2016] Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Ghadimi, S., Lan, G., Zhang, H.: Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming 155(1-2), 267–305 (2016) Nesterov and Spokoiny [2017] Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Nesterov, Y., Spokoiny, V.: Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17, 527–566 (2017) Gu et al. [2018] Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Gu, B., Huo, Z., Deng, C., Huang, H.: Faster derivative-free stochastic algorithm for shared memory machines. In: International Conference on Machine Learning, pp. 1812–1821 (2018). PMLR Huang et al. [2019] Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Huang, F., Gu, B., Huo, Z., Chen, S., Huang, H.: Faster gradient-free proximal stochastic methods for nonconvex nonsmooth optimization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1503–1510 (2019) Kalogerias and Powell [2022] Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Kalogerias, D.S., Powell, W.B.: Zeroth-order stochastic compositional algorithms for risk-aware learning. SIAM Journal on Optimization 32(2), 386–416 (2022) Balasubramanian and Ghadimi [2022] Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Balasubramanian, K., Ghadimi, S.: Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points. Foundations of Computational Mathematics, 1–42 (2022) Ghadimi and Lan [2013] Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4), 2341–2368 (2013) Kungurtsev and Rinaldi [2021] Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Kungurtsev, V., Rinaldi, F.: A zeroth order method for stochastic weakly convex optimization. Computational Optimization and Applications 80(3), 731–753 (2021) Pougkakiotis and Kalogerias [2022] Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Pougkakiotis, S., Kalogerias, D.S.: A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. arXiv:2205.01633 (2022) Cai et al. [2022] Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Cai, H., Mckenzie, D., Yin, W., Zhang, Z.: Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization 32(2), 687–714 (2022) Xiao and Zhang [2014] Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Xiao, L., Zhang, T.: A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization 24(4), 2057–2075 (2014) Defazio et al. [2014] Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Defazio, A., Bach, F., Lacoste-Julien, S.: SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems 27 (2014) Doikov and Grapiglia [2023] Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Doikov, N., Grapiglia, G.N.: First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians. arXiv:2309.02412 (2023) Nesterov and Polyak [2006] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Mathematical Programming 108(1), 177–205 (2006) Rockafellar [2015] Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Rockafellar, R.T.: Convex Analysis. Princeton University Press, ??? (2015) Davis and Drusvyatskiy [2019] Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Davis, D., Drusvyatskiy, D.: Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization 29(1), 207–239 (2019) Beck [2017] Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Beck, A.: First-order Methods in Optimization. SIAM, Philadelphia (2017) Xiao et al. [2018] Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Computing 76, 364–389 (2018) Li et al. [2018] Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Li, X., Sun, D., Toh, K.-C.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM Journal on Optimization 28(1), 433–458 (2018) Tibshirani [1996] Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1), 267–288 (1996) Xiao et al. [2021] Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Xiao, N., Liu, X., Yuan, Y.-x.: Exact penalty function for ℓ2,1subscriptℓ21\ell_{2,1}roman_ℓ start_POSTSUBSCRIPT 2 , 1 end_POSTSUBSCRIPT norm minimization over the Stiefel manifold. SIAM Journal on Optimization 31(4), 3097–3126 (2021) Wang et al. [2023] Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023) Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
- Wang, L., Liu, X., Zhang, Y.: A communication-efficient and privacy-aware distributed algorithm for sparse PCA. Computational Optimization and Applications 85(3), 1033–1072 (2023)
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
