Optimal Decentralized Composite Optimization for Convex Functions (2312.15845v3)
Abstract: In this paper, we focus on the decentralized composite optimization for convex functions. Because of advantages such as robust to the network and no communication bottle-neck in the central server, the decentralized optimization has attracted much research attention in signal processing, control, and optimization communities. Many optimal algorithms have been proposed for the objective function is smooth and (strongly)-convex in the past years. However, it is still an open question whether one can design an optimal algorithm when there is a non-smooth regularization term. In this paper, we fill the gap between smooth decentralized optimization and decentralized composite optimization and propose the first algorithm which can achieve both the optimal computation and communication complexities. Our experiments also validate the effectiveness and efficiency of our algorithm both in computation and communication.
- Decentralized proximal gradient algorithms with linear convergence rates. IEEE Transactions on Automatic Control, 66(6), 2787–2794.
- A linearly convergent proximal gradient algorithm for decentralized optimization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada (pp. 2844–2854).
- Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning, 3(1), 1–122.
- Next: In-network nonconvex optimization. IEEE Transactions on Signal and Information Processing over Networks, 2(2), 120–136.
- Decentralized optimal dispatch of distributed energy resources. In 2012 IEEE 51st IEEE conference on decision and control (CDC) (pp. 3688–3693).: IEEE.
- Accelerating variance-reduced stochastic gradient methods. Mathematical Programming, (pp. 1–45).
- Optimal and practical algorithms for smooth and strongly convex decentralized optimization. Advances in Neural Information Processing Systems, 33, 18342–18352.
- Revisiting extra for smooth distributed optimization. SIAM Journal on Optimization, 30(3), 1795–1821.
- Accelerated gradient tracking over time-varying graphs for decentralized optimization. arXiv preprint arXiv:2104.02596.
- A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates. IEEE Transactions on Signal Processing, 67(17), 4494–4506.
- Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30.
- Accelerated linear iterations for distributed averaging. Annual Reviews in Control, 35(2), 160–165.
- A decentralized second-order method with exact linear convergence rate for consensus optimization. IEEE Trans. Signal Inf. Process. over Networks, 2(4), 507–522.
- Nesterov, Y. (2003). Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media.
- Sarah: A novel method for machine learning problems using stochastic recursive gradient. In International conference on machine learning (pp. 2613–2621).: PMLR.
- Harnessing smoothness to accelerate distributed optimization. IEEE Transactions on Control of Network Systems, 5(3), 1245–1260.
- Accelerated distributed nesterov gradient descent. IEEE Transactions on Automatic Control, 65(6), 2566–2581.
- Sayed, A. H. et al. (2014). Adaptation, learning, and optimization over networks. Foundations and Trends® in Machine Learning, 7(4-5), 311–801.
- Optimal algorithms for smooth and strongly convex distributed optimization in networks. In international conference on machine learning (pp. 3027–3036).: PMLR.
- Extra: An exact first-order algorithm for decentralized consensus optimization. SIAM Journal on Optimization, 25(2), 944–966.
- A proximal gradient algorithm for decentralized composite optimization. IEEE Trans. Signal Process., 63(22), 6013–6023.
- Optimal gradient tracking for decentralized optimization. Mathematical Programming, (pp. 1–53).
- Distributed optimization based on gradient tracking revisited: Enhancing convergence rate via surrogation. SIAM Journal on Optimization, 32(2), 354–385.
- Genome-wide association analysis by lasso penalized logistic regression. Bioinformatics, 25(6), 714–721.
- Distributed algorithms for composite optimization: Unified framework and convergence analysis. IEEE Transactions on Signal Processing, 69, 3555–3570.
- Multi-consensus decentralized accelerated gradient descent. Journal of Machine Learning Research, 24(306), 1–50.
- Exact diffusion for distributed optimization and learning - part I: algorithm development. IEEE Trans. Signal Process., 67(3), 708–723.
- Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2), 301–320.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.