Papers
Topics
Authors
Recent
Search
2000 character limit reached

FedNS: A Fast Sketching Newton-Type Algorithm for Federated Learning

Published 5 Jan 2024 in cs.LG and cs.DC | (2401.02734v1)

Abstract: Recent Newton-type federated learning algorithms have demonstrated linear convergence with respect to the communication rounds. However, communicating Hessian matrices is often unfeasible due to their quadratic communication complexity. In this paper, we introduce a novel approach to tackle this issue while still achieving fast convergence rates. Our proposed method, named as Federated Newton Sketch methods (FedNS), approximates the centralized Newton's method by communicating the sketched square-root Hessian instead of the exact Hessian. To enhance communication efficiency, we reduce the sketch size to match the effective dimension of the Hessian matrix. We provide convergence analysis based on statistical learning for the federated Newton sketch approaches. Specifically, our approaches reach super-linear convergence rates w.r.t. the communication rounds for the first time. We validate the effectiveness of our algorithms through various experiments, which coincide with our theoretical findings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform. In Proceedings of the 38th annual ACM Symposium on Theory of Computing (STOC), 557–563.
  2. Fast randomized kernel ridge regression with statistical guarantees. In Advances in Neural Information Processing Systems 28 (NIPS), 775–783.
  3. Bach, F. 2013. Sharp analysis of low-rank kernel matrix approximations. In Proceedings of the 26th Annual Conference on Learning Theory (COLT), 185–209.
  4. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research (JMLR), 3(Nov): 463–482.
  5. Optimization methods for large-scale machine learning. SIAM Review, 60(2): 223–311.
  6. Convex optimization. Cambridge university press.
  7. Broyden, C. G. 1970. The convergence of a class of double-rank minimization algorithms 1. general considerations. IMA Journal of Applied Mathematics, 6(1): 76–90.
  8. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3): 331–368.
  9. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3): 1–27.
  10. Fast Statistical Leverage Score Approximation in Kernel Ridge Regression. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), 2935–2943. PMLR.
  11. Inexact newton methods. SIAM Journal on Numerical analysis, 19(2): 400–408.
  12. Numerical methods for unconstrained optimization and nonlinear equations, volume 16. SIAM.
  13. Newton-LESS: Sparsification without Trade-offs for the Sketched Newton Update. In Advances in Neural Information Processing Systems 34 (NeurIPS), 2835–2847.
  14. FedNew: A communication-efficient and privacy-preserving Newton-type method for federated learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), 5861–5877. PMLR.
  15. A Newton-type algorithm for federated learning based on incremental Hessian eigenvector sharing. arXiv preprint arXiv:2202.05800.
  16. Distributed newton can communicate less and resist byzantine workers. In Advances in Neural Information Processing Systems 33 (NeurIPS), volume 33, 18028–18038.
  17. Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), 9050–9090. PMLR.
  18. LocalNewton: Reducing communication bottleneck for distributed learning. arXiv preprint arXiv:2105.07320.
  19. The non-iid data quagmire of decentralized machine learning. In Proceedings of the 37th International Conference on Machine Learning (ICML), 4387–4398. PMLR.
  20. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning (ICML), 5132–5143. PMLR.
  21. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492.
  22. Effective dimension adaptive sketching methods for faster regularized least-squares optimization. In Advances in Neural Information Processing Systems 33 (NeurIPS), 19377–19387.
  23. Adaptive Newton sketch: linear-time optimization with quadratic convergence and effective Hessian dimensionality. In Proceedings of the 38th International Conference on Machine Learning (ICML), 5926–5936. PMLR.
  24. Optimal Convergence Rates for Agnostic Nyström Kernel Learning. In Proceedings of the 40th International Conference on Machine Learning (ICML).
  25. Optimal Convergence Rates for Distributed Nyström Approximation. Journal of Machine Learning Research (JMLR), 24: 1–39.
  26. Optimal Rates for Agnostic Distributed Learning. IEEE Transactions on Information Theory (TIT).
  27. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3): 50–60.
  28. Federated Optimization in Heterogeneous Networks. In Proceedings of Machine Learning and Systems 2020 (MLSys).
  29. On the Convergence of FedAvg on Non-IID Data. In Proceedings of the 8th International Conference on Learning Representations (ICLR).
  30. Optimal Convergence for Distributed Learning with Stochastic Gradient Methods and Spectral Algorithms. Journal of Machine Learning Research (JMLR), 21(147): 1–63.
  31. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1): 503–528.
  32. Effective Distributed Learning with Random Features: Improved Bounds and Algorithms. In Proceedings of the 9th International Conference on Learning Representations (ICLR).
  33. Leveraging Low-Rank Relations Between Surrogate Tasks in Structured Prediction. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97, 4193–4202.
  34. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), 1273–1282. PMLR.
  35. Agnostic federated learning. In Proceedings of the 36th International Conference on Machine Learning (ICML), 4615–4625. PMLR.
  36. Numerical optimization. Springer Science & Business Media.
  37. FedSplit: an algorithmic framework for fast federated optimization. In Advances in Neural Information Processing Systems 33 (NeurIPS), volume 33, 7057–7066.
  38. Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence. SIAM Journal on Optimization, 27(1): 205–245.
  39. On fast leverage score sampling and optimal learning. In Advances in Neural Information Processing Systems 31 (NeurIPS), 5672–5682.
  40. Generalization properties of learning with random features. In Advances in Neural Information Processing Systems 30 (NIPS), 3215–3225.
  41. FedNL: Making Newton-Type Methods Applicable to Federated Learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), 18959–19010. PMLR.
  42. Schraudolph, N. N. 2002. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14(7): 1723–1738.
  43. Learning theory estimates via integral operators and their approximations. Constructive Approximation, 26(2): 153–172.
  44. A Non-parametric View of FedAvg and FedProx: Beyond Stationary Points. arXiv preprint arXiv:2106.15216.
  45. Federated learning for non-iid data: From theory to algorithm. In PRICAI 2021: Trends in Artificial Intelligence: 18th Pacific Rim International Conference on Artificial Intelligence (PRICAI), 33–48. Springer.
  46. Non-IID Federated Learning With Sharper Risk Bound. IEEE Transactions on Neural Networks and Learning Systems (TNNLS).
  47. What Do We Mean by Generalization in Federated Learning? In Proceedings of the 10th International Conference on Learning Representations (ICLR).
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up