Papers
Topics
Authors
Recent
AI Research Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 80 tok/s
Gemini 2.5 Pro 49 tok/s Pro
GPT-5 Medium 26 tok/s Pro
GPT-5 High 32 tok/s Pro
GPT-4o 92 tok/s Pro
Kimi K2 182 tok/s Pro
GPT OSS 120B 438 tok/s Pro
Claude Sonnet 4 38 tok/s Pro
2000 character limit reached

SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learning (2402.04114v2)

Published 6 Feb 2024 in stat.ML, cs.LG, and math.OC

Abstract: In this paper, we analyze the sample and communication complexity of the federated linear stochastic approximation (FedLSA) algorithm. We explicitly quantify the effects of local training with agent heterogeneity. We show that the communication complexity of FedLSA scales polynomially with the inverse of the desired accuracy $\epsilon$. To overcome this, we propose SCAFFLSA a new variant of FedLSA that uses control variates to correct for client drift, and establish its sample and communication complexities. We show that for statistically heterogeneous agents, its communication complexity scales logarithmically with the desired accuracy, similar to Scaffnew. An important finding is that, compared to the existing results for Scaffnew, the sample complexity scales with the inverse of the number of agents, a property referred to as linear speed-up. Achieving this linear speed-up requires completely new theoretical arguments. We apply the proposed method to federated temporal difference learning with linear function approximation and analyze the corresponding complexity improvements.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. On a perturbation approach for the analysis of stochastic tracking algorithms. SIAM Journal on Control and Optimization, 39(3):872–899, 2000.
  2. On the generation of markov decision processes. Journal of the Operational Research Society, 46(3):354–361, 1995.
  3. A finite time analysis of temporal difference learning with linear function approximation. In Conference On Learning Theory, pp.  1691–1692, 2018.
  4. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pp. 2089–2099. PMLR, 2021.
  5. Randprox: Primal-dual optimization algorithms with randomized proximal updates. arXiv preprint arXiv:2207.12891, 2022.
  6. Provably doubly accelerated federated learning: The first theoretically successful combination of local training and compressed communication. arXiv preprint arXiv:2210.13277, 2022.
  7. Federated td learning over finite-rate erasure channels: Linear speedup under markovian sampling. IEEE Control Systems Letters, 2023.
  8. Finite sample analyses for TD(0) with function approximation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
  9. Policy evaluation with temporal differences: A survey and comparison. Journal of Machine Learning Research, 15:809–883, 2014.
  10. Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation on Multi-Agent Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning, pp.  1626–1635. PMLR, May 2019. URL https://proceedings.mlr.press/v97/doan19a.html. ISSN: 2640-3498.
  11. Doan, T. T. Local stochastic approximation: A unified view of federated learning and distributed multi-task reinforcement learning algorithms. arXiv preprint arXiv:2006.13460, 2020.
  12. Tight high probability bounds for linear stochastic approximation with fixed stepsize. In Ranzato, M., Beygelzimer, A., Nguyen, K., Liang, P. S., Vaughan, J. W., and Dauphin, Y. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  30063–30074. Curran Associates, Inc., 2021.
  13. Finite-time high-probability bounds for polyak-ruppert averaged iterates of linear stochastic approximation. arXiv preprint arXiv:2207.04475, 2022.
  14. Off-policy learning with eligibility traces: a survey. J. Mach. Learn. Res., 15(1):289–333, 2014.
  15. Local sgd: Unified theory and new efficient methods. In International Conference on Artificial Intelligence and Statistics, pp.  3556–3564. PMLR, 2021.
  16. Can 5th generation local training methods support client sampling? yes! In International Conference on Artificial Intelligence and Statistics, pp.  1055–1092. PMLR, 2023.
  17. Exponential stability of general tracking algorithms. IEEE Transactions on Automatic Control, 40(8):1376–1387, 1995.
  18. On the Convergence of Local Descent Methods in Federated Learning, December 2019. URL http://arxiv.org/abs/1910.14425. arXiv:1910.14425 [cs, stat].
  19. Tighter analysis for proxskip. In International Conference on Machine Learning, pp. 13469–13496. PMLR, 2023.
  20. Federated Reinforcement Learning with Environment Heterogeneity. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pp.  18–37. PMLR, May 2022. URL https://proceedings.mlr.press/v151/jin22a.html. ISSN: 2640-3498.
  21. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp. 5132–5143. PMLR, 2020.
  22. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pp.  4519–4529. PMLR, 2020.
  23. Federated reinforcement learning: Linear speedup under markovian sampling. In International Conference on Machine Learning, pp. 10997–11057. PMLR, 2022.
  24. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pp. 5381–5393. PMLR, 2020.
  25. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527, 2016.
  26. Sharp high-probability sample complexities for policy evaluation with linear function approximation, 2023a.
  27. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50–60, 2020.
  28. Accelerated and instance-optimal policy evaluation with linear function approximation. SIAM Journal on Mathematics of Data Science, 5(1):174–200, 2023b. doi: 10.1137/21M1468668. URL https://doi.org/10.1137/21M1468668.
  29. Federated reinforcement learning for training control policies on multiple iot devices. Sensors, 20(5):1359, 2020.
  30. Distributed td (0) with almost no communication. IEEE Control Systems Letters, 2023.
  31. Variance reduced proxskip: Algorithm, theory and application to federated learning. Advances in Neural Information Processing Systems, 35:15176–15189, 2022.
  32. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  33. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning, pp. 15750–15769. PMLR, 2022.
  34. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. Advances in Neural Information Processing Systems, 34:14606–14619, 2021.
  35. On the still unreasonable effectiveness of federated averaging for heterogeneous distributed learning. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities, 2023. URL https://openreview.net/forum?id=vhS68bKv7x.
  36. Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation. In International Conference on Artificial Intelligence and Statistics, pp.  5438–5448. PMLR, 2023.
  37. A remark on the stability of the LMS tracking algorithm. Stochastic analysis and applications, 16(1):119–129, 1998.
  38. Federated reinforcement learning: Techniques, applications, and open challenges. arXiv preprint arXiv:2108.11887, 2021.
  39. Federated learning’s blessing: Fedavg has linear speedup. In ICLR 2021-Workshop on Distributed and Private Machine Learning (DPML), 2021.
  40. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
  41. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3):400 – 407, 1951. doi: 10.1214/aoms/1177729586. URL https://doi.org/10.1214/aoms/1177729586.
  42. Finite-sample analysis of the Temporal Difference Learning. arXiv preprint arXiv:2310.14286, 2023.
  43. Finite-time error bounds for linear stochastic approximation and TD learning. In Conference on Learning Theory, pp.  2803–2830. PMLR, 2019.
  44. Sutton, R. S. Learning to predict by the methods of temporal differences. Machine learning, 3:9–44, 1988.
  45. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In Proceedings of the 26th annual international conference on machine learning, pp.  993–1000, 2009.
  46. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674–690, May 1997. ISSN 2334-3303. doi: 10.1109/9.580874.
  47. Federated temporal difference learning with linear function approximation under environmental heterogeneity. arXiv preprint arXiv:2302.02212, 2023.
  48. On the unreasonable effectiveness of federated averaging with heterogeneous data. arXiv preprint arXiv:2206.04723, 2022.
  49. Fedkl: Tackling data heterogeneity in federated reinforcement learning by penalizing kl divergence. IEEE Journal on Selected Areas in Communications, 41(4):1227–1242, 2023.
  50. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 3 posts and received 5 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube