CAESAR: Enhancing Federated RL in Heterogeneous MDPs through Convergence-Aware Sampling with Screening
Abstract: In this study, we delve into Federated Reinforcement Learning (FedRL) in the context of value-based agents operating across diverse Markov Decision Processes (MDPs). Existing FedRL methods typically aggregate agents' learning by averaging the value functions across them to improve their performance. However, this aggregation strategy is suboptimal in heterogeneous environments where agents converge to diverse optimal value functions. To address this problem, we introduce the Convergence-AwarE SAmpling with scReening (CAESAR) aggregation scheme designed to enhance the learning of individual agents across varied MDPs. CAESAR is an aggregation strategy used by the server that combines convergence-aware sampling with a screening mechanism. By exploiting the fact that agents learning in identical MDPs are converging to the same optimal value function, CAESAR enables the selective assimilation of knowledge from more proficient counterparts, thereby significantly enhancing the overall learning efficiency. We empirically validate our hypothesis and demonstrate the effectiveness of CAESAR in enhancing the learning efficiency of agents, using both a custom-built GridWorld environment and the classical FrozenLake-v1 task, each presenting varying levels of environmental heterogeneity.
- Openai gym. In arXiv preprint arXiv:1606.01540.
- Chapter 14 - Federated sequential decision making: Bayesian optimization, reinforcement learning, and beyond. In Federated Learning, Lam M. Nguyen, Trong Nghia Hoang, and Pin-Yu Chen (Eds.). Academic Press, 257–279. https://doi.org/10.1016/B978-0-44-319037-7.00023-5
- Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee. In 35th Conference on Neural Information Processing Systems (NeurIPS).
- FedHQL: Federated Heterogeneous Q-Learning. arXiv:2301.11135.
- Federated Reinforcement Learning for the Building Facilities. In 2022 IEEE International Conference on Omni-layer Intelligent Systems (COINS). 1–6. https://doi.org/10.1109/COINS54846.2022.9854959
- Federated reinforcement learning with environment heterogeneity. In International Conference on Artificial Intelligence and Statistics (AISTATS).
- Watkins Christopher JCH and Peter Dayan. 1992. Q-learning. In Machine learning 8.
- Decentralized Federated Policy Gradient with Byzantine Fault-Tolerance and Provably Fast Convergence. (2024). https://arxiv.org/abs/2401.03489
- Federated Reinforcement Learning: Linear Speedup Under Markovian Sampling. In Proceedings of the 39th International Conference on Machine Learning (ICML).
- Federated transfer reinforcement learning for autonomous driving. In Federated and Transfer Learning. Springer, 357–371.
- Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems. IEEE Robotics and Automation Letters 4, 4 (2019), 4555–4562. https://doi.org/10.1109/LRA.2019.2931179
- Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS).
- Federated Reinforcement Learning for Fast Personalization. In 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE). 123–127. https://doi.org/10.1109/AIKE.2019.00031
- Federated reinforcement learning: Techniques, applications, and open challenges. (2021).
- Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup. IEEE Transactions on Signal Processing (2023).
- Richard Sutton and Andrew Barto. 2018. Reinforcement learning: An introduction. MIT press.
- Federated deep reinforcement learning for Internet of Things with decentralized cooperative edge caching. IEEE Internet of Things Journal 7, 10 (2020), 9441–9455.
- The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyond. In Proceedings of the 40th International Conference on Machine Learning.
- A resource-constrained and privacy-preserving edge-computing-enabled clinical decision system: A federated reinforcement learning approach. IEEE Internet of Things Journal 8, 11 (2021), 9122–9138.
- Federated Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning (2020), 121–131.
- When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multitimescale Resource Management for Multiaccess Edge Computing in 5G Ultradense Network. IEEE Internet of Things Journal 8, 4 (2020), 2238–2251.
- Resilient Mechanism Against Byzantine Failure for Distributed Deep Reinforcement Learning. In 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE). IEEE, 378–389.
- Federated deep reinforcement learning. arXiv:1901.08277.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.