Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Peer Learning: Learning Complex Policies in Groups from Scratch via Action Recommendations (2312.09950v2)

Published 15 Dec 2023 in cs.LG, cs.AI, and cs.MA

Abstract: Peer learning is a novel high-level reinforcement learning framework for agents learning in groups. While standard reinforcement learning trains an individual agent in trial-and-error fashion, all on its own, peer learning addresses a related setting in which a group of agents, i.e., peers, learns to master a task simultaneously together from scratch. Peers are allowed to communicate only about their own states and actions recommended by others: "What would you do in my situation?". Our motivation is to study the learning behavior of these agents. We formalize the teacher selection process in the action advice setting as a multi-armed bandit problem and therefore highlight the need for exploration. Eventually, we analyze the learning behavior of the peers and observe their ability to rank the agents' performance within the study group and understand which agents give reliable advice. Further, we compare peer learning with single agent learning and a state-of-the-art action advice baseline. We show that peer learning is able to outperform single-agent learning and the baseline in several challenging discrete and continuous OpenAI Gym domains. Doing so, we also show that within such a framework complex policies from action recommendations beyond discrete action spaces can evolve.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297: 103500.
  2. A Framework for Behavioural Cloning. In Machine Intelligence 15, 103–129.
  3. Social learning theory, volume 1. Englewood cliffs Prentice Hall.
  4. Transition-independent decentralized Markov decision processes. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems, 41–48.
  5. A conceptual framework for externally-influenced agents: an assisted reinforcement learning review. Journal of Ambient Intelligence and Humanized Computing.
  6. Social facilitation: A meta-analysis of 241 studies. Psychological bulletin, 94(2): 265.
  7. OpenAI Gym. arXiv:1606.01540.
  8. Policy improvement via imitation of multiple oracles. Advances in Neural Information Processing Systems, 33: 5587–5598.
  9. Poisoning Attack for Inter-agent Transfer Learning. In Security and Privacy in Communication Networks: 17th EAI International Conference, SecureComm 2021, Virtual Event, September 6–9, 2021, Proceedings, Part II 17, 394–404. Springer.
  10. Simultaneously learning and advising in multiagent reinforcement learning. In Proceedings of the 16th conference on autonomous agents and multiagent systems, 1100–1108.
  11. Uncertainty-Aware Action Advising for Deep Reinforcement Learning Agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 5792–5799.
  12. Agents teaching agents: a survey on inter-agent transfer learning. Autonomous Agents and Multi-Agent Systems, 34(1): 1–17.
  13. An Advice Mechanism for Heterogeneous robot Teams. International Journal of Robotics and Automation, 35(1).
  14. Addressing Function Approximation Error in Actor-Critic Methods. In International conference on machine learning, 1587–1596. PMLR.
  15. Geen, R. G. 1991. Social motivation. Annual review of psychology, 42(1): 377–399.
  16. Guerin, B. 1983. Social facilitation and social monitoring: A test of three models. British Journal of Social Psychology, 22(3): 203–214.
  17. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 1861–1870. PMLR.
  18. Action advising with advice imitation in deep reinforcement learning. arXiv:2104.08441.
  19. The Impact of Study Groups and Roommates on Academic Performance. The Review of Economics and Statistics, 97(1): 44–54.
  20. Kretchmar, R. M. 2003. Reinforcement learning algorithms for homogenous multi-agent systems. In Workshop on Agent and Swarm Programming, volume 94.
  21. Minsky, M. 1961. Steps toward artificial intelligence. Proceedings of the IRE, 49(1): 8–30.
  22. Playing atari with deep reinforcement learning. arXiv:1312.5602.
  23. Human-level control through deep reinforcement learning. Nature, 518: 529–533.
  24. Emergent social learning via multi-agent reinforcement learning. In International Conference on Machine Learning, 7991–8004. PMLR.
  25. Algorithms for Inverse Reinforcement Learning. In Proceedings of the 17th International Conference on Machine Learning, volume 1, 663–670.
  26. On learning by exchanging advice. arXiv:cs/0203010.
  27. Exchanging advice and learning to trust. In Cooperative Information Agents VII: 7th International Workshop, CIA 2003, Helsinki, Finland, August 27-29, 2003. Proceedings 7, 250–265. Springer.
  28. Learning to teach in cooperative multiagent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 6128–6136.
  29. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, 8024–8035. Curran Associates, Inc.
  30. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268): 1–8.
  31. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. In Gordon, G.; Dunson, D.; and Dudík, M., eds., Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, 627–635. Fort Lauderdale, FL, USA: PMLR.
  32. Rotter, J. B. 1954. Social learning and clinical psychology. Prentice-Hall, Inc.
  33. Student use of out-of-class study groups in an introductory undergraduate biology course. CBE—Life Sciences Education, 10(1): 74–82.
  34. Skinner, B. F. 1988. The selection of behavior: The operant behaviorism of BF Skinner: Comments and consequences. CUP Archive.
  35. Slivkins, A. 2019. Introduction to Multi-Armed Bandits. Foundations and Trends® in Machine Learning, 12(1–2): 1–286.
  36. Reinforcement learning: An introduction. MIT press.
  37. Thorndike, E. L. 1898. Animal intelligence: An experimental study of the associative processes in animals. The Psychological Review: Monograph Supplements, 2(4): i.
  38. Thorpe, W. H.; et al. 1979. Origins and rise of ethology. Heinemann Educational Books.
  39. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 5026–5033. IEEE.
  40. Teaching on a budget: Agents advising agents in reinforcement learning. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, 1053–1060.
  41. Turing, A. M. 1951. Intelligent machinery, a heretical theory. The Turing digital archive (http://www.turingarchive.org), contents of AMT/B/4.
  42. Learning to make things happen: Infants’ observational learning of social and physical causal events. Journal of Experimental Child Psychology, 162: 58–71.
  43. Peer Teaching: To Teach Is To Learn Twice. ASHE-ERIC Higher Education Report No. 4, 1988. ERIC.
  44. Social enhancement and impairment of performance in the cockroach. Journal of Personality and Social Psychology, 13(2): 83.
  45. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, 321–384.
  46. A survey of multi-agent reinforcement learning with communication. arXiv:2203.08975.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets