Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Dynamics of Moral Behavior in Heterogeneous Populations of Learning Agents (2403.04202v6)

Published 7 Mar 2024 in cs.MA, cs.AI, cs.CY, and cs.LG

Abstract: Growing concerns about safety and alignment of AI systems highlight the importance of embedding moral capabilities in artificial agents: a promising solution is the use of learning from experience, i.e., Reinforcement Learning. In multi-agent (social) environments, complex population-level phenomena may emerge from interactions between individual learning agents. Many of the existing studies rely on simulated social dilemma environments to study the interactions of independent learning agents; however, they tend to ignore the moral heterogeneity that is likely to be present in societies of agents in practice. For example, at different points in time a single learning agent may face opponents who are consequentialist (i.e., focused on maximizing outcomes over time), norm-based (i.e., conforming to specific norms), or virtue-based (i.e., considering a combination of different virtues). The extent to which agents' co-development may be impacted by such moral heterogeneity in populations is not well understood. In this paper, we present a study of the learning dynamics of morally heterogeneous populations interacting in a social dilemma setting. Using an Iterated Prisoner's Dilemma environment with a partner selection mechanism, we investigate the extent to which the prevalence of diverse moral agents in populations affects individual agents' learning behaviors and emergent population-level outcomes. We observe several types of non-trivial interactions between pro-social and anti-social agents, and find that certain types of moral agents are able to steer selfish agents towards more cooperative behavior.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Reinforcement learning as a framework for ethical decision making. In Proceedings of the AAAI Workshop: AI, Ethics, and Society (AIES’16), pp.  54–61, 2016.
  2. Partner selection for the emergence of cooperation in multi-agent systems using reinforcement learning. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI’20), volume 34, pp.  7047–7054, 2020.
  3. Philip W. Anderson. More is different. Science, 177(4047):393–396, 1972.
  4. Aristotle. The Nicomachean Ethics. Oxford University Press.
  5. Bowen Baker. Emergent reciprocity and team formation from randomized uncertain social preferences. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS’20), volume 33, pp.  15786–15799, 2020.
  6. Partner choice creates competitive altruism in humans. Proceedings of the Royal Society B: Biological Sciences, 274(1610):749–753, 2007.
  7. Moral sentiments in multi-agent systems. In Jörg P. Müller, Anand S. Rao, and Munindar P. Singh (eds.), Intelligent Agents V: Agents Theories, Architectures, and Languages, pp.  113–131. Springer Berlin Heidelberg, 1999.
  8. Universality and cultural diversity in moral reasoning and judgment. Frontiers in Psychology, 12, 2021.
  9. Jeremy Bentham. An Introduction to the Principles of Morals and Legislation. Clarendon Press, 1780.
  10. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156–172, 2008.
  11. Colin F. Camerer. Behavioral Game Theory: Experiments in Strategic Interaction. Princeton University Press, 2011.
  12. Mathematical foundations of moral preferences. Journal of The Royal Society Interface, 18(175):20200880, 2021.
  13. Intrinsically motivated reinforcement learning. In Proceedings of the 18th International Conference on Neural Information Processing Systems (NeurIPS’04), volume 17, pp.  1281–1288, 2004.
  14. Reputation drives cooperative behaviour and network formation in human groups. Scientific Reports, 5(1):7843, 2015.
  15. Robyn M. Dawes. Social dilemmas. Annual Review of Psychology, 31(1):169–193, 1980.
  16. Learning to cooperate in multi-agent social dilemmas. In Proceedings of the 5th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’06), pp.  783–785, 2006.
  17. Why social preferences matter – the impact of non-selfish motives on competition, cooperation and incentives. The Economic Journal, 112(478):1–33, 2002.
  18. Social norms and human cooperation. Trends in Cognitive Sciences, 8(4):185–190, 2004.
  19. Bernard Gert. Common Morality: Deciding What to Do. Oxford University Press, New York, 2004.
  20. Corrado Gini. Variabilità e Mutabilità: Contributo allo studio delle distribuzioni e delle relazioni statistiche. [Fasc. I.]. Tipografia di P. Cuppini, 1912.
  21. Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology, 96(5):1029–1046, 2009.
  22. Moral Foundations Theory: The pragmatic validity of moral pluralism. In Advances in Experimental Social Psychology, volume 47, pp.  55–130. Academic Press, 2013.
  23. Evolutionary Games and Population Dynamics. Cambridge University Press, 1998.
  24. Inequity aversion improves cooperation in intertemporal social dilemmas. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS’18), volume 31, pp.  3330–3340, 2018.
  25. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning (ICML’19), pp.  3040–3049. PMLR, 2019.
  26. Immanuel Kant. Grounding for the Metaphysics of Morals. Cambridge University Press, 1785.
  27. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR’15), 2015.
  28. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’17), pp.  464–473, 2017.
  29. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00742, 2019.
  30. Peter D. Lifton. Individual differences in moral development: The relation of sex, gender, and personality to morality. Journal of Personality, 53(2):306–334, 1985.
  31. Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the 11th International Conference on International Conference on Machine Learning (ICML’94), pp.  157–163, 1994.
  32. Conor Mayo-Wilson and Kevin J. S. Zollman. The computational philosophy: simulation as a core philosophical method. Synthese, 199(1-2):3647–3673, 2021.
  33. Social diversity and social preferences in mixed-motive reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS’20), pp.  869–877, 2020.
  34. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 36(1):21, 2022.
  35. Alexander J. McKenzie. The Structural Evolution of Morality. Cambridge University Press, 2007.
  36. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
  37. Toby Ord. Moral trade. Ethics, 126(1):118–138, 2015.
  38. Consequentialist conditional cooperation in social dilemmas with imperfect information. In Proceedings of the 6th International Conference on Learning Representations (ICLR’18), 2018.
  39. Anatol Rapoport. Prisoner’s dilemma — recollections and observations. In Game Theory as a Theory of a Conflict Resolution, pp.  17–34. Springer, 1974.
  40. Thomas Reid on Practical Ethics. Princeton University Press, 1990.
  41. Multiagent reinforcement learning in the iterated prisoner’s dilemma. Biosystems, 37(1-2):147–166, 1996.
  42. Social diversity promotes the emergence of cooperation in public goods games. Nature, 454(7201):213–216, 2008.
  43. Prakash P. Shenoy. On coalition formation: A game-theoretical approach. In International Journal of Game Theory, volume 8, 1979.
  44. Karl Sigmund. The Calculus of Selfishness. Princeton University Press, 2010.
  45. Evolutionary game theory. Current Biology, 9(14):R503–R505, 1999.
  46. Walter Sinnott-Armstrong. Moral Psychology: The Evolution of Morality: Adaptations and Innateness. MIT Press, 2008.
  47. Reinforcement Learning: An Introduction. MIT Press, 2018. ISBN 0262039249.
  48. Modeling moral choices in social dilemmas with multi-agent reinforcement learning. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI’23), pp.  317–325, 2023a.
  49. Learning machine morality through experience and interaction. arXiv Preprint. arXiv:2312.01818, 2023b.
  50. Q-learning. Machine Learning, 8(3):279–292, 1992.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Elizaveta Tennant (4 papers)
  2. Stephen Hailes (23 papers)
  3. Mirco Musolesi (81 papers)
Citations (1)
View on Semantic Scholar