Rationality of Learning Algorithms in Repeated Normal-Form Games (2402.08747v1)
Abstract: Many learning algorithms are known to converge to an equilibrium for specific classes of games if the same learning algorithm is adopted by all agents. However, when the agents are self-interested, a natural question is whether agents have a strong incentive to adopt an alternative learning algorithm that yields them greater individual utility. We capture such incentives as an algorithm's rationality ratio, which is the ratio of the highest payoff an agent can obtain by deviating from a learning algorithm to its payoff from following it. We define a learning algorithm to be $c$-rational if its rationality ratio is at most $c$ irrespective of the game. We first establish that popular learning algorithms such as fictitious play and regret matching are not $c$-rational for any constant $c\geq 1$. We then propose and analyze two algorithms that are provably $1$-rational under mild assumptions, and have the same properties as (a generalized version of) fictitious play and regret matching, respectively, if all agents follow them. Finally, we show that if an assumption of perfect monitoring is not satisfied, there are games for which $c$-rational algorithms do not exist, and illustrate our results with numerical case studies.
- A cooperative multi-agent transportation management and route guidance system. Transportation Research Part C: Emerging Technologies, 10(5-6):433–454, 2002.
- Intrusion response systems for cyber-physical systems: A comprehensive survey. Computers & Security, 124:102984, 2023.
- Fictitious play and best-response dynamics in identical interest and zero-sum stochastic games. In International Conference on Machine Learning, pp. 1664–1690. PMLR, 2022.
- Berger, U. Fictitious play in 2×\times× n games. Journal of Economic Theory, 120(2):139–154, 2005.
- Multiagent learning using a variable learning rate. Artificial intelligence, 136(2):215–250, 2002.
- Efficient learning equilibrium. Advances in Neural Information Processing Systems, 15, 2002.
- Brown, G. W. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13(1):374, 1951.
- Convergence analysis of gradient-based learning with non-uniform learning rates in non-cooperative multi-agent settings. arXiv preprint arXiv:1906.00731, 2019.
- Distributed learning and cooperative control for multi-agent systems. Automatica, 45(12):2802–2814, 2009.
- Awesome: A general multiagent learning algorithm that converges in self-play and learns a best response against stationary opponents. Machine Learning, 67:23–43, 2007.
- Multi-agent reinforcement learning-based resource allocation for uav networks. IEEE Transactions on Wireless Communications, 19(2):729–743, 2020. doi: 10.1109/TWC.2019.2935201.
- Balancing adaptability and non-exploitability in repeated games. In Uncertainty in Artificial Intelligence, pp. 559–568. PMLR, 2022.
- On the impossibility of predicting the behavior of rational agents. Proceedings of the National Academy of Sciences, 98(22):12848–12853, 2001.
- The theory of learning in games, volume 2. MIT press, 1998.
- Bounds for regret-matching algorithms. In AI&M, 2006.
- A simple adaptive procedure leading to correlated equilibrium. Econometrica, 68(5):1127–1150, 2000.
- Hespanha, J. P. Noncooperative game theory: An introduction for engineers and computer scientists. Princeton University Press, 2017.
- Multiagent reinforcement learning: theoretical framework and an algorithm. In ICML, volume 98, pp. 242–250, 1998.
- Foolproof cooperative learning. In Asian Conference on Machine Learning, pp. 401–416. PMLR, 2020.
- On no-regret learning, fictitious play, and Nash equilibrium. In ICML, volume 1, pp. 226–233, 2001.
- Rational learning leads to nash equilibrium. Econometrica: Journal of the Econometric Society, pp. 1019–1045, 1993.
- Kosorok, M. R. Introduction to empirical processes and semiparametric inference, volume 61. Springer, 2008.
- A multiagent approach to q𝑞qitalic_q-learning for daily stock trading. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 37(6):864–877, 2007.
- Generalised weakened fictitious play. Games and Economic Behavior, 56(2):285–298, 2006.
- Littman, M. L. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157–163. Elsevier, 1994.
- Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017.
- Fictitious play property for games with identical interests. Journal of economic theory, 68(1):258–265, 1996.
- Nachbar, J. H. Bayesian learning in repeated games of incomplete information. Social Choice and Welfare, 18:303–326, 2001.
- Non-cooperative energy efficient power allocation game in d2d communication: A multi-agent deep reinforcement learning approach. IEEE Access, 7:100480–100490, 2019.
- Run the gamut: A comprehensive approach to evaluating game-theoretic algorithms. In AAMAS, volume 4, pp. 880–887, 2004.
- Multi-agent deep reinforcement learning for multi-robot applications: a survey. Sensors, 23(7):3625, 2023.
- Robinson, J. An iterative method of solving a game. Annals of mathematics, pp. 296–301, 1951.
- Fictitious play in zero-sum stochastic games. SIAM Journal on Control and Optimization, 60(4):2095–2114, 2022.
- Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008.
- Intelligent players in a fictitious play framework. IEEE Transactions on Automatic Control, 2023.
- An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583, 2020.
- Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321–384, 2021a.
- Mfvfd: A multi-agent q-learning approach to cooperative and non-cooperative tasks. In IJCAI, pp. 500–506, 2021b.
Collections
Sign up for free to add this paper to one or more collections.