Papers
Topics
Authors
Recent
2000 character limit reached

Policy Space Response Oracles: A Survey (2403.02227v2)

Published 4 Mar 2024 in cs.GT, cs.AI, and cs.MA

Abstract: Game theory provides a mathematical way to study the interaction between multiple decision makers. However, classical game-theoretic analysis is limited in scalability due to the large number of strategies, precluding direct application to more complex scenarios. This survey provides a comprehensive overview of a framework for large games, known as Policy Space Response Oracles (PSRO), which holds promise to improve scalability by focusing attention on sufficient subsets of strategies. We first motivate PSRO and provide historical context. We then focus on the strategy exploration problem for PSRO: the challenge of assembling effective subsets of strategies that still represent the original game well with minimum computational cost. We survey current research directions for enhancing the efficiency of PSRO, and explore the applications of PSRO across various domains. We conclude by discussing open questions and future research.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press, 2024.
  2. Z. An and L. Zhou. Double Oracle Algorithm for Game-Theoretic Robot Allocation on Graphs. arXiv:2312.11791, 2023.
  3. Competitive Environments Evolve Better Solutions for Complex Tasks. In ICGA, 1993.
  4. Competitive influence maximization: integrating budget allocation and seed selection. arXiv:1912.12283, 2019.
  5. DO-GAN: A Double Oracle Framework for Generative Adversarial Networks. In CVRP, 2022.
  6. Enumeration of nash equilibria for two-player games. Economic theory, 42, 2010.
  7. Re-evaluating evaluation. In NIPS, 2018.
  8. Open-ended learning in symmetric zero-sum games. In ICML, 2019.
  9. Ada: Adversarial data augmentation for object detection. In WACV. IEEE, 2019.
  10. Active learning for probabilistic structured prediction of cuts and matchings. In ICML, 2019.
  11. S. Behpour. Arc: Adversarial robust cuts for semi-supervised and multi-label classification. In CVPR, 2018.
  12. J. F. Benders. Partitioning procedures for solving mixed-variables programming problems. Numerische Mathematik, 4, 1962.
  13. Depth-limited solving for imperfect-information games. Advances in neural information processing systems, 31, 2018.
  14. Deep counterfactual regret minimization. In ICML, 2019.
  15. G. W. Brown. Iterative solution of games by fictitious play. Act. Anal. Prod Allocation, 13, 1951.
  16. J. Cui and X. Yang. Macta: A multi-agent reinforcement learning approach for cache timing attacks and detection. ICLR, 2023.
  17. Online double oracle. TMLR, 2022.
  18. F. Fang. Integrate learning with game theory for societal challenges. In IJCAI, 2019.
  19. Efficient and consistent adversarial bipartite matching. In ICML, 2018.
  20. Neural auto-curricula in two-player zero-sum games. In NeurIPS, 2021.
  21. A game-theoretic memory mechanism for coevolution. In GECCO, 2003.
  22. Actor-critic policy optimization in a large-scale imperfect-information game. In ICLR, 2022.
  23. Sample-based approximation of nash in large many-player games via gradient descent. In AAMAS, 2022.
  24. States as strings as strategies: Steering language models with game-theoretic solvers. arXiv:2402.01704, 2024.
  25. The non-zero-sum game of steganography in heterogeneous environments. IEEE Transactions on Information Forensics and Security, 2023.
  26. S. Govindan and R. Wilson. Computing nash equilibria by iterated polymatrix approximation. Journal of Economic Dynamics and Ctrl., 2004.
  27. Finding optimal abstract strategies in extensive-form games. In AAAI, 2012.
  28. Strategy exploration in empirical games. In AAMAS, 2010.
  29. Robust planning over restless groups: engagement interventions for a large-scale maternal telehealth program. In AAAI, volume 37, 2023.
  30. Exploiting extensive-form structure in empirical game-theoretic analysis. In WINE, 2022.
  31. A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning. NIPS, 2017.
  32. Z. Li and M. P. Wellman. Evolution strategies for approximate solution of bayesian games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5531–5540, 2021.
  33. Solving large-scale pursuit-evasion games using pre-trained strategies. In AAAI, volume 37, 2023.
  34. Combining Tree-Search, Generative Models, and Nash Bargaining Concepts in Game-Theoretic Reinforcement Learning. arXiv:2302.00797, 2023.
  35. Game-theoretic robust reinforcement learning handles temporally-coupled perturbations. arXiv:2307.12062, 2023.
  36. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. In NeurIPS, 2021.
  37. NeuPL: Neural population learning. arXiv:2202.07415, 2022.
  38. A Unified Diversity Measure for Multiagent Reinforcement Learning. In NeurIPS, 2022.
  39. A Survey on Population-Based Deep Reinforcement Learning. Mathematics, 11, 2023.
  40. Cooperative occupancy decision making of Multi-UAV in Beyond-Visual-Range air combat: A game theory approach. Ieee Access, 8, 2019.
  41. Red teaming game: A game-theoretic framework for red teaming language models. arXiv:2310.00322, 2023.
  42. Multi-agent training beyond zero-sum with correlated equilibrium meta-solvers. In ICML, 2021.
  43. Pipeline PSRO: A scalable approach for finding approximate Nash equilibria in large games. In NeurIPS, 2020.
  44. XDO: A double oracle algorithm for extensive-form games. In NeurIPS, 2021.
  45. Self-play psro: Toward optimal populations in two-player zero-sum games. arXiv:2207.06541, 2022.
  46. Anytime optimal psro for two-player zero-sum games. arXiv:2201.07700, 2022.
  47. ESCHER: Eschewing importance sampling in games by computing a history value function to estimate regret. In ICLR, 2023.
  48. Team-PSRO for learning approximate TMECor in large team games via cooperative reinforcement learning. In NeurIPS, 2023.
  49. Planning in the presence of cost functions controlled by an adversary. In ICML, 2003.
  50. A Generalized Training Approach for Multiagent Learning. In ICLR, 2020.
  51. Learning Equilibria in Mean-Field Games: Introducing Mean-Field PSRO. In AAMAS, 2022.
  52. A game-theoretic framework for controlled islanding in the presence of adversaries. In GameSec. Springer, 2021.
  53. The parallel Nash memory for asymmetric games. In GECCO, 2006.
  54. Beyond local Nash equilibria for adversarial networks. In BNAIC, 2019.
  55. α𝛼\alphaitalic_α-rank: Multi-agent evaluation by evolution. Scientific reports, 9, 2019.
  56. Modelling behavioural diversity for learning in open-ended games. In ICML, 2021.
  57. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378, 2022.
  58. Coevolutionary Principles, 2012.
  59. R. Savani and T. L. Turocy. Gambit: The Package for doing Computation in Game Theory, version 16.1.0., 2023.
  60. Exploring large strategy spaces in empirical game modeling. In AAMAS-AMEC Workshop, 2009.
  61. A game-theoretic framework for managing risk in multi-agent systems. In ICML, 2023.
  62. Co-Learning Empirical Games and World Models. arXiv:2305.14223, 2023.
  63. Strategic knowledge transfer. JMLR, 24, 2023.
  64. Learning deviation payoffs in simulation-based games. In AAAI, 2019.
  65. Regret-minimizing double oracle for extensive-form games. In ICML, 2023.
  66. Evolutionary stable strategies and game dynamics. Mathematical biosciences, 40, 1978.
  67. Finding needles in a moving haystack: Prioritizing alerts with adversarial reinforcement learning. In AAAI, volume 34, 2020.
  68. E. van Damme. Refinements of the Nash Equilibrium Concept. Springer Berlin Heidelberg, 2012.
  69. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575, 2019.
  70. B. von Stengel. Computing equilibria for two-person games. In Handbook of game theory with economic applications, volume 3. Elsevier, 2002.
  71. Y. Wang and M. P. Wellman. Empirical Game-Theoretic Analysis for Mean Field Games. In AAMAS, 2023.
  72. Y. Wang and M. P. Wellman. Regularization for Strategy Exploration in Empirical Game-Theoretic Analysis. arXiv:2302.04928, 2023.
  73. Y. Wang and M. P. Wellman. Generalized response objectives for strategy exploration in empirical game-theoretic analysis. In AAMAS, 2024.
  74. Deep reinforcement learning for green security games with real-time information. In AAAI, 2019.
  75. A game-theoretic approach for improving generalization ability of tsp solvers. arXiv preprint arXiv:2110.15105, 2021.
  76. Evaluating strategy exploration in empirical game-theoretic analysis. In AAMAS, 2022.
  77. M. P. Wellman. Methods for empirical game-theoretic analysis. In AAAI, 2006.
  78. Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949–980, 2014.
  79. Iterated deep reinforcement learning in games: History-aware training for improved stability. In ACM EC, 2019.
  80. Robust reinforcement learning under minimax regret for green security. In UAI, 2021.
  81. Y. Yang and J. Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv:2011.00583, 2020.
  82. Policy space diversity for non-transitive games. arXiv:2306.16884, 2023.
  83. Optimal defense against election control by deleting voter groups. Artificial Intelligence, 259, 2018.
  84. Computing optimal equilibria and mechanisms via learning in zero-sum extensive-form games. In NeurIPS, 2023.
  85. Efficient Policy Space Response Oracles. arXiv:2202.00633, 2022.
  86. Regret minimization in games with incomplete information. NIPS, 2007.
  87. Equilibrium Approximating and Online Learning for Anti-Jamming Game of Satellite Communication Power Allocation. Electronics, 11(21):3526, 2022.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Slide Deck Streamline Icon: https://streamlinehq.com

Whiteboard

Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Sign up for free to view the 3 tweets with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free