No-Press Diplomacy from Scratch (2110.02924v1)

Abstract: Prior AI successes in complex games have largely focused on settings with at most hundreds of actions at each decision point. In contrast, Diplomacy is a game with more than 10^20 possible actions per turn. Previous attempts to address games with large branching factors, such as Diplomacy, StarCraft, and Dota, used human data to bootstrap the policy or used handcrafted reward shaping. In this paper, we describe an algorithm for action exploration and equilibrium approximation in games with combinatorial action spaces. This algorithm simultaneously performs value iteration while learning a policy proposal network. A double oracle step is used to explore additional actions to add to the policy proposals. At each state, the target state value and policy for the model training are computed via an equilibrium search procedure. Using this algorithm, we train an agent, DORA, completely from scratch for a popular two-player variant of Diplomacy and show that it achieves superhuman performance. Additionally, we extend our methods to full-scale no-press Diplomacy and for the first time train an agent from scratch with no human data. We present evidence that this agent plays a strategy that is incompatible with human-data bootstrapped agents. This presents the first strong evidence of multiple equilibria in Diplomacy and suggests that self play alone may be insufficient for achieving superhuman performance in Diplomacy.

Overview of "No-Press Diplomacy from Scratch"

The paper, "No-Press Diplomacy from Scratch," by Bakhtin et al. addresses the complex challenge of developing AI capable of mastering the board game Diplomacy without relying on human gameplay data. This is significant due to the game's large combinatorial action space, which presents substantial difficulties in action exploration and equilibrium approximation. This work focuses on the development and evaluation of an algorithm designed to overcome these challenges through self-play, employing mechanisms such as double oracle methods and deeper reinforcement learning strategies.

Main Contributions

1. Algorithm Development: The authors introduce a new algorithm to train AI agents for games like Diplomacy, characterized by high branching factors due to numerous legal actions per turn. This involves innovative use of value iteration alongside a policy proposal network to systematically explore action spaces.
2. Double Oracle Reinforcement Learning: By integrating double oracle (DO) methodologies, the authors bolster action exploration capabilities during training. This DO process allows the agent to discover and integrate additional strategies dynamically, improving the robustness of learned policies in vast action landscapes.
3. Training from Scratch: The paper successfully trains an agent for Diplomacy from scratch, bypassing the traditional reliance on human-data-bootstrapped models. This approach highlights the potential for AI to independently learn superhuman strategies in complex games.
4. Empirical Evaluation: The trained agent demonstrates superhuman performance in a two-player variant of Diplomacy without any human data. The outcome reveals that the agent develops strategies significantly distinct from those trained on human data, exemplifying the presence of multiple equilibria within the game.
5. Benchmarking Multi-agent Systems: The paper corroborates Diplomacy's validity as a critical benchmark for advancing AI in multi-agent systems, particularly focusing on how self-play might limit performance when contrasting with human-like equilibria.

Key Results

• The trained agent not only outperformed previous agents that depended on human gameplay data but also showed strategic behavior that deviated markedly from conventional human strategies.
• These results underline the possibility of achieving superhuman performance through self-play alone, despite the wide branching factor of the action space in Diplomacy.
• The authors present direct evidence that self-play can converge on multiple inequivalent equilibria, suggesting that learning AI strategies solely from self-play might not always align well with human strategic understanding.

Implications and Future Directions

This research has significant implications for both theoretical and applied aspects of AI:

• Theoretical Implications: The identification of multiple equilibria within complex games like Diplomacy highlights nuanced understanding necessary when designing AI training regimes in multi-agent environments. It demonstrates that equilibria discovered through self-play can be fundamentally different from those established by human players.
• Practical Implications: This algorithm sets a precedent for deploying AI systems in environments where human data is scarce or unavailable. It marks an advancement for applications requiring AI that can independently derive strategies in high-dimensional, multi-agent settings.
• Future Developments: This work opens several avenues for future research. Expanding these techniques to handle communication aspects in Diplomacy, exploring multi-equilibrium strategies in non-zero-sum or cooperative settings, and refining the integration of reward shaping to align agent strategies closer to human play are potential directions.

In summary, Bakhtin et al.'s research is a substantial stride toward understanding and leveraging AI capabilities in realms matching human-level strategic complexity. The methodological innovations and results presented here lay a robust groundwork for exploring the full potential of AI in intricate game environments and beyond.