Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning

Abstract: We introduce DeepNash, an autonomous agent capable of learning to play the imperfect information game Stratego from scratch, up to a human expert level. Stratego is one of the few iconic board games that AI has not yet mastered. This popular game has an enormous game tree on the order of $10^{535}$ nodes, i.e., $10^{175}$ times larger than that of Go. It has the additional complexity of requiring decision-making under imperfect information, similar to Texas hold'em poker, which has a significantly smaller game tree (on the order of $10^{164}$ nodes). Decisions in Stratego are made over a large number of discrete actions with no obvious link between action and outcome. Episodes are long, with often hundreds of moves before a player wins, and situations in Stratego can not easily be broken down into manageably-sized sub-problems as in poker. For these reasons, Stratego has been a grand challenge for the field of AI for decades, and existing AI methods barely reach an amateur level of play. DeepNash uses a game-theoretic, model-free deep reinforcement learning method, without search, that learns to master Stratego via self-play. The Regularised Nash Dynamics (R-NaD) algorithm, a key component of DeepNash, converges to an approximate Nash equilibrium, instead of 'cycling' around it, by directly modifying the underlying multi-agent learning dynamics. DeepNash beats existing state-of-the-art AI methods in Stratego and achieved a yearly (2022) and all-time top-3 rank on the Gravon games platform, competing with human expert players.

• The paper introduces DeepNash, a novel AI model that uses model-free multiagent reinforcement learning to converge toward an approximate Nash equilibrium in the complex, imperfect-information game of Stratego.
• It leverages Regularized Nash Dynamics to manage the vast game tree of 10^535 nodes without relying on heavy search techniques, setting a new benchmark in game strategy learning.
• The AI achieved an 84% win rate and ranked among the top three human experts on the Gravon platform, highlighting its potential for real-world strategic applications.

Overview of "Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning"

The paper introduces DeepNash, an innovative AI model designed to master the game of Stratego, utilizing model-free multiagent reinforcement learning techniques. Stratego is characterized by its vast game complexity and imperfect information aspect, making it notably more challenging than games like chess or Go, which have already been mastered by AI in the past. The intricacy of Stratego, with its game tree complexity on the order of $10^{535}$ nodes—a figure significantly larger than Go's—imposes considerable challenges for AI, mainly due to the imperfect information setting where players cannot see all pieces on the board until they interact directly.

Methodology

DeepNash is based on the Regularized Nash Dynamics (R-NaD), a game-theoretic algorithm facilitating convergence to an approximate Nash equilibrium by altering multi-agent learning dynamics. This approach diverges from traditional methods that use heavy search techniques, by opting for a model-free strategy, beneficial in handling the vast state space and lengthy episode nature of Stratego. DeepNash enables the learning of both deployment and gameplay phases of Stratego without human intervention or predefined strategies, leveraging dynamic learning through self-play.

Results

DeepNash achieves remarkable performance, outperforming current state-of-the-art AI methods in Stratego, evidenced by its significant win rates against established AI Stratego bots. The demonstrated superiority of DeepNash is further illustrated by its performance on the Gravon games platform, where it not only achieved a top-three ranking among human expert players but sustained an 84% win rate. These results underscore the practical implications of DeepNash, showcasing AI's capability to learn strategic behaviors such as bluffing and trading-off material versus information without human-designed strategies imparted through data.

Implications and Future Developments

This research marks a significant milestone in the field of AI, highlighting the potential of model-free reinforcement learning in mastering complex strategic interactions in imperfect information settings. The use of R-NaD within DeepNash paves the way for applications beyond game playing, particularly in real-world scenarios that require decision-making under uncertainty. Future developments could extend such methodologies to broader applications, enabling AI to tackle challenges in domains where strategic planning amidst incomplete information is vital.

Concluding Remarks

DeepNash's success in mastering Stratego offers a profound insight into the capabilities of modern AI in overcoming long-standing challenges posed by complex games. It affirms the applicability of model-free reinforcement learning and provides a framework that could inspire subsequent research endeavors across various domains of artificial intelligence requiring strategy formation under uncertainty.