Strategy Game-Playing with Size-Constrained State Abstraction

Abstract: Playing strategy games is a challenging problem for AI. One of the major challenges is the large search space due to a diverse set of game components. In recent works, state abstraction has been applied to search-based game AI and has brought significant performance improvements. State abstraction techniques rely on reducing the search space, e.g., by aggregating similar states. However, the application of these abstractions is hindered because the quality of an abstraction is difficult to evaluate. Previous works hence abandon the abstraction in the middle of the search to not bias the search to a local optimum. This mechanism introduces a hyper-parameter to decide the time to abandon the current state abstraction. In this work, we propose a size-constrained state abstraction (SCSA), an approach that limits the maximum number of nodes being grouped together. We found that with SCSA, the abstraction is not required to be abandoned. Our empirical results on $3$ strategy games show that the SCSA agent outperforms the previous methods and yields robust performance over different games. Codes are open-sourced at https://github.com/GAIGResearch/Stratega.

• The paper introduces the SCSA method, a novel extension to MCTS that limits state abstractions using fixed-size nodes.
• It demonstrates superior AI performance in games like Kill The King and Push Them All through robust statistical comparisons.
• The research highlights the practical use of a uniform size limit for abstraction while noting scalability challenges in highly complex environments.

Strategy Game-Playing with Size-Constrained State Abstraction

Introduction

The paper "Strategy Game-Playing with Size-Constrained State Abstraction" addresses the complex problem of enhancing AI performance in strategy games by reducing the search space through the application of state abstraction techniques. Traditional deep reinforcement learning (DRL) agents, while effective, suffer from a lack of generalizability across different game variants due to their dependence on game-specific network training and tuning. Search-based algorithms like Monte Carlo Tree Search (MCTS) present an alternative with demonstrated efficacy in general video game-playing. However, the combinatorial search space of strategy games presents significant challenges, particularly in maintaining accurate action-value approximations.

Size-Constrained State Abstraction (SCSA)

The primary innovation presented is the Size-Constrained State Abstraction (SCSA), an extension of MCTS that aggregates similar states into abstract nodes with a fixed size limit. This method contrasts sharply with the Elastic MCTS approach, which employs approximate MDP homomorphism and an early stop mechanism to abandon poor-quality abstractions. In SCSA, node group sizes are limited by a predetermined parameter to ensure that the abstraction quality remains consistently manageable without the need for abandonment.

Algorithmic Details

The construct of the SCSA algorithm is rooted in iterative updating of state abstraction groups after every fixed batch of MCTS iterations. The abstract nodes are bounded by a size constraint (SIZE_LIMIT), ensuring that overly large state abstractions do not dominate the search tree and compromise search accuracy. The empirical results indicate that this approach not only eliminates the need for an early stop threshold but also maintains robustness across various games, using uniform parameters.

Experimental Framework and Results

The experiments involved evaluation across three distinct strategy games from the Stratega platform: Kill The King (KTK), Push Them All (PTA), and Two Kingdoms (TK). The SCSA agent was rigorously compared against several baselines: Rule-based agents, standard MCTS, MCTS with unit ordering (MCTS$_u$), randomized state abstraction (RG MCTS$_u$), and Elastic MCTS$_u$. The tests were performed using different initial unit positions with multiple evaluations per position to ensure statistical robustness.

Numerical Performance

The results demonstrated that the SCSA agent generally performed strongly, particularly excelling in KTK and PTA. In the more complex TK, SCSA showed competitive results, suggesting that while effective, the approach may have scalability constraints in highly complex environments. The win rates and statistical significance across different pairs of agents validate the superior performance and reliability of the SCSA approach in managing state abstraction.

Implications and Future Developments

The findings indicate several critical implications for AI research in game-playing:

• Generalizability: The SCSA’s performance across different games underscores its potential for broader applicability in various strategic and real-time scenarios.
• Scalability: Although effective, the performance gap in complex environments such as TK highlights the need for enhanced strategies, potentially integrating SCSA with pruning mechanisms to manage larger state spaces more efficiently.
• Practical Utility: The use of a uniform SIZE_LIMIT across different games simplifies the parameter optimization process, making SCSA a practical choice for developers seeking robust AI implementations in dynamic game environments.

In future work, researchers may focus on hybrid approaches that blend the benefits of state abstraction with other techniques like pruning to address scalability issues. Further exploration into the long-term impact of SCSA on real-world applications, particularly in varying game complexities and scenarios, will be pivotal in refining and advancing AI gameplay strategies.

Conclusion

The paper presents a novel and effective approach to handling large search spaces in strategy games through Size-Constrained State Abstraction. The SCSA method shows substantial promise in enhancing AI performance by maintaining functional abstraction sizes and avoiding the pitfalls associated with poor-quality state abstractions. The empirical results validate the approach across multiple game environments, positioning it as a valuable contribution to the field of game-playing AI research. The ongoing work will need to explore scalability solutions and potential integrations with other methodologies to further improve performance in highly complex real-world applications.