Mastering Board Games by External and Internal Planning with Language Models (2412.12119v3)

Abstract: Advancing planning and reasoning capabilities of LLMs is one of the key prerequisites towards unlocking their potential for performing reliably in complex and impactful domains. In this paper, we aim to demonstrate this across board games (Chess, Fischer Random / Chess960, Connect Four, and Hex), and we show that search-based planning can yield significant improvements in LLM game-playing strength. We introduce, compare and contrast two major approaches: In external search, the model guides Monte Carlo Tree Search (MCTS) rollouts and evaluations without calls to an external game engine, and in internal search, the model is trained to generate in-context a linearized tree of search and a resulting final choice. Both build on a LLM pre-trained on relevant domain knowledge, reliably capturing the transition and value functions in the respective environments, with minimal hallucinations. We evaluate our LLM search implementations against game-specific state-of-the-art engines, showcasing substantial improvements in strength over the base model, and reaching Grandmaster-level performance in chess while operating closer to the human search budget. Our proposed approach, combining search with domain knowledge, is not specific to board games, hinting at more general future applications.

• The paper introduces dual planning strategies—external search via MCTS and internal state tree generation—that significantly boost board game decision-making.
• It demonstrates near Grandmaster-level performance in chess and improved win rates across games by reducing errors like illegal moves and mispredicted states.
• The research paves the way for broader AI applications by enabling language models to autonomously tackle complex, sequential decision-making tasks.

Exploring Board Game Mastery with External and Internal Planning Using LLMs

The paper under discussion explores a novel application of LLMs (LMs) in the context of board games, specifically focusing on enhancing the LLMs' capability to tackle complex, sequential decision-making tasks inherent in board games such as chess, Connect Four, and Hex. The research is anchored in the integration of planning mechanisms within the framework of LLMs by employing techniques of both external and internal searches.

Overview of Approaches

The authors develop two complementary approaches tailored to harness the strength of LLMs in structured planning exercises usually seen in strategic games:

1. External Search: This approach leverages Monte Carlo Tree Search (MCTS) where the LM guides rollouts and evaluations. Instead of using an external engine, the model assesses possible moves and states internally to assist the decision-making process.
2. Internal Search: This unique strategy involves the LLM autonomously generating a tree of potential game states and preferred outcomes. The LM is tasked with directly producing a linearized view of future game possibilities, encouraging an in-context exploration of consequences and strategic considerations.

Both methodologies are grounded in a LLM pre-trained with relevant domain knowledge, which forms the basis for evaluating state transitions and assessing value functions across different games.

Performance and Results

The performance of the proposed methods is substantiated through empirical findings which demonstrate robust improvement in playing strength against existing state-of-the-art bots. Notably, the external search technique achieved near Grandmaster-level proficiency in chess by considering moves similar in count to those analyzed by human Grandmasters. This degree of efficacy is significant given:

• Accurate Decision-Making: Both internal and external search mechanisms notably enhanced win rates; the LM could effectively minimize errors such as legal move violations and state prediction misalignment.
• Robust Pre-Training: The LM's foundational pre-training minimized the occurrence of hallucinations—erroneous predictions disconnected from the true state of the game—bolstering overall decision-making reliability.

The versatility of the testing grounds, spanning games with distinct mechanics and complexities, underscores the general applicability of the approaches and their potential for broader AI applications.

Implications and Future Directions

The research underscores the potential for cross-domain extensions whereby the core search and planning formulation may benefit general LLM inference and training strategies. By excising the dependency on external engines and allowing LMs to autonomously reason through substantial planning depths, the groundwork is laid for more generalized AI systems capable of sophisticated, contextually aware decision-making that mirrors human-like reasoning.

The future trajectory of this research could include:

• Broader Game Coverage: Extending methodologies to accommodate a wider array of board games with varying complexities.
• Interdisciplinary Applications: Real-world problem-solving scenarios that entail complex planning, such as scheduling, logistics, or strategic negotiations, could significantly benefit from the models developed.
• Enhanced Model Architectures: Further refinement in model architectures to improve computational efficiency and strategy assessment at scale.

By showing how LMs can evolve from sequential, text-focused models to proactive agents capable of sophisticated in-game planning and decision facilitation, this paper demonstrates a progressive step forward in the intersection of strategic reasoning and artificial intelligence capabilities.