GTBench: LLM Strategic Reasoning Evaluation

• GTBench is a benchmark that evaluates LLMs' logical and strategic reasoning using 10 diverse game tasks—from Tic-Tac-Toe to negotiation.
• It categorizes challenges by complete versus incomplete information, deterministic versus probabilistic rules, and static versus dynamic play.
• The framework reveals LLM limitations in strict strategy, underscores code-pretraining benefits, and establishes reproducible metrics for future research.

GTBench is a unified, language-driven benchmark designed to rigorously evaluate the strategic and logical reasoning abilities of LLMs in game-theoretic settings. By assembling a diverse suite of 10 well-known board, card, and negotiation games, GTBench systematically assesses LLMs on tasks that emphasize pure logic, strategy, and interaction, isolating these faculties from broader narrative or role-playing skills. Its protocol and error analyses establish standardized metrics, reveal granular limitations in LLM strategic reasoning, and provide a reproducible foundation for further research in this domain (2402.12348).

1. Composition and Design of GTBench

GTBench comprises 10 tasks selected to cover the major axes of game-theoretic evaluation: complete versus incomplete information; deterministic versus probabilistic rules; and static versus dynamic move structures. Notable examples include:

• Complete Information, Deterministic: Tic-Tac-Toe, Connect-4, Breakthrough, Nim.
• Incomplete Information, Probabilistic: Kuhn Poker, Liar’s Dice, Negotiation, Pig.
• Static vs. Dynamic Play: Iterated Prisoner’s Dilemma (simultaneous decisions), versus sequential games with history.

Each game requires LLMs to execute reasoning skills such as numeric calculation (e.g., Nim’s XOR), board analysis (e.g., Connect-4), bluffing (e.g., Liar’s Dice), or negotiation strategies (e.g., labor division for Pareto efficiency). Unlike story-based benchmarks, GTBench's task selection foregrounds deductive and strategic logic over generative language ability.

The benchmark formalizes the evaluation using a multi-turn competition framework, typically pitting LLMs against each other, random baselines, or classical solvers (such as Monte Carlo Tree Search, MCTS).

2. Game-Theoretic Evaluation Methodology

GTBench structures evaluations around the following key distinctions:

• Complete vs. Incomplete Information: Tasks like Tic-Tac-Toe (full board visibility) contrast with Kuhn Poker (hidden cards), allowing analysis of reasoning under certainty and uncertainty.
• Probabilistic vs. Deterministic Rules: Games such as Pig include stochastic die rolls, while Connect-4 is wholly deterministic.
• Static vs. Dynamic Environments: Some tasks (e.g., Iterated Prisoner's Dilemma) are static, requiring simultaneous moves; others are dynamic, allowing models to adapt to evolving state histories.

Performance is quantified with the Normalized Relative Advantage (NRA), a metric that captures comparative performance:

$$\mathrm{NRA}(\mathcal{M}_i, \mathcal{M}_o, f_s) = \frac{\sum_m f_s(\mathcal{M}_i, m) - \sum_m f_s(\mathcal{M}_o, m)}{\sum_m |f_s(\mathcal{M}_i, m)| + \sum_m |f_s(\mathcal{M}_o, m)|}$$

where $\mathcal{M}_i$ and $\mathcal{M}_o$ are the two agents, and $f_s$ denotes the score function for match $m$. The NRA ranges from –1 (total loss) to +1 (total victory), allowing for normalized comparisons across tasks.

3. Performance Analysis of LLMs

Findings derived from GTBench indicate pronounced differences in LLM performance across task types:

• Complete, Deterministic Games: All evaluated LLMs (including GPT-4 and leading open-source models) failed to surpass strong solvers like MCTS, consistently realizing NRA = –1.
• Probabilistic or Incomplete-Information Games: In these settings, LLMs—particularly those with code-pretraining—occasionally achieve NRA values near zero against classical solvers, reflecting approximately competitive performance.
• Model Comparisons: Open-source models such as CodeLlama-34b-Instruct outperformed Llama-2-70b-chat, and matched GPT-3.5-turbo in moderate state spaces, attributing improved strategic ability to code-pretraining. Commercial models (e.g., GPT-4) maintained superiority, particularly in complex scenarios.

These observations underscore that LLM strategic reasoning is strongly context-sensitive, with pronounced deficiencies in environments demanding deterministic, exhaustive logic.

4. Influence of Code-Pretraining and Reasoning Methods

GTBench highlights the differential impact of model training and inference strategies:

• Code-Pretraining: Models exposed to code (e.g., CodeLlama-34b-Instruct) are markedly more adept at tasks requiring logic and structured reasoning, closing much of the gap with commercial LLMs in moderate-complexity games.
• Advanced Reasoning Techniques: Methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) reasoning—frequently effective in complex language tasks—were found to offer little or inconsistent benefit in GTBench. For weaker models, advanced strategies often led to error amplification rather than mitigation. Conversely, for high-performing LLMs, such techniques may provide marginal gains.

A plausible implication is that the introduction of intermediate reasoning steps may introduce compounding errors unless the LLM is already proficient at both task logic and self-reflection.

5. Game-Theoretic Properties and Error Profiling

Beyond raw win/loss records, GTBench quantitatively explores game-theoretic concepts:

• Equilibrium and Pareto Efficiency: In negotiation tasks, outcomes are analyzed for convergence toward Pareto optimality. Some commercial LLMs (e.g., GPT-4) systematically pursue harder bargains, while open-source LLMs more readily accept suboptimal splits.
• Error Typology: The benchmark catalogs frequent error classes, including:
  • Misinterpretation errors (e.g., misreading board state)
  • Factual rule violations (illegal moves)
  • Calculation errors (e.g., arithmetic mistakes in Nim)
  • Over-confidence (optimistic cooperation despite evidence)
  • Endgame misdetection (e.g., failing to identify forced wins/losses)

This systematic error profiling supports targeted diagnosis and informs further LLM training methodologies.

6. Implications and Outlook

GTBench's findings have several ramifications for the design and deployment of LLMs in strategic domains:

• Present-day LLMs remain fundamentally limited in pure logical and strategic reasoning under certainty, suggesting caution in delegating critical decision tasks to such models without supplementary safeguards.
• The efficacy of code-pretraining motivates further research into curriculum design and structured logical datasets as a path to enhancing LLM strategy.
• The limited effectiveness and even potential pitfalls of advanced reasoning strategies point to the need for new inference protocols resilient to compounding errors.
• Open leaderboards and reproducible benchmarks as provided by GTBench are poised to drive iterative progress and fair comparison across the community.

Further, hybridizing LLM outputs with classical search (e.g., MCTS) or incorporating diagnostic feedback from fine-grained error profiles may constitute fruitful directions for advancing LLM capabilities in strategic and competitive environments.

7. Architecture Overview

GTBench is organized around three main components, as illustrated in the original work:

• Environment: Implements game state management for multi-turn competitive play.
• Prompt Adapter: Translates game state into LLM-compatible prompts and back.
• Participant: Manages LLMs or baselines, mediating input/output for each turn.

This modular design facilitates the robust evaluation of a wide spectrum of LLMs and reasoning protocols in a controlled, reproducible multi-agent framework.