ChessArena: Testbed for LLM Strategic Reasoning

• ChessArena is a competitive testbed that assesses LLMs' strategic reasoning using chess, focusing on long-term planning and multi-turn memory.
• It offers diverse play modes—Bullet, Blitz, Standard, and Blindfold—that impose varying constraints from immediate move selection to chain-of-thought reasoning.
• The framework integrates advanced evaluation metrics and fine-tuning techniques, including GRPO-based reinforcement learning, to improve rule adherence and decision quality.

ChessArena is a competitive framework for evaluating the strategic reasoning capabilities of LLMs through chess gameplay and fine-grained reasoning tasks. As presented in (Liu et al., 29 Sep 2025), its design addresses whether LLMs possess genuine strategic reasoning skills or simply excel at pattern recognition learned from large datasets. Chess—requiring long-term planning, strict rule following, and multi-turn conversation memory—serves as a rigorous domain for probing these capabilities. ChessArena implements multiple play modes, specialized ranking algorithms, and a public leaderboard, collectively enabling the nuanced assessment of over a dozen LLMs across more than 800 games.

1. Conceptual Foundations and Objectives

ChessArena was established to rigorously test LLMs’ ability to manifest complex strategic reasoning in dynamic, adversarial, multi-turn situations. Unlike tasks solvable via shallow pattern matching, chess demands not only legal move prediction but also foresight, adaptation, and persistent memory across many turns. ChessArena is therefore constructed to measure:

• Long-term strategic planning
• Rule comprehension (e.g., legality of moves, check/checkmate enforcement)
• Multi-turn conversation memory, especially in modes where the board state must be reconstructed internally
• Real-time decision making with or without explicit chain-of-thought reasoning

The testbed includes both holistic game evaluation and fine-grained reasoning challenges (basic rule understanding, move selection, puzzle solving).

2. Platform Structure: Play Modes and Ranking System

ChessArena supports four distinct play modes to dissect various dimensions of reasoning:

Mode	Input Provided	Output Required	Reasoning Constraint
Bullet	Board state (FEN/UCI)	Move only	No chain-of-thought
Blitz	Board state	Move; chain-of-thought optional	Fast reasoning allowed
Standard	Board state	Move plus chain-of-thought	Explicit chain-of-thought
Blindfold	Move sequence history only	Move (board must be reconstructed internally)	Memory dependence

Each mode probes different cognitive aspects: Bullet tests immediate decision-making, Blitz and Standard assess deliberation and explicit reasoning, and Blindfold mode tests multi-turn memory and board reconstruction.

Rankings are computed using the Glicko system, a Bayesian extension of Elo that provides both a skill rating (r) and rating deviation (RD) to capture uncertainty. After a minimum of 30 games per participant, rating stabilization is achieved, and models are displayed on a public leaderboard. Rating updates use formulas of the form

$$r' = r + \left(\frac{q}{1/\mathrm{RD}^2 + 1/d^2}\right) \, g(\mathrm{RD}_o) \left(s-E(s|r, r_o, \mathrm{RD}_o)\right)$$

where $s$ is the match outcome, $r_o$ is the opponent’s rating, and $g$, $q$, $d$ are system parameters.

3. Fine-Grained Reasoning: Rule, Move, and Puzzle Tests

ChessArena employs targeted evaluation tasks beyond direct gameplay:

• Basic Understanding: Given a FEN representation and a queried square, models must (a) identify the correct piece and (b) enumerate all legal moves. Metrics include Piece Match Accuracy (PMA), Precision, and Recall, even under board perturbations like empty squares or turn mismatches.
• Move Selection: For a given board, models select one move. Metrics:
  • Legal Rate (LR): Percent of outputs that are legal chess moves.
  • Top Rate (TR): Frequency of selecting a move within Stockfish’s top-three recommendations.
  • Move Advantage Rate (MAR):

  $$\mathrm{MAR} = \frac{1}{N}\sum_{i=1}^N \frac{Q(\text{FEN}_i, \text{Move}_\mathrm{pred}) - \mathrm{AWR}_i}{\mathrm{AWR}_i}$$

  where $Q$ is Stockfish’s win probability for the predicted move and AWR is the average over all legal moves in the position.

• Puzzle Solving: Sequence recognition—match the exact moves of a curated chess composition or tactical problem. Metric: Puzzle Solving Accuracy (PSA).

4. Evaluation Results and Shortcomings of Current LLMs

Evaluation across >13 LLMs and >800 games yielded several critical findings:

• No model defeated Maia-1100 (an engine approximating amateur human skill at 1600 Elo).

• Multiple LLMs failed to consistently outperform a random move generator.

• Deficiencies included:

  • Insufficient follow-through on instructions (e.g., output formatting errors)
  • Tactical ineptitude; predicted moves were often suboptimal compared to legal-move baselines
  • Poor multi-turn coherence, especially pronounced in Blindfold mode
  • Subpar performance in puzzle solving (most LLMs ≤15%, whereas O3 reached ~55.6%)

Aggregate metrics such as MAR and PSA consistently trailed behind human-calibrated engines.

5. Baseline Model Improvements via Fine-Tuning

To establish progress potentials, the Qwen3-8B model served as a baseline for targeted enhancement:

• Initial performance positioned Qwen3-8B at the leaderboard’s bottom.
• Supervised fine-tuning using domain-specific chess dialogue data led to substantial performance gains.
• Final improvement leveraged reinforcement learning with Stockfish-derived rewards ("format reward," "legal move reward," "top move reward"), combined as a weighted sum, via the Group Relative Policy Optimization (GRPO) technique.
• The fine-tuned Qwen3-8B-Chess achieved competitive ratings in Blitz mode, outperforming its untuned version, and demonstrated ability to generate more compliant and strategically sound moves.

This empirical progression suggests that high-quality training on domain-specific data—especially incorporating rule and strategy compliance via machine evaluation—can significantly boost strategic reasoning for LLMs.

6. Implications for LLM Research and Future Directions

ChessArena’s findings indicate serious limitations in current LLM strategic reasoning, particularly for multi-step, rule-intensive adversarial domains:

• LLMs exhibit robust pattern recognition but insufficient strategic planning, tactical calculation, and consistency over multiple turns.
• Post-training with high-quality domain examples and reinforcement using expert engine feedback can yield measurable improvements.
• The work suggests promising research avenues:
  • Exploring continued pre-training strategies to further reduce reliance on external legal move lists.
  • Enhancing chain-of-thought and multi-turn memory, especially for internally reconstructed game states.
  • Leveraging strategic reasoning improvements to benefit other domains—mathematical problem solving, planning, code synthesis, etc.—as evidenced by observed gains in AIME2025 and ZebraLogic logical reasoning benchmarks.

The ChessArena framework offers a template for future developments. By integrating competitive gameplay and refined reasoning metrics, it sets a rigorous standard for LLM evaluation and improvement in complex sequential domains.

7. Technical Summary Table

Component	Description	Key Metric/Formulas
Play Modes	Bullet, Blitz, Standard, Blindfold	Board inputs, reasoning styles
Leaderboard	Glicko two-parameter rating system	$r$, RD, Bayesian update formula
Fine-Grained Tests	Basic understanding, move selection, puzzles	PMA, LR, TR, MAR, PSA
MAR Sample Formula	Move quality relative to mean	$$\mathrm{MAR} = (1/N) \sum [Q - \mathrm{AWR}]/\mathrm{AWR}$$
Baseline Model Steps	Qwen3-8B → SFT → GRPO fine-tuning	Weighted engine-based reward signals

ChessArena is a robust test environment that exposes the distinction between pattern recognition and genuine strategic reasoning in LLMs, inviting focused research efforts to narrow this gap and set benchmarks for progress.