---
title: LLM Go Evaluation Benchmark
url: https://www.emergentmind.com/topics/llm-go-evaluation-benchmark
type: topic
---

# LLM Go Evaluation Benchmark

Large Language Model (LLM) Go Evaluation Benchmarks are standardized protocols and datasets designed to rigorously assess the capabilities of LLMs in performing Go-related tasks, including both static next-move prediction and dynamic full-game play. These benchmarks emerged in response to the marked performance gap between general-purpose LLMs and Go-specific AI agents, serving as a platform for quantifiable comparison and targeted advancement of LLM strategic reasoning and gameplay proficiency [2601.16447]. The introduction of holistic evaluation frameworks—such as the KataGo-Bench-1K—has enabled clear separation of reasoning skills, empirical calibration against professional engines and human expertise, and methodical progress tracking in specialized domains.

## 1. Benchmark Structure and Task Formulation

LLM Go Evaluation Benchmarks typically feature two complementary components, each probing different facets of model competence. The static “KataGo-Bench-1K” next-move prediction task presents a sequence of historical moves from human-played 19×19 games, requiring the LLM to forecast the coordinate of the next stone with reference to KataGo—an open-source Monte Carlo Tree Search (MCTS) Go engine. Success is defined as the LLM's predicted move appearing within KataGo’s top-10 candidates for a given board state.

The dynamic full-game match protocol evaluates an LLM by direct competitive play against baseline models and engines under standard Chinese/area scoring rules. The LLM must select moves in an autoregressive fashion, alternating with its opponent, with victory determined by final territory counts. This dual-axis approach distinguishes static recall accuracy (the ability to match professional-level intuition) from dynamic performance (actual win rate against strong adversaries) [2601.16447].

## 2. Dataset Composition and Annotation Procedures

The development of the Go evaluation benchmarks involves curation of multiple large-scale datasets specifically tailored to both training and evaluation:

- **Next-Move Prediction Corpus:** Sourced from 5 million human-played Go records, providing 10 million uniformly sampled mid-game positions. Each position is annotated by KataGo with top-10 candidate moves, per-move win-rate estimates, and one-step rollouts to create robust ground-truth references.
- **Natural Language Commentary Dataset:** Comprises 100,000 human-written commentaries linked to individual board states, emphasizing both strategic context and proper Go terminology.
- **Chain-of-Thought Reasoning Data:** The Go-specific datasets are mixed with general reasoning corpora (including Openthoughts-114K and NuminaMath-QwQ-CoT-5M), yielding a comprehensive set of over 6 million CoT examples in code, math, and broader inference domains.

Each data point is embedded in a heuristic reasoning template, requiring models to (1) identify the next player to act, (2) analyze move candidates with brief strategic variations, (3) summarize the optimal decision, and (4) output a structured move prediction with associated win-rate. This annotation procedure enforces both semantic precision and strategic depth [2601.16447].

## 3. Evaluation Protocols and Metrics

LLM Go benchmarks utilize a standardized suite of metrics and experimental controls:

- **Next-Move Accuracy:** For test set of size \(N = 1000\), the metric is \(\mathrm{Acc} = \frac{1}{N}\sum_i \mathbb{I}(\hat a_i \in M_i)\), where \(\hat a_i\) is the model’s prediction and \(M_i\) the top-10 KataGo moves.
- **Win Rate:** In dynamic matches, \(\mathrm{WinRate} = \frac{\# \mathrm{wins}}{\text{games played}}\).
- **Elo Rating:** Tournament outcomes are mapped to Elo ratings, computed as \(E_A = \frac{1}{1 + 10^{(\mathrm{ELO}(B) - \mathrm{ELO}(A)) / 400}}\) with post-match updates via \( \mathrm{ELO}(A)_{\mathrm{new}} = \mathrm{ELO}(A)_{\mathrm{old}} + K(S_A - E_A) \), using \(K=32\).
- **Correlation:** Elo ratings are empirically validated by their strong correlation (\(r = 0.92\)) with static move recall, ensuring that next-move accuracy is a fair proxy for overall playing strength.

All competitive matches use industry-standard board representations, with carefully sampled queries and strict separation between training and test splits to prevent information leakage [2601.16447].

## 4. Baseline Models and Experimental Results

Benchmarks include both general-purpose LLMs and Go-specialized engines as baselines. Table 1 below summarizes static next-move prediction accuracy (KataGo-Bench-1K):

| Model                       | KataGo-Bench (%) |
|-----------------------------|------------------|
| DeepSeek-R1                 | 17.6             |
| o1-mini                     | 27.3             |
| Claude3.7-Sonnet            | 34.3             |
| Qwen2.5-7B-Instruct         | 8.0              |
| LoGos(7B)                   | 88.1             |
| LoGos(32B)                  | 88.6             |
| KataGo-HumanSL-9d           | 87.8             |

General LLMs—including DeepSeek-R1, Claude-3.7-Sonnet, and Qwen variants—consistently underperform (<35% accuracy) compared to Go-specific engines and LoGos. The LoGos model, trained with mixed Go expertise and broad CoT data followed by Group Relative Policy Optimization, surpasses even top-amateur engines (KataGo-HumanSL-9d) and achieves ≈50% win rate against professional-level artificial opponents. LoGos wins >95% of games against all LLM-shaped baselines [2601.16447].

## 5. Training Methodologies and Integration of Expert Knowledge

Model training leverages a two-stage pipeline:

- **Mixed Cold-Start Supervised Fine-Tuning:** LoGos is trained on 10 million next-move samples, 100K commentaries, and 6M general CoT reasoning corpus. Sequence lengths reach 32,768 tokens; batch sizes up to 512; hardware includes arrays of A800 GPUs.
- **Self-Exploration via Reinforcement Learning:** The Group Relative Policy Optimization (GRPO) algorithm omits simple top-1 rewards in favor of tiered feedback based on both rank and win-rate proximity. Reward function parameters (\(c_1, c_2, c_3, \alpha_1, \alpha_2, \beta_1, \beta_2\)) enforce fine-grained matching with KataGo’s outputs, penalizing deviation from optimal moves but also rewarding strategic exploration.

Ablation studies reveal that omitting Go-specific cold-start, heuristic templates, or using sparse rewards sharply limits achievable accuracy (ceilings ≤67%), indicating the necessity of domain-specific scaffolding for high-fidelity Go reasoning. Rendering board states as 2-D arrays further mitigates long-range context degradation, maintaining accuracy even on deep (>200 move) sequences [2601.16447].

## 6. Interpretative Insights and Forward Directions

The LLM Go evaluation benchmark sets a robust foundation for systematic quantification of reasoning and gameplay in a domain where search-based AI previously set the performance ceiling. Predominantly, the striking delta between LoGos and general LLMs highlights the inadequacy of naïve next-move fitting and the importance of mixed data regimes plus tailored RL. High correlation between static recall and dynamic Elo substantiates the use of KataGo-Bench-1K as a meaningful proxy for true playing strength.

A plausible implication is that rigorous benchmarking in Go—encompassing reasoning templates, diverse human-played positions, and fine-grained evaluation—can inform development of LLMs in other combinatorial games and decision sciences. The benchmark’s release opens avenues for further research on natural-language reasoning, transfer learning, and game-specific self-optimization protocols for LLMs [2601.16447].

Source: https://www.emergentmind.com/topics/llm-go-evaluation-benchmark