---
title: 'Kaggle Game Arena: Evaluate Large Language Models'
url: https://www.emergentmind.com/topics/kaggle-game-arena
type: topic
---

# Kaggle Game Arena: Evaluate Large Language Models

Kaggle Game Arena is an open, evolving platform developed by Kaggle, Google DeepMind, and Google Cloud for evaluating large language models (LLMs) through repeated competitive gameplay. Rather than measuring performance on a fixed collection of questions, it places models in structured environments whose outcomes are determined by game rules, opponent behavior, uncertainty, and long-horizon interaction. The initial pilot environments—Chess, Poker, and Werewolf—cover perfect-information, imperfect-information, and multiplayer settings, enabling analysis of planning, adaptation, risk management, deception, communication, cooperation, and robustness under uncertainty [2609.31473].

## 1. Concept and motivation

Game Arena is based on the premise that strategic competence is better assessed through interactive policies than through isolated question answering. A static benchmark evaluates a fixed mapping from prompt to answer, whereas Game Arena evaluates a sequential process:

$$
\text{state}_t,\text{history}_t \longrightarrow \text{action}_t \longrightarrow \text{new state}_{t+1}.
$$

The opponent’s behavior changes the future test distribution. Consequently, a stronger model does not merely receive a more difficult static item; it faces stronger and more adaptive opponents.

Static benchmarks such as MMLU, GSM8K, and HellaSwag are useful for knowledge, mathematics, and commonsense reasoning, but they are vulnerable to performance saturation and contamination. Once frontier models approach the ceiling, a benchmark may cease to distinguish them reliably. Fixed test items may also leak into model training data.

Dynamic preference benchmarks such as Chatbot Arena and MT-Bench reduce saturation through pairwise comparison, but their judgments are subjective. Human or LLM evaluators may respond to verbosity, style, sycophancy, confidence, or other presentation effects rather than reasoning quality. Game Arena instead uses ground-truth outcomes, including wins, losses, draws, chip returns, votes, and team victories.

The platform is characterized as:

- **Dynamic**: gameplay trajectories evolve as models and opponents improve.
- **Interactive**: decisions depend on state, history, and opponent behavior.
- **Outcome-grounded**: results are determined by game rules rather than subjective judges.
- **Evergreen**: new games, variants, models, and evaluation procedures can be added over time.
- **Diagnostic**: complete trajectories, reasoning traces, invalid actions, role-specific outcomes, and temporal performance are retained.

The framework should not be interpreted as producing a universal scalar measure of intelligence. The pilot results show that model rankings differ substantially across games. Gemini 3 Pro is strongest in Chess and Werewolf but loses money in Poker; GPT-5.2 is strongest in Poker and competitive in Chess but ranks below Gemini models in Werewolf; Grok 4 is a strong poker performer but weaker in Chess and Werewolf. These differences support the interpretation that strategic ability is multidimensional.

## 2. Platform architecture and evaluation harness

Each Game Arena environment has two principal components. A declarative JSON specification defines the model’s observations, permitted actions, rewards, outcome structure, and general configuration. A Python interpreter implements game mechanics, validates actions, updates state, enforces information visibility, and determines termination and outcomes. The environments are adapted from, or compatible with, Gymnasium, PettingZoo, and OpenSpiel; some are developed specifically for Kaggle competitions.

At every decision point, a model receives a natural-language representation of the current state, relevant public and private history, its role or player identity, the available objective, and a required output format. It returns an action. Invalid outputs trigger limited retries or a defined fallback.

The harness records state snapshots, model actions, explanations or reasoning traces when required, visibility and information state, outcomes and rewards, retries, and parsing failures. Every turn produces an immutable state snapshot, and these snapshots form a complete episodic trajectory serialized as JSON. This supports replay, auditing, error analysis, and retrospective evaluation.

The platform includes a TypeScript/React/Canvas visualization layer with game-specific renderers, timeline navigation, playback controls, event inspection, a reasoning panel displaying model explanations, and streaming or condensed trace views. The environments, harnesses, prompt templates, evaluation scripts, and full gameplay datasets are open-sourced and hosted on Kaggle.

All pilot environments use a common competitive protocol:

1. Models enter a full round-robin.
2. Every model pair is evaluated under standardized conditions.
3. The harness supplies structured state and history.
4. The model produces an action in a prescribed format.
5. Invalid actions receive a limited number of retries or a defined fallback.
6. Outcomes are logged at the finest available granularity.
7. Game-specific metrics are computed.
8. Bootstrap confidence intervals are reported.

The evaluation scales differ by environment: Chess uses 40 games per model pair, Poker uses 20,000 hands per pair, and the reported Werewolf evaluation contains approximately 31,472 games. Large sample sizes and variance-reduction procedures are intended to make pairwise comparisons more reliable.

| Environment | Information structure | Players | Primary metric |
|---|---|---:|---|
| Chess | Perfect, deterministic | 2 | Elo / Bradley–Terry |
| Poker | Imperfect, stochastic | 2 | BB/100 |
| Werewolf | Imperfect, asymmetric, stochastic | 8 | Game-theoretic evaluation |

## 3. Chess: planning, legality, and long-horizon control

Chess is implemented as a deterministic, two-player, perfect-information game following standard FIDE rules, including legal movement, pawn captures, castling, the 50-move rule, check, checkmate, and other standard termination rules. The environment records games in PGN.

At each turn, a model receives the board in Forsyth–Edwards Notation (FEN), the complete move history in PGN, its color, and a request for its strongest legal move. It must return a move in Standard Algebraic Notation (SAN). The harness does not provide a list of legal actions, so the model must infer legality independently. An invalid move causes a retry. After the retry budget is exhausted, the game is scored as a loss. These retries are called rethinks.

Each model pair plays 40 games: 20 with one model as White and 20 with that model as Black. The color-balanced design reduces first-move effects. The Chess Opening variant initializes games from one of 20 popular two-ply openings drawn from the Lichess database rather than from the standard initial position. It tests adaptation to varied opening structures and whether rankings depend excessively on familiar opening patterns.

Chess primarily probes long-horizon planning, search-like reasoning, legality and state tracking, tactical calculation, strategic positional judgment, adaptation as positions become more complex, and reliability of structured action output.

### Rating methodology

The paper fits a Bradley–Terry model to pairwise results, treating a draw as 0.5. The probability that model \(i\) defeats model \(j\) is:

$$
P(i \succ j)=
\frac{\exp(\theta_i)}
{\exp(\theta_i)+\exp(\theta_j)}
=
\frac{1}{1+\exp(\theta_j-\theta_i)},
$$

where \(\theta_i\) and \(\theta_j\) are latent model strengths. The reported Elo values are an affine representation of these relative strengths. Since ratings are identifiable only up to an additive constant, the lowest-rated model is assigned 0. The paper reports 95% bootstrap confidence intervals.

The Chess Text results are:

| Model | Elo | 95% confidence interval |
|---|---:|---:|
| Gemini 3 Pro Preview | 1325 | 1217–1428 |
| Gemini 3 Flash Preview | 1297 | 1205–1405 |
| o3 | 1009 | 928–1107 |
| GPT-5.2 | 933 | 849–1018 |
| Grok 4 | 773 | 701–868 |
| Grok 4.1 Fast Reasoning | 632 | 556–725 |
| GPT-5 mini | 525 | 425–603 |
| Claude Opus 4.5 | 236 | 163–323 |
| Claude Sonnet 4.5 | 189 | 124–277 |
| Claude Haiku 4.5 | 122 | 50–211 |
| DeepSeek V3.2 | 0 | 0–0 |

The Chess Opening ratings preserve essentially the same tier structure. Gemini 3 Flash Preview and Gemini 3 Pro Preview remain at the top, followed by o3 and GPT-5.2, with the remaining models occupying lower tiers. The reported differences between Chess Text and Chess Opening are not significant relative to the gaps between tiers.

### Temporal performance and failure modes

Stockfish is used to analyze where games diverge. Engine evaluations are converted into approximate win probabilities and averaged by normalized game progress. Models are relatively similar in the opening, while performance increasingly diverges in the middlegame and endgame. Strong models accumulate positional advantages, whereas weak models show a steady decline.

Rethink behavior provides a separate reliability measure. Gemini models produce fewer than approximately 0.2 rethinks per game, while GPT-5.2 is generally below one rethink per game. Grok 4, Grok 4.1 Fast Reasoning, and o3 produce approximately 0.5–1.7 rethinks per game. DeepSeek V3.2 and Claude Haiku 4.5 often produce more than five and sometimes more than ten rethinks per game. Weak-model errors are concentrated later in games; one common failure is proposing a move while in check that does not resolve the check.

## 4. Poker: uncertainty, adaptation, and risk management

Poker is heads-up no-limit Texas Hold’em, an imperfect-information, stochastic game. The configuration uses two players, 1–2 blinds, a 100-big-blind starting stack, stack resets after every hand, private hole cards, public community cards, betting history, no communication channel beyond actions, and a 60-minute action limit.

The model receives the position, chip stacks, hole cards, community cards, betting history, and prior episode history. The system prompt instructs models to maximize expected value, use game-theoretic-optimal play as a baseline, deviate from GTO when exploiting opponent tendencies, and explain decisions using range advantage, pot odds, fold equity, position, and related concepts.

Matchups are divided into independent 100-hand episodes. The complete history of earlier hands is available, and both players’ hole cards are revealed after each hand, including hands without a showdown. Models can therefore update beliefs about opponent tendencies.

To reduce variance, the evaluation uses a duplicate or hand-mirroring format. A 100-hand sequence is played and then replayed with player seats and cards swapped. Each pair plays:

$$
20{,}000\text{ hands}
=
10{,}000\text{ individual deals}\times2.
$$

With ten models, there are 45 pairwise matchups and 900,000 hands overall, or approximately 180,000 hands per model.

The primary metric is big blinds won per 100 hands:

$$
\mathrm{BB/100}
=
100\cdot
\frac{\text{net big blinds won}}
{\text{number of hands}}.
$$

Positive values indicate profitability relative to the opponent, while negative values indicate losses. Confidence intervals are obtained using a block bootstrap over episodes, accounting for correlation among hands within a 100-hand episode.

| Model | Hands | BB/100 |
|---|---:|---:|
| GPT-5.2 | 180,000 | 46.56 |
| o3 | 180,000 | 29.69 |
| Grok 4 | 180,000 | 27.11 |
| Claude Opus 4.5 | 179,936 | 17.62 |
| Claude Sonnet 4.5 | 180,000 | 12.83 |
| Gemini 3 Flash Preview | 180,000 | 5.49 |
| DeepSeek V3.2 | 179,936 | -10.05 |
| Gemini 3 Pro Preview | 180,000 | -15.17 |
| Grok 4.1 Fast Reasoning | 180,000 | -19.19 |
| GPT-5 mini | 180,000 | -94.87 |

GPT-5.2 is the only model with an advantage over every opponent, winning between 57% and 76% of matchups. o3 wins or draws eight of nine pairings, losing to GPT-5.2 with a 42% win rate and splitting with Claude Sonnet at 50%. Grok 4 achieves the most dominant individual result against GPT-5 mini, winning 90% of those games, but has a winning record in only three of nine pairings.

The results show that profitability is not reducible to aggression. GPT-5 mini opens 98.3% of button hands but loses \(-94.87\) BB/100, whereas GPT-5.2 opens 91.7% and earns \(+46.56\) BB/100. Grok 4 uses an aggressive defense, including a 56.7% three-bet rate against button opens, and performs particularly well against weaker opponents. Claude models are more call-oriented but remain profitable. Gemini 3 Pro is unusually conservative, with a 52.8% button-opening rate, a 43.8% big-blind fold rate, and a 6.0% three-bet rate; the paper interprets this pattern as excessive passivity.

Poker consequently evaluates belief updating, expected-value reasoning, opponent modeling, risk management, adaptation over repeated interaction, and robustness to stochastic outcomes.

## 5. Werewolf: asymmetric information and multi-agent evaluation

Werewolf is an eight-player, general-sum, multiplayer social-deduction game with imperfect and asymmetric information, stochastic role assignment, natural-language communication, team incentives, deception, and coalition formation. The roles are two Werewolves, one Seer, one Doctor, and four Villagers.

The Werewolves know one another, while other players initially do not know the identities of the remaining players. The Werewolves win when their number reaches parity with the Village; the Village wins by eliminating all Werewolves.

The game alternates between private Night phases and public Day phases. At Night, the Doctor protects one player, subject to no self-save and no consecutive protection of the same player. The Seer investigates one player and learns that player’s role. Werewolves vote sequentially to select an elimination target; the second Werewolf sees the first Werewolf’s vote, the first voter rotates between nights, and ties cause no elimination.

During the Day, the moderator announces deaths, eliminated roles are revealed, survivors debate publicly, and players vote sequentially to exile a player. Discussion uses a round-robin protocol of at most two cycles, while speaking and voting orders rotate. Ties cause no exile.

### Information control and interaction protocol

Werewolf uses an append-only event log in which every event includes visibility permissions. A centralized event bus filters events and dispatches only information available to each player. The prompt uses a ReAct-style format containing the system rule and role description, public state, private role-specific information, chronological memory, and phase-specific task instructions.

Models return JSON containing private reasoning and a public action or message. The harness includes lenient JSON5 parsing, exponential-backoff retries, context truncation preserving the latest 75% on overflow, a deterministic inappropriate-language filter, and turn forfeiture after repeated parsing failure.

### Rule balancing

Standard Werewolf rules are highly favorable to the Village because a Seer can reveal their identity and the Doctor can repeatedly protect them. Using 500 Gemini 3 Pro self-play games per configuration, the reported Village win rates are:

| Configuration | Village win rate | 95% uncertainty |
|---|---:|---:|
| Standard | 73.4% | ±3.9% |
| No Doctor self-save | 72.4% | ±3.9% |
| Fixed order and no tie break | 60.6% | ±4.3% |
| Balanced configuration | 56.7% | ±4.3% |

The official configuration combines no Doctor self-save, no consecutive Doctor saves, peaceful ties with no elimination, and rotating speaking and voting order. These results are an empirical balance proxy rather than a formal game-theoretic equilibrium: they depend on the model and strategy profile used in self-play.

### Game-Theoretic Evaluation

Raw individual win rates confound model ability with role assignment. Game-Theoretic Evaluation (GTE) addresses this problem through a model-versus-model-versus-role meta-game. For model \(m\) and role \(r\), the empirical role-specific win rate is:

$$
v(m,r)
=
\frac{\text{wins by model }m\text{ in role }r}
{\text{games played by model }m\text{ in role }r}.
$$

For models \(A\) and \(B\), the role-specific payoff is:

$$
u_A(r,A,B)=v(A,r)-v(B,r),
$$

$$
u_B(r,A,B)=-u_A(r,A,B),
$$

$$
u_{\text{role}}(r,A,B)=|u_A(r,A,B)|.
$$

A maximum-entropy correlated equilibrium is used rather than a single Nash equilibrium. This spreads probability across models and roles, reducing dependence on a single noisy role estimate. Uncertainty is incorporated through bootstrap resampling of approximately 31,472 games; the correlated equilibrium is recomputed for 1,000 bootstrap samples, with the 2.5th and 97.5th percentiles forming the 95% confidence interval.

| Model | Equilibrium rating | 95% confidence interval |
|---|---:|---:|
| Gemini 3 Pro Preview | 0.10% | -0.01% to 0.21% |
| Gemini 3 Flash Preview | -0.36% | -0.67% to -0.05% |
| GPT-5.2 | -1.12% | -1.72% to -0.52% |
| Claude Opus 4.5 | -1.31% | -1.98% to -0.64% |
| Claude Sonnet 4.5 | -3.67% | -4.33% to -3.01% |
| Grok 4.1 Fast Reasoning | -5.23% | -6.05% to -4.41% |
| Grok 4 | -5.28% | -6.17% to -4.39% |
| GPT-5 mini | -6.78% | -7.93% to -5.63% |

Gemini 3 Pro and Gemini 3 Flash are the strongest overall models. Claude Sonnet and Grok 4.1 Fast Reasoning have especially negative Seer contributions, suggesting difficulty communicating private information without becoming suspicious. GPT-5 mini is weak across essentially all roles.

The full tournament graph is acyclic, producing a clean dominance hierarchy. Across 1,000 bootstrap samples of all 31,472 games, the average number of cycles is 0.047; when each sample is reduced to 1,000 games, the average rises to 2.048 cycles. This demonstrates the effect of sample size on ranking stability.

The platform also evaluates cooperation. For coalition \(C\), its characteristic value is:

$$
V(C)
=
\frac{1}{|I_C|}
\sum_{i\in I_C}v_i,
$$

where \(I_C\) is the set of games containing the coalition and \(v_i\) indicates whether the coalition won. The substitution index measures the effect of replacing model \(a\) with model \(b\):

$$
S(a,b)
=
\frac{1}{|\mathcal C_a|}
\sum_{C\in\mathcal C_a}
\left[
V(C)
-
V\left((C\setminus\{a\})\cup\{b\}\right)
\right].
$$

Replacing another model with Gemini 3 Pro generally improves coalition performance. Gemini 3 Flash and GPT-5.2 are also strong teammates, whereas GPT-5 mini, Grok 4, and Grok 4.1 Fast Reasoning are generally weak teammates.

## 6. Statistical interpretation, limitations, and future development

Game Arena’s central methodological advantage is the combination of standardized environments, repeated head-to-head interaction, large samples, game-specific ground truth, and detailed trajectories. Its principal metrics are adapted to each game:

- **Chess** uses Elo and Bradley–Terry modeling because outcomes are deterministic, paired, and zero-sum.
- **Poker** uses BB/100 because profitability is naturally expressed through chip returns under stochastic imperfect information.
- **Werewolf** uses role-aware GTE because raw win rates confound skill with asymmetric role assignment and team structure.

This prevents a single superficial metric from dominating all environments. The same model can be strong in one game and weak in another, providing a multidimensional profile rather than a universal rank.

The framework also supports temporal and procedural analysis. Chess trajectories reveal where positional advantages emerge. Poker provides repeated hands and revealed hole cards for opponent adaptation. Werewolf records private and public information, communication, role assignments, and coalition outcomes. These records can support evaluation of state tracking, invalid actions, strategic adaptation, communication, deception, and cooperation.

Several limitations remain. Fixed match counts are computationally inefficient, and adaptive scheduling is needed to allocate more games to close matchups. Model churn can fragment the comparison graph when models are added or retired. Cold-start evaluation and longitudinal comparability remain open problems. The platform has no consolidated cross-game rating, and multiplayer environments create difficult credit-assignment and team-level aggregation problems.

Ground-truth outcomes do not eliminate every confound. Chess performance is affected by action formatting and legality; Poker results contain stochastic variance despite duplicate dealing; Werewolf outcomes depend on role assignment, coalition structure, communication, and rule balance. Consequently, objective game outcomes should not be equated with a complete measurement of reasoning. They are observable consequences of policy behavior under specified environments.

The pilot results also show why heuristic social-deduction metrics are insufficient as primary scores. Key Survival Rate often approaches 0.5 and penalizes strategically valuable sacrifices. Identification Precision is approximately 0.65 for much of the model pool. Voting Success Score approaches 1.0 because players often converge on public consensus, even when private reasoning differs. A model may identify a Werewolf privately but vote differently to preserve its cover. Role-aware evaluation is therefore necessary.

Game Arena is designed to resist saturation through head-to-head competition, opponent co-evolution, large stochastic interaction spaces, long-horizon play, new game variants, fresh trajectories, and rule-enforced outcomes. However, it does not make contamination impossible. Public games, prompts, model outputs, evaluation scripts, and datasets may eventually enter training corpora. Continued usefulness depends on adding environments, changing configurations, maintaining hidden or refreshed evaluation conditions, and preserving statistical comparability over time.

The infrastructure supports several future directions:

- **Expanded game coverage**: additional deterministic board games, imperfect-information games, economic games, negotiation environments, and multiplayer social settings.
- **Adaptive tournament scheduling**: allocate simulation budget where rankings are uncertain or pairwise differences are small.
- **Cross-game analysis**: distinguish general strategic competence from game-specific specialization.
- **Cost-performance evaluation**: compare strategic performance with inference cost per turn or per game and identify Pareto-efficient models.
- **Human and agent interaction**: evaluate models against human players, fixed agents, learned agents, or evolving populations.
- **Longitudinal benchmarking**: track capability changes while preserving links among old and new models.
- **Trajectory-based research**: analyze reasoning traces, invalid actions, temporal degradation, role-specific behavior, and strategic adaptation.

Game Arena therefore constitutes both an evaluation platform and a research infrastructure for interactive model assessment. Its defining contribution is not simply a leaderboard but a combination of evolving head-to-head competition, declarative environment specifications, executable game interpreters, immutable trajectory logging, game-specific statistical models, variance reduction, and extensibility to new strategic environments. The resulting measurements are most informative when treated as complementary capability profiles rather than as a single universal ranking.

Source: https://www.emergentmind.com/topics/kaggle-game-arena