Game Arena: Strategic LLM Evaluation in Competitive Environments

Abstract: We introduce Kaggle Game Arena, an open and ever-expanding platform to evaluate LLMs through competitive games. Different from static benchmarks, game arena enables models to play head-to-head matchups in structured environments where the gameplay strength naturally increases as models evolve, preventing performance saturation. This technical report details the infrastructure behind Game Arena and describes the three pilot game environments: Chess, Poker, and Werewolf. These environments span perfect information, imperfect information, and multiplayer game settings, enabling a systematic study of models' strategic planning, adaptation, and robustness under uncertainty. For each game, we provide a detailed description of the environment, evaluation metrics, and results from running full competitions across models. Through robust infrastructure and large-scale ground-truth based evaluation, Game Arena ensures reproducibility, transparency and generalizability to new games and variants over time.

• The paper proposes 'Game Arena,' a novel benchmark using competitive gaming to evaluate LLMs, reducing subjectivity and providing objective outcomes.
• The benchmark tests three environments: Chess, Poker, and Werewolf, each revealing distinct strategic competencies, with Gemini 3 Pro Preview dominating Chess and Werewolf while o3 and GPT-5.2 perform notably in Poker.
• The study finds that models' competitive abilities are not uniformly transferable across different gameplay scenarios and game rules must be validated to avoid exploiting deterministic mechanics while emphasizing deception and deductive reasoning.

Evaluation problem and central contribution

“Game Arena: Strategic LLM Evaluation in Competitive Environments” (2609.31473) proposes an open, continuously extensible benchmark for evaluating LLMs through competitive gameplay rather than static question answering or subjective preference judgments. Its central claim is that games provide a favorable evaluation substrate because they generate adaptive trajectories, expose models to increasingly capable opponents, and produce objective outcomes such as wins, losses, draws, and chip returns. The benchmark is therefore designed to reduce both saturation and evaluator subjectivity, while retaining sufficient structure for statistical inference and behavioral diagnosis.

The paper positions Game Arena between several existing evaluation paradigms. Static benchmarks are vulnerable to saturation and contamination, while human-preference and LLM-as-a-judge evaluations introduce variance from annotator inconsistency, verbosity bias, and sycophancy. Prior game-based benchmarks—including AgentBench (Liu et al., 2023), GTBench (Duan et al., 2024), SmartPlay (Wu et al., 2023), and Game Reasoning Arena (Cipolina-Kun et al., 5 Aug 2025)—demonstrate the value of interactive evaluation but generally operate as fixed datasets or isolated experiments. Game Arena instead emphasizes an evergreen tournament infrastructure: new models, games, variants, and evaluation procedures can be integrated without replacing the underlying harness.

The empirical release spans three strategically distinct environments: Chess, heads-up no-limit Texas Hold’em, and eight-player Werewolf. These environments vary along information structure, stochasticity, number of players, and communication requirements. Chess tests deterministic planning and search under perfect information; Poker tests belief updating, risk sensitivity, and opponent modeling under imperfect information; Werewolf tests asymmetric information, deception, coalition formation, and natural-language social inference.

Infrastructure and common protocol

The benchmark uses a uniform text-based interaction protocol. At each decision point, a model receives a natural-language representation of the current state and history and must produce an action in a prescribed format. The environment validates the action, permits a small number of retries, updates the game state, and records the resulting trajectory. Invalid actions are not treated merely as formatting artifacts: persistent failure produces a loss in Chess, a conservative default action in Poker, and a forfeited turn in Werewolf.

Each environment is implemented through a declarative specification and an executable interpreter. The specification defines observations, action spaces, and reward semantics; the interpreter enforces rules, validates actions, updates state, and determines termination. The design incorporates components from Gymnasium (Towers et al., 2024), PettingZoo (Terry et al., 2020), and OpenSpiel (Lanctot et al., 2019), while supporting game-specific implementations. Every turn yields an immutable state snapshot containing the state, action, outcome-relevant metadata, and available reasoning trace. These records enable replay, visualization, bootstrap analysis, and post-hoc behavioral diagnostics.

Figure 1: Game Arena infrastructure for specifying, executing, recording, and visualizing competitive LLM interactions.

The evaluation protocol has three important statistical properties. First, each model pair participates in a full round-robin under matched conditions. Second, the benchmark uses environment-appropriate metrics rather than forcing all games into a single win-rate statistic. Third, uncertainty is estimated through bootstrap procedures, with resampling units chosen to reflect the dependence structure of the data. These choices are particularly important for Poker, where individual hands are highly variable, and Werewolf, where role assignment and multiplayer interactions confound raw win rates.

Chess: planning quality emerges after the opening

The Chess environment follows standard FIDE rules and represents positions using FEN together with the complete PGN move history. Models must return legal SAN moves without being given the legal-move list. Each pair plays 40 games, balanced across colors. A model that remains illegal after the permitted retries loses the game.

The benchmark includes both a free-form Chess Text condition and a Chess Opening condition. In the latter, games begin from one of 20 popular two-ply openings sampled from Lichess data. This variant tests whether performance depends on repeatedly selecting a familiar opening sequence rather than on downstream position evaluation.

The Chess Text leaderboard is sharply stratified. Gemini 3 Pro Preview obtains an internal Elo of 1325, followed by Gemini 3 Flash Preview at 1297, o3 at 1009, and GPT-5.2 at 933. Grok 4, Grok 4.1 Fast Reasoning, and GPT-5 mini form a substantially weaker tier at 773, 632, and 525, respectively. The Claude 4.5 variants occupy the bottom of the ranking, with Opus at 236, Sonnet at 189, and Haiku at 122; DeepSeek V3.2 is used as the zero-point anchor. The confidence intervals remain sufficiently separated across the main tiers to support a strong ranking claim, although the absolute Elo scale is internal and its Stockfish-based external calibration is explicitly less reliable outside the engine calibration range.

Figure 2: Pairwise Chess Text outcomes and bootstrapped win-rate distributions, showing a dominant Gemini tier and substantial separation from lower-performing models.

The Chess Opening condition preserves the same broad ordering. Gemini 3 Flash Preview leads at 1263 Elo, narrowly ahead of Gemini 3 Pro Preview at 1253; o3 and GPT-5.2 follow at 934 and 889. The stability of the ranking under randomized opening positions is important because it weakens the alternative explanation that the Chess Text results primarily reflect opening-repertoire familiarity. However, the opening intervention changes only the initial two plies, so it does not isolate all forms of training-data memorization or opening preparation.

The paper’s Stockfish analysis localizes the performance gap temporally. Models are approximately balanced during the opening, after which their trajectories diverge in the middlegame. Strong models gradually accumulate positional advantage and convert it in the endgame, whereas weaker models show a sustained decline in engine-estimated win probability.

Figure 3: Stockfish-derived win-probability trajectories showing that model differences accumulate mainly in the middlegame and endgame.

This result implies that the benchmark is measuring more than opening selection. The principal separation appears in maintaining and improving a position as tactical and strategic complexity increases. Gemini 3 Pro Preview’s advantage expands against nearly every opponent and approaches near-certain victory in the late stages, with Gemini 3 Flash Preview the principal exception.

The retry analysis provides an independent reliability signal. Gemini models require fewer than 0.2 rethinks per game on average, whereas mid-tier models typically require approximately 0.5–1.7. DeepSeek V3.2 and Claude Haiku 4.5 often require more than five and sometimes more than ten retries. Retries are concentrated later in games, with only approximately 10–20% occurring in the first half for models with high retry rates. This pattern is consistent with a failure mode in which models can reproduce standard opening structures but become unable to maintain legality and board-state consistency in complex middlegame and endgame positions.

Figure 4: Chess Text rethink behavior, showing that weaker models incur more retries and that retries concentrate after the opening phase.

Poker: large-scale variance control exposes cross-model reversals

The Poker environment evaluates heads-up no-limit Texas Hold’em with 100-big-blind stacks and 1–2 blinds. Models receive private cards, public cards, positions, stacks, and betting history. Each matchup comprises 20,000 hands organized into 100-hand episodes. The benchmark uses duplicate poker: the same deal sequence is replayed with seats and cards mirrored, reducing luck-induced variance while preserving strategic divergence after players choose different actions.

The scale is unusually large for LLM poker evaluation. A ten-model round robin produces approximately 900,000 hands, or about 180,000 hands per model. This is orders of magnitude larger than the approximately 3,800 hands reported for PokerBattle.ai and the fewer than 2,000 hands used in PokerBench (Zhuang et al., 14 Jan 2025). Block bootstrap over episodes accounts for within-episode correlation and supports confidence intervals for BB/100.

Poker produces a ranking that is markedly different from Chess. GPT-5.2 leads at +46.6 BB/100, followed by o3 at +29.7 and Grok 4 at +27.1. Claude Opus 4.5 (+17.6), Claude Sonnet 4.5 (+12.8), and Gemini 3 Flash Preview (+5.5) are profitable but weaker. DeepSeek V3.2, Gemini 3 Pro Preview, Grok 4.1 Fast Reasoning, and GPT-5 mini are unprofitable at -10.1, -15.2, -19.2, and -94.9 BB/100, respectively.

This reversal is one of the paper’s strongest findings: Gemini 3 Pro Preview, the strongest Chess model, is a losing Poker model, while GPT-5.2 leads Poker despite ranking below both Gemini variants in Chess. The result directly supports the paper’s claim that “strategic competence” is not a single transferable capability.

Figure 5: Poker head-to-head win rates and bootstrapped BB/100 distributions, showing GPT-5.2’s separation from the field and GPT-5 mini’s extreme negative return.

GPT-5.2 is the only model with a positive record against every opponent, with pairwise win rates from 57% to 76%. o3 wins or draws eight of nine matchups. Grok 4 illustrates why aggregate profitability and pairwise dominance are not interchangeable: it achieves a 90% win rate against GPT-5 mini but has a winning record in only three of nine pairings. Its positive BB/100 therefore reflects large margins in favorable matchups rather than uniform superiority.

The preflop analysis further shows why simple behavioral descriptors are inadequate. GPT-5.2 and GPT-5 mini both open extremely wide and maintain high VPIP, yet their outcomes differ by more than 140 BB/100. GPT-5 mini raises first-in on 98.3% of button hands and loses -94.9 BB/100, whereas GPT-5.2 raises on 91.7% and wins +46.6 BB/100. Grok 4 also opens widely and earns +27.1 BB/100, while Grok 4.1 Fast Reasoning opens less frequently and loses -19.2 BB/100.

The implication is that aggression is neither sufficient nor monotonically beneficial. Postflop execution, sizing, range construction, and opponent adaptation account for substantial residual variance. Grok 4’s 56.7% 3-bet rate against button opens is highly aggressive but profitable in this population. In contrast, Gemini 3 Pro Preview folds the big blind 43.8% of the time and 3-bets only 6.0%, a conservative profile that likely concedes too much equity. The paper therefore treats preflop statistics as diagnostic features rather than primary measures of strategic skill.

Werewolf: role-sensitive evaluation in a multiplayer setting

Werewolf introduces eight-player, general-sum interaction with two Werewolves, one Seer, one Doctor, and four Villagers. The game alternates between private night actions and public discussion and voting. A centralized event bus enforces role-dependent visibility, while append-only event logs preserve the complete public and private trajectory. Models use a ReAct-style protocol and return separate private reasoning and public actions.

The ruleset was modified to reduce structural dominance by the Village team. Under standard rules, self-play with Gemini 3 Pro Preview produced a 73.4% Villager win rate. Disabling Doctor self-save alone reduced this only to 72.4%. The final balanced configuration disables self-save and consecutive saves, uses peaceful ties, and rotates discussion and voting order. These changes reduce the Villager win rate to 56.7%.

This ablation is consequential because it demonstrates that benchmark validity depends on game design, not merely on sample size. A heavily imbalanced ruleset can reward exploitation of deterministic mechanics rather than deception, deduction, or coalition management. The authors acknowledge that self-play with a single model is only an empirical proxy for equilibrium analysis; nevertheless, the ablation provides evidence that the selected configuration is substantially less exploitable.

Raw win rates are unsuitable for direct model comparison because roles differ in base difficulty and strategic objective. Game-Theoretic Evaluation (GTE) addresses this problem by decomposing model performance into role-specific contributions within a model-versus-model-versus-role meta-game. The role player emphasizes roles that maximize discrimination between models, while maximum-entropy correlated equilibrium prevents the rating from depending exclusively on one noisy role.

Gemini 3 Pro Preview obtains the highest Werewolf GTE rating at approximately 0.10%, with a bootstrap interval spanning -0.01% to 0.21%. Gemini 3 Flash Preview follows at -0.36%, then GPT-5.2 at -1.12% and Claude Opus 4.5 at -1.31%. GPT-5 mini is last at -6.78%. Role decomposition reveals that Claude Sonnet 4.5 and Grok 4.1 Fast Reasoning are particularly weak as Seers, while GPT-5 mini exhibits negative contributions across essentially every role.

Figure 6: GTE Werewolf ratings and role-specific contributions across approximately 31,472 games.

GTE is not only a ranking replacement for Elo or OpenSkill; it changes the interpretation of model performance. A model can be strong overall because it performs well in common or structurally easy roles while failing in discriminative roles. By weighting informative role comparisons and decomposing role contributions, GTE makes these failure patterns visible. The authors report that GTE produces tighter confidence intervals and greater separation than conventional group Elo and OpenSkill.

The Werewolf results are also robust to alternative ranking formulations. The tournament graph is acyclic in the full dataset, and Elo, Copeland, Ranked Pairs, and GTE produce the same model ordering. Bootstrap samples contain an average of only 0.047 cycles at the full sample size, compared with 2.048 cycles when the sample is reduced to 1,000 games. This result supports the claim that the approximately 31,000-game dataset is large enough to suppress much of the pairwise-ranking noise.

Figure 7: Comparison of Elo, OpenSkill, and GTE, with GTE providing tighter intervals and stronger discrimination among top models.

The cooperative analysis extends the evaluation from individual performance to substitution value. Across role-specific coalitions, replacing another model with Gemini 3 Pro Preview is generally beneficial, whereas the value of replacing a model such as Claude Sonnet 4.5 depends on the coalition and role. This is a useful distinction: a model’s individual rating does not fully determine its contribution as a teammate.

Figure 8: Role-specific substitution analysis showing that team value depends on both the inserted model and the coalition context.

The paper also finds that several intuitive heuristics are poor primary metrics. Key Role Survival Rate remains near 0.5 for many models, even though sacrificing a key role can be strategically optimal. Identification Precision remains around 0.65, while Voting Success Score approaches 1.0 because public consensus can reflect strategic conformity rather than private deduction. These metrics have low discriminative variance and can reward behavior that is not aligned with team victory.

Cross-game rankings and cost-performance structure

The cross-environment results reject the assumption that one leaderboard captures general strategic ability. Gemini variants dominate Chess and Werewolf, while GPT-5.2 dominates Poker. The ranking reversals reflect differences in the computational demands of the environments: deterministic search, stochastic decision-making, and social inference are not interchangeable competencies.

The cost-performance analysis further shows that efficient model choice is environment dependent. In Werewolf, the reported Pareto frontier includes Grok 4.1 Fast Reasoning as a low-cost point, Gemini 3 Flash Preview as a favorable intermediate point, and Gemini 3 Pro Preview as the maximum-performance point. Poker reshapes the frontier because its ranking differs substantially from Chess and Werewolf. Consequently, an aggregate benchmark score would conceal practically important trade-offs between inference cost and the type of strategic capability being measured.

Limitations and open questions

The fixed number of games per matchup is statistically simple but computationally inefficient. Highly unequal pairings receive the same allocation as close matchups, even though close matchups require more samples to resolve. An adaptive scheduler based on uncertainty could improve sample efficiency, but such a scheduler would need stopping rules that preserve comparability and control ranking error.

Longitudinal interpretation is also complicated by model churn. Models may become unavailable, and new entrants may lack direct comparisons with older systems. Maintaining a connected match graph and preserving rating comparability under cold starts remains unresolved.

The paper does not provide a validated cross-game meta-rating. Its environment-specific metrics are appropriate to the different outcome structures, but they prevent a single quantitative statement about overall strategic competence. A consolidated rating would require assumptions about the relative value of Chess Elo, Poker BB/100, and Werewolf GTE, while multiplayer cooperation and role-specific credit assignment make such normalization nontrivial.

Several methodological choices also constrain interpretation. The benchmark evaluates models through specific prompts, inference budgets, formatting requirements, retry policies, and context-window handling. These are part of the measured system behavior, but they can confound model capability with interface robustness. The Werewolf balance study relies on self-play by one high-capability model rather than equilibrium computation or a broad population of policies. Finally, reasoning traces are useful for diagnosis but should not be interpreted as faithful accounts of the causal computations producing actions.

Conclusion

Game Arena establishes a reproducible infrastructure for competitive LLM evaluation across perfect-information, imperfect-information, and multiplayer social-deduction environments (2609.31473). Its main empirical result is not a universal model ordering but the opposite: Chess, Poker, and Werewolf produce materially different rankings and expose different failure modes. The benchmark’s strongest methodological contributions are large-scale variance control in Poker, temporal engine-based analysis in Chess, and role-sensitive GTE ratings in Werewolf.

The results show that objective game outcomes can support finer-grained evaluation than static accuracy or subjective preference alone, provided that the game rules, sampling design, uncertainty estimates, and rating model are treated as integral components of the benchmark. The principal open question is whether the proposed infrastructure can maintain statistically comparable, diagnostically meaningful rankings as new games, models, interaction protocols, and cooperative structures are added.