GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

Abstract: LLMs are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings. Anticipating their behavior in any specific deployment is hard. Existing strategic-reasoning benchmarks evaluate models on fixed canonical games. These benchmarks may saturate as the frontier improves, and they do not allow evaluators to generalize with confidence from benchmark performance to the varied and messy strategic environments that actual deployments involve. We introduce GENSTRAT, which uses procedurally generated strategic environments to address these challenges. Concretely, we generate a distribution of two-player zero-sum imperfect-information card games. The generator can draw fresh games on demand, allowing for evergreen evaluation and resistance to contamination. We pair the game distribution with a capability-profile methodology that decomposes model competence across six axes (state space, temporal depth, information sensitivity, opponent modeling, risk, and brittleness). We also introduce a jaggedness measure of within-distribution smoothness that detects when a model's advantage jumps unpredictably between strategically similar games. We sample 50 benchmark games from a 2,000-game generated pool and evaluate nine frontier and open-weight LLMs in a head-to-head tournament with over 36,000 matches. Newer frontier-tier models score higher on average. Beyond that average, models with near-identical overall strength show qualitatively different capability profiles, and two of the top three leaderboard models (gpt-5 and claude) are noticeably more locally volatile than the third (gemini-3.1-pro), despite being close in overall strength. Together, the capability profile and the jaggedness measure give a deployment-relevant diagnostic that the overall ranking alone cannot provide.

• The paper introduces a procedural game generation framework that constructs strategic benchmarks for LLMs by varying game configurations and diagnostic axes.
• The methodology employs farthest-point sampling and tournament-style evaluations to ensure comprehensive coverage and robust differentiation among models.
• Diagnostic decomposition along independent axes, including risk and brittleness, provides nuanced insights for risk-aware deployment of LLMs.

GENSTRAT: Procedural Game Generation and Diagnostic Decomposition for Strategic Reasoning in LLMs

Introduction

Strategic reasoning in multi-agent settings, particularly economic games of imperfect information, is a critical dimension for evaluating decision-theoretic capabilities of LLMs. Prior benchmarks, often saturating on fixed canonical games, face severe limitations in representational coverage, generalizability, and contamination control. "GENSTRAT: Toward a Science of Strategic Reasoning in LLMs" (2605.23238) introduces a procedural-game generation framework for constructing a strategic reasoning benchmark, designed with extensibility and analytical depth for the investigation of LLM capabilities. The framework combines a generative methodology for two-player zero-sum imperfect information card games with a six-dimensional diagnostic axis space covering state space, temporal depth, information sensitivity, opponent modeling, risk, and brittleness.

Procedural Generation of Strategic Environments

GENSTRAT's game generator constructs Generalized Betting Games (GBGs), extending the domain of games such as Kuhn and Leduc poker through structural variation at both the surface and compositional levels. The generator samples over deck configuration, hand structures, phased bidding/observation/auction rounds, and dynamic conditional branches. Structural parameters are manipulated to ensure broad coverage over strategic complexity (Figure 1).

Figure 1: The 50-game benchmark in diagnostic axes (state space vs. information sensitivity), exposing the coverage of procedurally generated environments.

Selection for the 50-game benchmark uses farthest-point sampling (FPS) from an accepted pool of 2,000 Monte Carlo-evaluated seeds, maximizing embedding-space dispersion across the six diagnostic axes. Coverage on individual axes is validated via kernel density and tertile-based analysis (Figure 2).

Figure 2: Marginal distributions for each diagnostic axis over the 50 games; FPS ensures games are not concentrated in a few axis directions but instead cover the full spectrum of complexity.

Pairwise Pearson correlations between axis scores remain well below collinearity thresholds, enabling interpretable decomposition and joint analysis. Risk and brittleness axes are nearly orthogonal to the others, as visualized in the correlation matrix (Figure 3).

Figure 3: Pairwise Pearson correlation matrix for the six complexity axes, demonstrating substantial independence especially for risk and brittleness.

Benchmark Methodology and Evaluation Design

A round-robin tournament is constructed using nine LLM agents, spanning the latest frontier models (gpt-5-4-high, gemini-3.1-pro-preview, claude-sonnet-4-6-max) and open-weight baselines, producing over 36,000 matches. For each game, seat assignments are balanced, prompt specifications are strictly controlled via auto-generated rulebooks, and parsing fallbacks are quantitatively analyzed for influence on outcomes. Model edges are estimated using additive paired-comparison regression on signed margin data (chips per game), with sampling error accounted for by bootstrap clustering.

Key findings include:

• A clean leaderboard separation (~3 chips/game range), with top models (gpt-5-4-high, gemini-3.1-pro-preview) separated from lower tiers (llama-3.3-70b-together). Bootstrap CIs confirm the robustness of gaps.
• The leaderboard persists under various ablation regimes, including leave-one-game-out, leave-one-model-out, and complexity-tertile splits, and correlates with performance against a CFR+ baseline across tractable seeds.

Diagnostic Decomposition: Profiles and Local Jaggedness

A critical contribution is the explicit decomposition of model performance along the six diagnostic axes. For each model, the OLS regression coefficient (per-axis slope of margin on standardized axis value) forms a capability profile vector visualized in Figure 4.

Figure 4: Per-model capability profiles, illustrating how overall strength decomposes into specific strategic advantages and deficits along complexity axes.

Notable outcomes:

• Brittleness is the most distinguishing axis among top-tier models; all three (gpt-5-4-high, gemini-3.1-pro-preview, claude-sonnet-4-6-max) exhibit positive BH-significant slopes, indicating growing advantage in high-brittleness regimes.
• Gemini-3.1-pro-preview is notable for a capability profile that is elevated across multiple axes (state space, opponent modeling, brittleness), whereas claude-sonnet-4-6-max's gain is sharply concentrated on brittleness.
• Llama-3.3-70b-together is substantially disadvantaged on information sensitivity and brittleness.

Beyond aggregate performance, the local jaggedness metric ($J_m$), visualized in Figure 5, quantifies the volatility of model advantage across axis-space neighborhoods, normalized by intrinsic game stakes. Higher values indicate substantial variation in margin between structurally similar games.

Figure 5: Local jaggedness $J_m$ per model; higher $J_m$ corresponds to unpredictably high swings in model performance between neighboring strategic environments.

Regime analysis shows that top-tier models can be either volatile (gpt-5-4-high, claude-sonnet-4-6-max) or smooth (gemini-3.1-pro-preview), demonstrating that strength and reliability are decoupled diagnostic dimensions. This has direct deployment implications for robustness and risk calibration in agentic applications.

Per-Game Rank Stability and Complexity Sensitivity

GENSTRAT demonstrates that while the overall leaderboard is predictive for high-complexity games, per-game rankings exhibit significant departures on low-complexity games (Figure 6).

Figure 6: Reversal significance across axis space, showing that statistical disagreement with the overall ranking is concentrated in low-complexity regions.

Across all 50 games, 30% yield per-game rankings that differ significantly from the mean leaderboard beyond what sampling noise predicts, and these instances concentrate in the lowest tertile of composite complexity. As game complexity increases, rankings stabilize and the top models win with greater margin, whereas lower-complexity games see more frequent reversals among mid-pack models.

Model Reasoning Mode Ablation

The authors investigate the marginal value of explicit high-effort "thinking" modes within model families by holding anchor matches constant and varying only the reasoning parameter. For gemini-3.1-pro-preview and gpt-5-4-high, the high-effort variant secures a statistically significant increase in chip margin (+0.44 and +0.21, respectively, with 95% CI excluding zero), indicating that accessible reasoning budget correlates with improved strategic play, a result consistent across both strong and mid-tier anchor opponents.

Implications and Future Directions

GENSTRAT's key contributions are methodological. By restructuring benchmark construction to rely on an open-ended procedural distribution, it delivers a test-bed with both contamination resistance and coverage extensibility—essential for tracking LLM progress into progressively more challenging multi-agent settings. The axis-based diagnostic decomposition and jaggedness measures distill model evaluation beyond monolithic scores, providing guidance for risk-aware deployment, model selection tailored to the relevant task-structure axes, and future research into emergent dynamics of multi-agent LLM interaction.

The analysis highlights that while current frontier LLMs exhibit strong mean performance, considerable within-distribution variance persists, especially as models move through the space of structurally similar but distinct strategic environments. Practical deployment in economic-agent roles should thus weigh both the overall strength and the smoothness of the performance surface over relevant axis regions.

Further extension to cooperative, n-player, or open-ended strategic settings, together with dynamic policy adaptation and meta-reasoning, remains a compelling path for future research. Expansion of procedural complexity dials will be required to keep pace with anticipated LLM progress.

Conclusion

GENSTRAT establishes a scalable, diagnostic, and contamination-resilient benchmark platform for the systematic evaluation of LLMs in strategic reasoning. Through procedural game generation and multidimensional decomposition, it exposes both aggregate strengths and nuanced deficiencies of modern LLMs, guiding theory and practice toward reliable AI deployment in complex multi-agent environments.