GENSTRAT: Strategic Benchmarking for LLMs

• GENSTRAT is a framework that evaluates strategic reasoning in large language models using procedurally generated two-player imperfect-information card games.
• It decomposes model performance into a detailed capability profile over six axes including state space, temporal depth, and risk.
• The framework quantifies performance smoothness via jaggedness, offering actionable insights for deploying LLMs in economic and bidding environments.

Searching arXiv for the GENSTRAT paper and closely related benchmark context. GENSTRAT is a framework for evaluating strategic reasoning in LLMs through a procedurally generated distribution of two-player zero-sum imperfect-information card games, rather than through a fixed suite of canonical benchmarks (Shadarevian et al., 22 May 2026). It was introduced to address two problems in existing strategic-reasoning evaluation: fixed benchmarks may saturate as models improve, and performance on a small set of textbook games does not generalize cleanly to the varied strategic environments encountered in deployments such as marketplaces, auctions, and bidding settings (Shadarevian et al., 22 May 2026). The framework couples on-demand game generation with a capability-profile methodology over six complexity axes—state space, temporal depth, information sensitivity, opponent modeling, risk, and brittleness—and with a jaggedness measure that quantifies within-distribution smoothness of model performance (Shadarevian et al., 22 May 2026).

1. Concept and evaluative objective

GENSTRAT is organized around a shift from fixed-game benchmarking to distributional evaluation. Instead of asking whether a model performs well on a small set of named games, it asks how a model behaves across a procedurally generated family of novel strategic environments whose structure can vary along interpretable dimensions (Shadarevian et al., 22 May 2026). The framework is explicitly motivated by deployments in which LLMs act as economic agents, including simulated shops, marketplaces, pricing tasks, and bidding environments, where performance depends on reasoning about private information, timing, opponent behavior, and risk (Shadarevian et al., 22 May 2026).

The central criticism of earlier benchmarks is twofold. First, fixed suites are vulnerable to saturation and contamination: once frontier models are strong on canonical games, the benchmark loses discriminative value, and any overlap between benchmark tasks and training corpora complicates interpretation (Shadarevian et al., 22 May 2026). Second, a single score on a fixed benchmark does not reveal how performance changes when the strategic environment varies in size, horizon length, information structure, or payoff geometry (Shadarevian et al., 22 May 2026). GENSTRAT addresses both issues by generating fresh games on demand and by decomposing model competence into a profile over six axes rather than a single scalar (Shadarevian et al., 22 May 2026).

A common misconception is that GENSTRAT is merely a larger game leaderboard. In fact, the paper treats overall ranking as only one layer of analysis. Its deployment-relevant summary is the combination of a model’s overall strength, its capability profile over the six axes, and its jaggedness, which measures whether performance changes smoothly or erratically across strategically similar games (Shadarevian et al., 22 May 2026). This suggests that two models with near-identical mean strength may nonetheless differ materially in how predictable their behavior is under small environmental changes.

2. Generated environment family: generalized betting games

GENSTRAT instantiates its benchmark as a family of generalized betting games, or GBGs. A GBG is defined as a two-player zero-sum extensive-form game with imperfect information consisting of a deck, private hands, other card piles, structured phases, and conditions that gate branches of the game or otherwise control the occurrence of events (Shadarevian et al., 22 May 2026). The generated games extend poker-like structures such as Kuhn and Leduc into a broader class of card-based strategic environments with modular phase composition (Shadarevian et al., 22 May 2026).

The game engine is built from a modular builder. Its core layer includes GameState, Rulebook, and Phase; its action primitives include operations such as Deal, ChipTransfer, Shuffle, PeekAtHand, SwapWithOpponent, StealCardMove, Wager, and ScoreAdjustment; and its phase templates include make_action_round(), make_observation_round(), make_simultaneous_round(), and make_position_assignment_round() (Shadarevian et al., 22 May 2026). Phases are connected in a phase graph with conditional transitions gated by quantities such as chip counts, card comparisons, and counters, allowing branches, bounded loops, and conditional sub-phases (Shadarevian et al., 22 May 2026).

The generator exposes a complexity dial $c \in [0,1]$ that changes the draw probabilities for more complex structural and surface features, so increasing $c$ shifts the distribution from Kuhn-like simplicity toward multi-phase complexity (Shadarevian et al., 22 May 2026). Reproducibility is ensured by deterministic reconstruction from an integer seed and builder version hash: $G = \text{reconstruct}(s).$

Candidate games are filtered by Monte Carlo quality gates before entering the accepted pool. Using 2,000 episodes of random play per candidate, a game is accepted only if its average number of moves per player is at most 10, each phase fires in at least 5% of episodes with no more than 30% of phases falling below that threshold, and no more than 34% of conditional branches remain dead across the Monte Carlo run (Shadarevian et al., 22 May 2026). From 12,351 candidate seeds, 2,000 accepted games were collected, and a 50-game benchmark was then selected from this pool by farthest-point sampling in six-dimensional axis space (Shadarevian et al., 22 May 2026).

This construction has two implications. First, the benchmark is evergreen in the sense that fresh games can be drawn on demand. Second, the benchmark games are samples from a measured distribution of strategic environments rather than isolated handpicked tasks.

3. Complexity axes and capability profiles

Each accepted game is represented in a six-dimensional strategic-complexity space computed from Monte Carlo statistics (Shadarevian et al., 22 May 2026). GENSTRAT uses 3,000 random-play episodes, 1,500 $L1$ episodes of best responses against $L0$, and 320 opponent policies sampled via Sobol sequences for opponent-modeling estimates (Shadarevian et al., 22 May 2026). The information state for player $p$ is written as

$$I_p = (dt, \text{hand}, \text{action path}, \text{chip bin}, \text{signals}, \text{roles}),$$

where $$dt = (\text{phase}, \text{move type}, \text{move name})$$ (Shadarevian et al., 22 May 2026).

Axis	Formal basis	High-level meaning
State space	$\log_{10}\!\left(\sum_p |\mathcal{I}_p|\right)$	Number of distinct observable contexts
Temporal depth	$\sum_d f_d \eta_d^2 r_d$	Consequential long-horizon decision structure
Information sensitivity	Visit-weighted argmax mismatch	Dependence of optimal action on private information
Opponent modeling	Modal best-response instability across opponents	Dependence on opponent policy
Risk	EV loss from choosing safer actions	EV versus downside trade-off
Brittleness	$c$0 sensitivity to policy perturbation	Fragility of strategic margins

The state-space axis measures combinatorial complexity: $c$1 Low values correspond to small, Kuhn-like games, whereas high values correspond to games with many distinct information states, spanning roughly three orders of magnitude in the observed pool (Shadarevian et al., 22 May 2026).

Temporal depth measures how consequential early actions are for downstream payoff: $c$2 Here $c$3 is the mean frequency of decision type $c$4, $c$5 is the fraction of payoff variance attributable to action choice at $c$6, and $c$7 is the mean number of later decisions faced by the same player (Shadarevian et al., 22 May 2026). This axis is near zero in myopic games and larger in games where early moves shape long chains of later consequences.

Information sensitivity measures how often the best action depends on private information rather than merely on the public decision type: $c$8 Low values indicate that one near-optimal action works across many private states; high values indicate that the agent must condition tightly on hidden information (Shadarevian et al., 22 May 2026).

Opponent modeling measures whether the best response changes across sampled opponent policies: $c$9 If the same action is optimal against nearly all opponents, the axis is near zero; if the optimal response varies substantially with opponent policy, the axis is high (Shadarevian et al., 22 May 2026).

Risk measures the expected-value penalty incurred by choosing the action with the best 10th-percentile payoff floor rather than the EV-maximizing action: $G = \text{reconstruct}(s).$0 This captures how much value must be sacrificed to avoid downside tails (Shadarevian et al., 22 May 2026).

Brittleness measures sensitivity of payoffs to small perturbations of a best-response policy. After moving 3% of policy mass to random alternative actions and estimating payoff sensitivity per decision type, the game-level aggregate is

$G = \text{reconstruct}(s).$1

High values indicate that small execution errors cause large payoff swings (Shadarevian et al., 22 May 2026).

GENSTRAT then defines a capability profile for each model by regressing per-game strength on the z-scored axes: $G = \text{reconstruct}(s).$2 The slopes $G = \text{reconstruct}(s).$3 describe how a model’s edge changes per standard deviation of each axis, controlling for the others (Shadarevian et al., 22 May 2026). This turns benchmark performance into a structured diagnostic rather than a single leaderboard number.

4. Evaluation protocol and jaggedness

The benchmark games are evaluated in a head-to-head tournament. The reported study uses 50 games selected from the 2,000-game accepted pool and evaluates nine frontier and open-weight LLMs in over 36,000 matches (Shadarevian et al., 22 May 2026). Each model pair on each game plays 40 matches, with seat balancing and paired seeds so that chance realizations are shared across seat-swapped runs (Shadarevian et al., 22 May 2026). Game rules are rendered into natural-language rulebooks as system prompts, turn observations are rendered as user prompts listing the visible state and legal actions, and the model must output JSON actions (Shadarevian et al., 22 May 2026).

Overall strength is estimated through an additive paired-comparison model on signed chip margins: $G = \text{reconstruct}(s).$4 where $G = \text{reconstruct}(s).$5 is Alice-minus-Bob chips for match $G = \text{reconstruct}(s).$6 (Shadarevian et al., 22 May 2026). Refitting the same model per game yields per-$G = \text{reconstruct}(s).$7 strengths $G = \text{reconstruct}(s).$8, which are then used in the capability-profile regression (Shadarevian et al., 22 May 2026).

Jaggedness is introduced to quantify within-distribution smoothness. For model $G = \text{reconstruct}(s).$9 on game $L1$0, let

$L1$1

and studentize by a game-level stakes scale $L1$2: $L1$3 After min-max normalizing the six axes to $L1$4, GENSTRAT defines a neighborhood $L1$5 consisting of $L1$6 and its $L1$7 nearest neighbors in axis space (Shadarevian et al., 22 May 2026). Jaggedness is then the average local standard deviation: $L1$8

High $L1$9 indicates that performance jumps unpredictably between nearby games, whereas low $L0$0 indicates smoother local generalization (Shadarevian et al., 22 May 2026). The paper explicitly notes that this measure does not subtract the fitted capability-profile surface before computing local volatility, so steep but smooth global trends still contribute to jaggedness; a residual-only variant is left to future work (Shadarevian et al., 22 May 2026). This is important, because jaggedness should not be interpreted as a pure noise-corrected measure of brittleness.

5. Empirical findings

The reported evaluation includes nine models: gpt-5-4-high, gemini-3.1-pro-preview, claude-sonnet-4-6-max, gemini-2.5-pro, gemma-4-31b-it, deepseek-v3.1-together, gemini-3.1-flash-lite-preview, qwen-3.5-together, and llama-3.3-70b-together (Shadarevian et al., 22 May 2026). On overall strength, measured in chips per game, the top three are gpt-5-4-high at $L0$1, gemini-3.1-pro-preview at $L0$2, and claude-sonnet-4-6-max at $L0$3, while llama-3.3-70b is an extreme outlier at $L0$4 (Shadarevian et al., 22 May 2026). The paper emphasizes that newer frontier-tier models score higher on average, but also that models with near-identical mean strength can have qualitatively different profiles (Shadarevian et al., 22 May 2026).

The capability profiles differentiate the leading systems. gemini-3.1-pro-preview shows a broad profile with gains on state space, opponent modeling, and brittleness, including BH-significant positive slopes of $L0$5 chips per standard deviation on state space, $L0$6 on opponent modeling, and $L0$7 on brittleness (Shadarevian et al., 22 May 2026). gpt-5-4-high is positive on state space, information sensitivity, opponent modeling, and especially brittleness, with a BH-significant brittleness slope of $L0$8 (Shadarevian et al., 22 May 2026). claude-sonnet-4-6-max has the most concentrated profile among the top three, with its largest effect on brittleness at $L0$9, the largest single-axis gain reported in the regression table (Shadarevian et al., 22 May 2026). This suggests that the top models are not interchangeable even when their average strength is similar.

The weaker models exhibit more specialized deficits. llama-3.3-70b has strongly negative BH-significant slopes on information sensitivity and brittleness, $p$0 and $p$1, respectively, indicating that it falls further behind in games where private information matters and strategic margins are narrow (Shadarevian et al., 22 May 2026). gemini-3.1-flash-lite-preview is comparatively better on temporal depth and information sensitivity but worse on state space, opponent modeling, and brittleness (Shadarevian et al., 22 May 2026). qwen-3.5 loses ground on risk and opponent modeling (Shadarevian et al., 22 May 2026). Mid-pack models such as gemini-2.5-pro, gemma-4-31b-it, and deepseek-v3.1 are comparatively flat after control for the six axes (Shadarevian et al., 22 May 2026).

Jaggedness introduces a second empirical ranking orthogonal to mean strength. The highest reported $p$2 belongs to llama-3.3-70b at approximately $p$3, making it both weak and locally volatile (Shadarevian et al., 22 May 2026). Among the top-tier systems, gpt-5-4-high and claude-sonnet-4-6-max are noticeably more jagged than gemini-3.1-pro-preview, with approximate values $p$4, $p$5, and $p$6, respectively (Shadarevian et al., 22 May 2026). The smoothest models overall are deepseek-v3.1, gemini-2.5-pro, and gemma-4-31b-it (Shadarevian et al., 22 May 2026). The paper’s central interpretive claim is therefore not simply that gpt-5 and gemini-3.1-pro are strong, but that gemini-3.1-pro combines high strength with relatively smooth local behavior, whereas gpt-5 and claude are strong but more locally volatile (Shadarevian et al., 22 May 2026).

A plausible implication is that GENSTRAT is most useful not when a single model is clearly dominant, but when models have similar average strength and differ instead in where and how predictably they succeed. That is the circumstance in which capability profiles and jaggedness become informative for deployment choice.

6. Interpretation, limitations, and terminological ambiguity

GENSTRAT is explicitly framed as a deployment-relevant diagnostic rather than a definitive model of strategic rationality. The benchmark family is limited to two-player, zero-sum, English-language, card-based betting games; it does not cover multiplayer, general-sum, cooperative, or non-card economic settings (Shadarevian et al., 22 May 2026). The six axes are also complementary rather than orthogonal, and the authors note that state space correlates with information sensitivity and temporal depth (Shadarevian et al., 22 May 2026). Moreover, prompting effects remain entangled with strategic skill because the models act through natural-language rulebooks and JSON interfaces rather than through symbolic game representations (Shadarevian et al., 22 May 2026).

The paper also highlights methodological caveats. Jaggedness currently includes both local volatility and smooth global trends, as well as sampling noise in the per-game strength estimates (Shadarevian et al., 22 May 2026). Evaluation is relative to the model pool because strengths are estimated under a sum-to-zero contrast, though a CFR+ baseline on five seeds is used to provide some absolute orientation (Shadarevian et al., 22 May 2026). Future directions include extending the GBG family to multiplayer and non-zero-sum settings, adding richer action primitives and larger state spaces, developing residual-based or noise-adjusted jaggedness, and using GENSTRAT distributions in training or fine-tuning (Shadarevian et al., 22 May 2026).

The term “GENSTRAT” is also ambiguous in adjacent literatures. In human genetics, the provided material notes that “GENSTRAT” is often used loosely for PCA-based stratification correction in the style of EIGENSTRAT, and related work compares such methods with explicit spatial models such as GAP and SCGAP for GWAS correction (Bhaskar et al., 2016). In survey methodology, the term can also refer more broadly to genetic-algorithm-based stratification and sample allocation, including grouping genetic algorithms for partitions of atomic strata (O'Luing et al., 2017). These usages are conceptually unrelated to the 2026 strategic-reasoning benchmark. In current arXiv usage, however, the exact title “GENSTRAT: Toward a Science of Strategic Reasoning in LLMs” denotes the procedurally generated strategic-evaluation framework described above (Shadarevian et al., 22 May 2026).

Taken as a whole, GENSTRAT reframes strategic-reasoning evaluation from performance on a few canonical games to behavior across a controllable distribution of novel games, summarized by overall strength, axis-specific capability, and within-distribution smoothness (Shadarevian et al., 22 May 2026). This suggests a broader methodological shift: benchmark design for strategic LLMs may need to resemble experimental science over distributions of environments, rather than static testing on a small list of tasks.