---
title: 'GENSTRAT: Strategic Benchmarking for LLMs'
url: https://www.emergentmind.com/topics/genstrat
type: topic
---

# GENSTRAT: Strategic Benchmarking for LLMs

Searching arXiv for the GENSTRAT paper and closely related benchmark context.
GENSTRAT is a framework for evaluating strategic reasoning in large language models through a procedurally generated distribution of two-player zero-sum imperfect-information card games, rather than through a fixed suite of canonical benchmarks [2605.23238]. It was introduced to address two problems in existing strategic-reasoning evaluation: fixed benchmarks may saturate as models improve, and performance on a small set of textbook games does not generalize cleanly to the varied strategic environments encountered in deployments such as marketplaces, auctions, and bidding settings [2605.23238]. The framework couples on-demand game generation with a capability-profile methodology over six complexity axes—state space, temporal depth, information sensitivity, opponent modeling, risk, and brittleness—and with a jaggedness measure that quantifies within-distribution smoothness of model performance [2605.23238].

## 1. Concept and evaluative objective

GENSTRAT is organized around a shift from fixed-game benchmarking to distributional evaluation. Instead of asking whether a model performs well on a small set of named games, it asks how a model behaves across a procedurally generated family of novel strategic environments whose structure can vary along interpretable dimensions [2605.23238]. The framework is explicitly motivated by deployments in which LLMs act as economic agents, including simulated shops, marketplaces, pricing tasks, and bidding environments, where performance depends on reasoning about private information, timing, opponent behavior, and risk [2605.23238].

The central criticism of earlier benchmarks is twofold. First, fixed suites are vulnerable to saturation and contamination: once frontier models are strong on canonical games, the benchmark loses discriminative value, and any overlap between benchmark tasks and training corpora complicates interpretation [2605.23238]. Second, a single score on a fixed benchmark does not reveal how performance changes when the strategic environment varies in size, horizon length, information structure, or payoff geometry [2605.23238]. GENSTRAT addresses both issues by generating fresh games on demand and by decomposing model competence into a profile over six axes rather than a single scalar [2605.23238].

A common misconception is that GENSTRAT is merely a larger game leaderboard. In fact, the paper treats overall ranking as only one layer of analysis. Its deployment-relevant summary is the combination of a model’s overall strength, its capability profile over the six axes, and its jaggedness, which measures whether performance changes smoothly or erratically across strategically similar games [2605.23238]. This suggests that two models with near-identical mean strength may nonetheless differ materially in how predictable their behavior is under small environmental changes.

## 2. Generated environment family: generalized betting games

GENSTRAT instantiates its benchmark as a family of generalized betting games, or GBGs. A GBG is defined as a two-player zero-sum extensive-form game with imperfect information consisting of a deck, private hands, other card piles, structured phases, and conditions that gate branches of the game or otherwise control the occurrence of events [2605.23238]. The generated games extend poker-like structures such as Kuhn and Leduc into a broader class of card-based strategic environments with modular phase composition [2605.23238].

The game engine is built from a modular builder. Its core layer includes `GameState`, `Rulebook`, and `Phase`; its action primitives include operations such as `Deal`, `ChipTransfer`, `Shuffle`, `PeekAtHand`, `SwapWithOpponent`, `StealCardMove`, `Wager`, and `ScoreAdjustment`; and its phase templates include `make_action_round()`, `make_observation_round()`, `make_simultaneous_round()`, and `make_position_assignment_round()` [2605.23238]. Phases are connected in a phase graph with conditional transitions gated by quantities such as chip counts, card comparisons, and counters, allowing branches, bounded loops, and conditional sub-phases [2605.23238].

The generator exposes a complexity dial \(c \in [0,1]\) that changes the draw probabilities for more complex structural and surface features, so increasing \(c\) shifts the distribution from Kuhn-like simplicity toward multi-phase complexity [2605.23238]. Reproducibility is ensured by deterministic reconstruction from an integer seed and builder version hash:
\[
G = \text{reconstruct}(s).
\]

Candidate games are filtered by Monte Carlo quality gates before entering the accepted pool. Using 2,000 episodes of random play per candidate, a game is accepted only if its average number of moves per player is at most 10, each phase fires in at least 5% of episodes with no more than 30% of phases falling below that threshold, and no more than 34% of conditional branches remain dead across the Monte Carlo run [2605.23238]. From 12,351 candidate seeds, 2,000 accepted games were collected, and a 50-game benchmark was then selected from this pool by farthest-point sampling in six-dimensional axis space [2605.23238].

This construction has two implications. First, the benchmark is evergreen in the sense that fresh games can be drawn on demand. Second, the benchmark games are samples from a measured distribution of strategic environments rather than isolated handpicked tasks.

## 3. Complexity axes and capability profiles

Each accepted game is represented in a six-dimensional strategic-complexity space computed from Monte Carlo statistics [2605.23238]. GENSTRAT uses 3,000 random-play episodes, 1,500 \(L1\) episodes of best responses against \(L0\), and 320 opponent policies sampled via Sobol sequences for opponent-modeling estimates [2605.23238]. The information state for player \(p\) is written as
\[
I_p = (dt, \text{hand}, \text{action path}, \text{chip bin}, \text{signals}, \text{roles}),
\]
where \(dt = (\text{phase}, \text{move type}, \text{move name})\) [2605.23238].

| Axis | Formal basis | High-level meaning |
|---|---|---|
| State space | \(\log_{10}\!\left(\sum_p |\mathcal{I}_p|\right)\) | Number of distinct observable contexts |
| Temporal depth | \(\sum_d f_d \eta_d^2 r_d\) | Consequential long-horizon decision structure |
| Information sensitivity | Visit-weighted argmax mismatch | Dependence of optimal action on private information |
| Opponent modeling | Modal best-response instability across opponents | Dependence on opponent policy |
| Risk | EV loss from choosing safer actions | EV versus downside trade-off |
| Brittleness | \(\log_{10}\) sensitivity to policy perturbation | Fragility of strategic margins |

The state-space axis measures combinatorial complexity:
\[
\mathrm{state\_space}_{\log_{10}} = \log_{10}\!\left(\sum_{p}|\mathcal{I}_p|\right).
\]
Low values correspond to small, Kuhn-like games, whereas high values correspond to games with many distinct information states, spanning roughly three orders of magnitude in the observed pool [2605.23238].

Temporal depth measures how consequential early actions are for downstream payoff:
\[
\mathrm{temporal\_depth} = \sum_d f_d \, \eta_d^2 \, r_d.
\]
Here \(f_d\) is the mean frequency of decision type \(d\), \(\eta_d^2\) is the fraction of payoff variance attributable to action choice at \(d\), and \(r_d\) is the mean number of later decisions faced by the same player [2605.23238]. This axis is near zero in myopic games and larger in games where early moves shape long chains of later consequences.

Information sensitivity measures how often the best action depends on private information rather than merely on the public decision type:
\[
\mathrm{info\_sens} = \sum_{I_p} \frac{f_{I_p}}{\sum_{I'_p} f_{I'_p}}
\mathbf{1}\!\Big[\, \arg\max_a \mathbb E[U \mid I_p, a] \ne \arg\max_a \mathbb E[U \mid dt(I_p), a]\,\Big].
\]
Low values indicate that one near-optimal action works across many private states; high values indicate that the agent must condition tightly on hidden information [2605.23238].

Opponent modeling measures whether the best response changes across sampled opponent policies:
\[
\mathrm{opp\_mod} = \sum_{dt} w_{dt}\,\Big(1 - \mathrm{modal\_share}_{\pi}\big[\, \arg\max_a \mathbb E[U \mid dt, a, \pi]\,\big]\Big).
\]
If the same action is optimal against nearly all opponents, the axis is near zero; if the optimal response varies substantially with opponent policy, the axis is high [2605.23238].

Risk measures the expected-value penalty incurred by choosing the action with the best 10th-percentile payoff floor rather than the EV-maximizing action:
\[
\mathrm{risk} = \sum_{I_p} w_{I_p}\,
\frac{\mathrm{EV}(a^*_{I_p}) - \mathrm{EV}(a_{\mathrm{safe},\,I_p})}{\sigma_U}.
\]
This captures how much value must be sacrificed to avoid downside tails [2605.23238].

Brittleness measures sensitivity of payoffs to small perturbations of a best-response policy. After moving 3% of policy mass to random alternative actions and estimating payoff sensitivity per decision type, the game-level aggregate is
\[
\mathrm{brittleness}_{\log_{10}} = \log_{10}\!\Bigg(\sum_{dt} w_{dt}\, \frac{|\hat\beta_{dt}|}{\sigma_U}\Bigg).
\]
High values indicate that small execution errors cause large payoff swings [2605.23238].

GENSTRAT then defines a capability profile for each model by regressing per-game strength on the z-scored axes:
\[
\hat\alpha_{m,g} = \beta_{m,0} + \sum_{a=1}^{6} \beta_{m,a}\,z_a(g) + \varepsilon_{m,g}.
\]
The slopes \(\beta_{m,a}\) describe how a model’s edge changes per standard deviation of each axis, controlling for the others [2605.23238]. This turns benchmark performance into a structured diagnostic rather than a single leaderboard number.

## 4. Evaluation protocol and jaggedness

The benchmark games are evaluated in a head-to-head tournament. The reported study uses 50 games selected from the 2,000-game accepted pool and evaluates nine frontier and open-weight LLMs in over 36,000 matches [2605.23238]. Each model pair on each game plays 40 matches, with seat balancing and paired seeds so that chance realizations are shared across seat-swapped runs [2605.23238]. Game rules are rendered into natural-language rulebooks as system prompts, turn observations are rendered as user prompts listing the visible state and legal actions, and the model must output JSON actions [2605.23238].

Overall strength is estimated through an additive paired-comparison model on signed chip margins:
\[
y_s = \alpha_{i^{(s)}} - \alpha_{j^{(s)}} + \varepsilon_s,
\qquad \text{subject to}\quad \sum_m \alpha_m = 0,
\]
where \(y_s\) is Alice-minus-Bob chips for match \(s\) [2605.23238]. Refitting the same model per game yields per-\((m,g)\) strengths \(\hat\alpha_{m,g}\), which are then used in the capability-profile regression [2605.23238].

Jaggedness is introduced to quantify within-distribution smoothness. For model \(m\) on game \(g\), let
\[
\delta_{m,g} = \hat\alpha_{m,g} - \hat\alpha_m
\]
and studentize by a game-level stakes scale \(\sigma_g\):
\[
z_{m,g} = \frac{\hat\alpha_{m,g} - \hat\alpha_m}{\sigma_g}.
\]
After min-max normalizing the six axes to \([0,1]\), GENSTRAT defines a neighborhood \(\mathcal{N}(g)\) consisting of \(g\) and its \(K=3\) nearest neighbors in axis space [2605.23238]. Jaggedness is then the average local standard deviation:
\[
J_m = \frac{1}{|G|}\sum_{g \in G}
\sqrt{\frac{1}{|\mathcal{N}(g)|}
\sum_{g' \in \mathcal{N}(g)} \big(z_{m,g'} - \bar z_{m,g}\big)^2}.
\]

High \(J_m\) indicates that performance jumps unpredictably between nearby games, whereas low \(J_m\) indicates smoother local generalization [2605.23238]. The paper explicitly notes that this measure does not subtract the fitted capability-profile surface before computing local volatility, so steep but smooth global trends still contribute to jaggedness; a residual-only variant is left to future work [2605.23238]. This is important, because jaggedness should not be interpreted as a pure noise-corrected measure of brittleness.

## 5. Empirical findings

The reported evaluation includes nine models: `gpt-5-4-high`, `gemini-3.1-pro-preview`, `claude-sonnet-4-6-max`, `gemini-2.5-pro`, `gemma-4-31b-it`, `deepseek-v3.1-together`, `gemini-3.1-flash-lite-preview`, `qwen-3.5-together`, and `llama-3.3-70b-together` [2605.23238]. On overall strength, measured in chips per game, the top three are `gpt-5-4-high` at \(+0.85\), `gemini-3.1-pro-preview` at \(+0.83\), and `claude-sonnet-4-6-max` at \(+0.64\), while `llama-3.3-70b` is an extreme outlier at \(-2.37\) [2605.23238]. The paper emphasizes that newer frontier-tier models score higher on average, but also that models with near-identical mean strength can have qualitatively different profiles [2605.23238].

The capability profiles differentiate the leading systems. `gemini-3.1-pro-preview` shows a broad profile with gains on state space, opponent modeling, and brittleness, including BH-significant positive slopes of \(+0.22\) chips per standard deviation on state space, \(+0.13\) on opponent modeling, and \(+0.15\) on brittleness [2605.23238]. `gpt-5-4-high` is positive on state space, information sensitivity, opponent modeling, and especially brittleness, with a BH-significant brittleness slope of \(+0.23\) [2605.23238]. `claude-sonnet-4-6-max` has the most concentrated profile among the top three, with its largest effect on brittleness at \(+0.27\), the largest single-axis gain reported in the regression table [2605.23238]. This suggests that the top models are not interchangeable even when their average strength is similar.

The weaker models exhibit more specialized deficits. `llama-3.3-70b` has strongly negative BH-significant slopes on information sensitivity and brittleness, \(-0.40\) and \(-0.42\), respectively, indicating that it falls further behind in games where private information matters and strategic margins are narrow [2605.23238]. `gemini-3.1-flash-lite-preview` is comparatively better on temporal depth and information sensitivity but worse on state space, opponent modeling, and brittleness [2605.23238]. `qwen-3.5` loses ground on risk and opponent modeling [2605.23238]. Mid-pack models such as `gemini-2.5-pro`, `gemma-4-31b-it`, and `deepseek-v3.1` are comparatively flat after control for the six axes [2605.23238].

Jaggedness introduces a second empirical ranking orthogonal to mean strength. The highest reported \(J_m\) belongs to `llama-3.3-70b` at approximately \(0.152\), making it both weak and locally volatile [2605.23238]. Among the top-tier systems, `gpt-5-4-high` and `claude-sonnet-4-6-max` are noticeably more jagged than `gemini-3.1-pro-preview`, with approximate values \(0.092\), \(0.086\), and \(0.062\), respectively [2605.23238]. The smoothest models overall are `deepseek-v3.1`, `gemini-2.5-pro`, and `gemma-4-31b-it` [2605.23238]. The paper’s central interpretive claim is therefore not simply that `gpt-5` and `gemini-3.1-pro` are strong, but that `gemini-3.1-pro` combines high strength with relatively smooth local behavior, whereas `gpt-5` and `claude` are strong but more locally volatile [2605.23238].

A plausible implication is that GENSTRAT is most useful not when a single model is clearly dominant, but when models have similar average strength and differ instead in where and how predictably they succeed. That is the circumstance in which capability profiles and jaggedness become informative for deployment choice.

## 6. Interpretation, limitations, and terminological ambiguity

GENSTRAT is explicitly framed as a deployment-relevant diagnostic rather than a definitive model of strategic rationality. The benchmark family is limited to two-player, zero-sum, English-language, card-based betting games; it does not cover multiplayer, general-sum, cooperative, or non-card economic settings [2605.23238]. The six axes are also complementary rather than orthogonal, and the authors note that state space correlates with information sensitivity and temporal depth [2605.23238]. Moreover, prompting effects remain entangled with strategic skill because the models act through natural-language rulebooks and JSON interfaces rather than through symbolic game representations [2605.23238].

The paper also highlights methodological caveats. Jaggedness currently includes both local volatility and smooth global trends, as well as sampling noise in the per-game strength estimates [2605.23238]. Evaluation is relative to the model pool because strengths are estimated under a sum-to-zero contrast, though a CFR+ baseline on five seeds is used to provide some absolute orientation [2605.23238]. Future directions include extending the GBG family to multiplayer and non-zero-sum settings, adding richer action primitives and larger state spaces, developing residual-based or noise-adjusted jaggedness, and using GENSTRAT distributions in training or fine-tuning [2605.23238].

The term “GENSTRAT” is also ambiguous in adjacent literatures. In human genetics, the provided material notes that “GENSTRAT” is often used loosely for PCA-based stratification correction in the style of EIGENSTRAT, and related work compares such methods with explicit spatial models such as GAP and SCGAP for GWAS correction [1610.07306]. In survey methodology, the term can also refer more broadly to genetic-algorithm-based stratification and sample allocation, including grouping genetic algorithms for partitions of atomic strata [1709.03076]. These usages are conceptually unrelated to the 2026 strategic-reasoning benchmark. In current arXiv usage, however, the exact title “GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models” denotes the procedurally generated strategic-evaluation framework described above [2605.23238].

Taken as a whole, GENSTRAT reframes strategic-reasoning evaluation from performance on a few canonical games to behavior across a controllable distribution of novel games, summarized by overall strength, axis-specific capability, and within-distribution smoothness [2605.23238]. This suggests a broader methodological shift: benchmark design for strategic LLMs may need to resemble experimental science over distributions of environments, rather than static testing on a small list of tasks.

Source: https://www.emergentmind.com/topics/genstrat