---
title: 'TavernBench: Hearthstone Battlegrounds Decision Benchmark'
url: https://www.emergentmind.com/topics/tavernbench
type: topic
---

# TavernBench: Hearthstone Battlegrounds Decision Benchmark

TavernBench is a decision benchmark for Hearthstone Battlegrounds designed to evaluate whether a model can make the decisions required to play the game, rather than merely describe its rules or provide advice. It contains 960 tasks organized into 40 scenarios spanning rules, economy, combat, positioning, ordering, targets, selection, and strategy. The benchmark was introduced alongside AutoTavern, a self-improving research system that developed TavernGym, a full-content game reproduction, and TavernAgent, an 18-million-parameter policy trained with PPO. TavernBench reports that TavernAgent achieves 91% play accuracy on a shared 368-task evaluation, compared with at most 63% for the directly evaluated frontier LLMs, while retaining a 90% real-time score under a one-second decision threshold [2610.33845].

## 1. Scope and purpose

TavernBench targets the distinction between knowing the rules of Hearthstone Battlegrounds and knowing how to play. Battlegrounds combines stochastic tavern offers and combat outcomes, partial observability, a shared card pool, nearly 1,000 interacting cards, heroes, spells, trinkets and effects, hundreds of decisions per game, and delayed reward because the primary objective is final placement. Its decision space includes both tactical choices, such as combat positioning, and long-horizon choices, such as whether to upgrade, buy, refresh, or save gold.

The benchmark evaluates several layers of competence:

- **Observation and state interpretation**: extracting relevant facts from a player-visible state.
- **Rule execution**: determining the effects of cards, triggers, and actions.
- **Reasoning**: predicting combats, targets, triggers, and interactions.
- **Action selection**: choosing purchases, positions, targets, heroes, trinkets, or play orders.
- **Short-horizon planning**: selecting the option that improves the next combat.
- **Long-horizon strategy**: selecting a turn plan whose consequences are evaluated by playing out the rest of the game.
- **Real-time decision-making**: producing a correct answer quickly enough for live play.

The benchmark therefore evaluates direct decision competence rather than verbal game knowledge. Frontier models can execute many rules correctly while performing substantially worse on decisions whose quality is determined by simulated play.

## 2. Benchmark composition and task inputs

TavernBench contains 960 tasks in 40 scenarios, averaging 24 tasks per scenario. The scenarios are divided into eight categories:

| Category | Scenarios | Primary content |
|---|---:|---|
| Rules | 7 | Legal actions, effects, Battlecries, spells, magnetic effects, turn effects, triples |
| Economy | 5 | Gold accounting, upgrade costs, triples, gold chains, best purchase |
| Combat | 6 | Attack order, combat outcome, expected damage, lethal risk, start-of-combat effects |
| Positioning | 5 | Arrangements involving Taunt, Divine Shield, Deathrattle, Reborn, Venomous, or Cleave |
| Ordering | 5 | Sell timing, play triggers, adjacent Battlecries, spell and minion order |
| Targets | 5 | Spell, Hero Power, activation, magnetic, and sell targets |
| Selection | 5 | Heroes, trinkets, Discover choices, natural-game choices, and card selection |
| Strategy | 2 | Early- and late-game plans involving upgrade, buy, refresh, or saving gold |

The category counts sum to 40 scenarios. Ten scenarios concern rule execution, while 30 are play scenarios in which TavernAgent or language models must choose actions or plans. A common cross-system comparison uses 368 tasks, described as eight tasks per included scenario. TavernAgent achieves 91% on this shared set and 92% across all play tasks in the full benchmark; the 91% figure is therefore not the accuracy over all 960 tasks.

A static task supplies a player-visible game state in text, the current turn, health, gold, tavern tier, warband, hand, offers, relevant opponent information, rule text for every shown card, a question or objective, and an explicit answer format, usually a single JSON object. Because relevant card rules are included in the prompt, the task does not require memorization of card text from outside the input. A target-selection task, for example, requires an answer of the form:

```json
{"answer": <warband position>}
```

The benchmark combines controlled diagnostic scenes with naturally occurring decisions. Static states begin from real game states but are reshaped by scenario generators controlling factors such as turn, tavern tier, gold, health, warband composition, opponent warband, card pool, and legality. Strategy and natural-choice tasks are cut from actual games in the main exam lobby at decisions of the relevant type.

## 3. Task construction and labeling

Rules tasks ask what the game engine does after an action sequence. A representative task asks for the Attack and Health of every minion after casting Misplaced Tea Set. Such labels are generated by executing actions in TavernGym. If randomness is involved, the task is retained only when two or three independently reseeded copies agree on the outcome.

Combat tasks evaluate attack order, win, tie, or loss, outcome probabilities, expected damage, lethal risk, and start-of-combat effects. Ordinary combat tasks use 4,096 simulated combats, while the damage-distribution scenario uses 400 individual combats.

Positioning tasks compare candidate warband arrangements, with each arrangement simulated for 2,048 combats. A positioning task is retained when the best arrangement is sufficiently separated from alternatives, typically by at least five percentage points of win rate.

Ordering, targeting, and selection tasks evaluate candidate options using 1,024 simulated combats after each option. Tasks are retained when the best option beats the median option by 5–8 win-rate points and no more than one quarter of options are within three points of the best. An answer is correct when its option is within three percentage points of the best option’s simulated win rate. A graded decision score is based on the selected option’s win rate relative to the best option’s win rate.

Strategy tasks are the longest-horizon component. They are extracted from the first decision of a turn in natural games and compare four plans:

1. Upgrade the Tavern.
2. Buy the minion preferred by the labeller.
3. Refresh first.
4. End the turn and save the gold.

Each plan is played out 128 times by TA-21, an earlier TavernAgent version, which makes the remaining decisions and plays the rest of the game. A task is retained only when the best plan exceeds the second-best plan by at least 0.25 placement points and the difference is at least two standard errors. The labels are outcomes of play rather than subjective preferences. On these tasks, TA-21 obtains 73%, final TavernAgent TA-23 obtains 75%, and TA-1 obtains 57%.

Hero and trinket choices use public statistics from top human players. An offer is retained when the best hero leads by at least 0.1 placement points or the best trinket leads by at least 0.15 placement points. These labels are therefore indirect and reflect public performance statistics rather than a direct causal estimate of intrinsic option value.

## 4. Evaluation protocol

Language models receive a common system prompt and a user prompt containing the rules summary, state, relevant card rules, objective, and answer format. They must return one JSON object within a fixed output budget. Reasoning is included in that budget, and a response that exceeds the budget and is cut off is scored as wrong. The main comparison evaluates models without tools.

The model mode is selected through a pilot containing one task from each scenario. Kimi-K2.6, DeepSeek-V4.1-Flash, and Intern-S2-Preview use reasoning off; the other evaluated LLMs use low-effort reasoning. Prompts use approximately 0.9–3.9k input tokens and 9–2,772 output tokens, except for Grok, whose median output is much larger. Over-budget rates include 48% for Grok 4.7, 23% for Kimi-K2.6, 16% for Gemini 3.8 Flash, and at most 4% for the other models.

For ordinary categorical tasks, accuracy is the proportion of correct answers among evaluated tasks. For decision tasks, an answer is correct when the selected option is within three percentage points of the best option’s simulated win rate. If $w(a)$ is the simulated win rate of option $a$ and $w^*$ is the maximum candidate win rate, correctness requires:

$$
w^* - w(a) \leq 0.03.
$$

The benchmark also reports a graded decision score based on the ratio of the selected option’s win rate to the best option’s win rate. For probability-triple questions, the predicted most likely outcome must match the simulated most likely outcome, and the Brier score must not exceed 0.1.

TavernGym uses seeded randomness, allowing exact replay of a state and action sequence. Benchmark labels generally use many independent simulations: 4,096 combats for combat tasks, 2,048 combats per positioning arrangement, 1,024 combats after each candidate decision, and 128 full play-outs per strategic plan. The game remains partially observable. TavernAgent receives its complete state, public leaderboard information about opponents, and opponents’ warbands only as last seen in combat. It does not receive opponents’ hidden hands or unobserved future tavern outcomes.

Real-time performance counts only correct answers produced within one second:

$$
\operatorname{RealTimeScore}
=
\frac{N_{\text{correct and }t\leq 1\text{ s}}}{N_{\text{tasks}}}
\times 100\%.
$$

The one-second threshold is motivated as the maximum acceptable time per decision for live play. LLM task time is estimated as the sum of provider-standard median first-token time and answer-generation time from Artificial Analysis. Using actual output-token counts instead produces median task times of 3–50 seconds without changing real-time scores.

## 5. Results and comparative performance

On the shared 368-task comparison, TavernAgent achieves the following results:

| System | Play | Real-time | Combat | Position | Order | Targets | Select | Strategy | Rules | Median time |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| TavernAgent | 91 | 90 | 100 | 98 | 100 | 98 | 64 | 75 | — | 0.03 s |
| Claude Opus 5.5 | 52 | 0 | 67 | 32 | 60 | 52 | 41 | 38 | 92 | 6.2 s |
| Claude Sonnet 5.5 | 53 | 0 | 58 | 45 | 68 | 52 | 39 | 29 | 85 | 1.4 s |
| GPT-6 Astra | 63 | 0 | 75 | 55 | 72 | 55 | 50 | 54 | 95 | 2.4 s |
| GPT-6.1 Sol | 46 | 0 | 50 | 42 | 48 | 28 | 55 | 43 | 56 | 2.0 s |
| Gemini 3.8 Flash | 44 | 0 | 52 | 32 | 32 | 48 | 41 | 43 | 90 | 9.9 s |
| Grok 4.7 | 22 | 0 | 29 | 0 | 5 | 20 | 38 | 37 | 79 | 4.8 s |
| GLM-5.3 | 40 | 0 | 54 | 20 | 40 | 38 | 41 | 34 | 65 | 26 s |
| Kimi-K2.6 | 28 | 0 | 31 | 8 | 22 | 38 | 36 | 27 | 56 | 16 s |
| Intern-S2-Preview | 32 | 0 | 31 | 12 | 32 | 40 | 44 | 34 | 54 | 2.8 s |
| DeepSeek-V4.1-Flash | 26 | 26 | 29 | 18 | 15 | 32 | 25 | 19 | 29 | 0.91 s |
| Jev | 37 | 37 | 56 | 25 | 20 | 45 | 40 | 17 | 35 | 0.26 s |

TavernAgent is strongest on local tactical choices, particularly combat, positioning, ordering, and targeting. Its lower selection score concerns heroes, trinkets, Discover options, and other broader evaluations. Its 75% strategy score provides direct evidence of competence on choices whose quality depends on longer-horizon consequences.

GPT-6 Astra is the strongest directly evaluated LLM at 63% play accuracy. The evaluated LLMs perform substantially better on rules than on play: GPT-6 Astra achieves 95% on rules, Claude Opus 92%, and Gemini 3.8 Flash 90%, while their play scores are 63%, 52%, and 44%, respectively. This supports the distinction between correctly executing rules and selecting actions that produce better outcomes.

TavernAgent’s median decision time is 0.03 seconds. LLM median times range from 0.91 seconds for DeepSeek-V4.1-Flash to 26 seconds for GLM-5.3. Under the one-second cutoff, TavernAgent achieves a 90% real-time score, compared with 26% for DeepSeek-V4.1-Flash, 37% for Jev, and 0% for every other listed LLM. The 0.03-second value is a median benchmark-task latency; it does not mean that 90% of complete live games are won or that every live action takes exactly 0.03 seconds.

The paper does not report confidence intervals, standard errors, hypothesis tests, or p-values for the headline accuracy and timing results. The statistical uncertainty of the system-level scores is therefore not quantified.

## 6. Tool use, simulator dependence, and system relationships

A tool-use ablation gives five LLMs access to TavernGym through `test_play(actions)`, which applies recruit actions and simulates the next combat 256 times, and `inspect(actions)`, which inspects the resulting state. Models may make at most 12 tool calls per task. On the same 144 short-term decision tasks, tool access substantially improves performance:

| Model | Text-only | With tools | TavernAgent | Tool calls | Time |
|---|---:|---:|---:|---:|---:|
| GPT-6 Astra | 62 | 97 | 99 | 5.2 | 21 s |
| GLM-5.3 | 38 | 81 | 99 | 5.8 | 11 s |
| Kimi-K2.6 | 26 | 65 | 99 | 8.9 | 39 s |
| Intern-S2-Preview | 32 | 41 | 99 | 9.8 | 18 s |
| DeepSeek-V4.1-Flash | 24 | 64 | 99 | 8.6 | 42 s |

GPT-6 Astra improves from 55% to 98% on positioning, from 72% to 100% on ordering, and from 55% to 92% on targets. Residual failures include illegal or mis-indexed actions, hand-position shifts after plays, failure to test all candidate actions within the tool budget, and large purchase-sequence search spaces. Tool access therefore reduces the burden of mentally simulating the game but does not reproduce TavernAgent’s performance or latency. Jev, a typed decision model selecting among explicitly enumerated candidates rather than generating free text, achieves 37% overall and 37% in real time at 0.26 seconds per task.

TavernGym is a deterministic, seeded Rust reproduction of the full game. It implements all 978 objects in patch 36.6.3, provides 245 discrete actions, exact legality masks, player-view observations, deterministic save and restore, batched games, and high-throughput complete-game and combat simulation. It supplies much of TavernBench’s labeling and simulation infrastructure.

TavernAgent is an entity transformer trained with PPO. It receives engine state rather than TavernBench text, uses up to 56 entity tokens plus a global token, card identity and static features, rule-text embeddings, a 245-action pointer head, and value and placement heads. In TavernBench, it answers through its policy, value head, and one-step lookahead through TavernGym.

AutoTavern is the research loop that built the simulator, opponents, benchmark, and policy. Specialized LLM agents perform research leadership, environment implementation, training, opponent modeling, evaluation, and deployment. They communicate through typed, versioned research memory and use five verification gates. Live games feed discrepancies back into the simulator, opponent model, controller, and evaluation process.

## 7. Interpretation and limitations

TavernBench is anchored to one game version, patch 36.6.3, and one Battlegrounds rule environment. Because the game changes frequently, performance may decay after a patch. The live study covers one patch, one account, one deployment period, and a limited number of games.

Most labels come from TavernGym or TavernAgent play-outs. TavernAgent was trained with TavernGym, TavernBench labels are generated by TavernGym and TA-21 for strategy tasks, and TavernAgent uses TavernGym for some lookahead. High TavernAgent performance therefore establishes strong consistency with the constructed simulator and its policy, but not complete correctness in every possible real-game situation. Live evaluation mitigates this concern without eliminating it.

Strategy labels are policy-relative: they measure which plan performs best under TA-21’s continuation policy and the benchmark’s opponent environment, rather than an absolute game-theoretic optimum. Human-statistics labels for heroes and trinkets may reflect popularity, selection effects, patch-specific metagame effects, and confounding factors.

Task filtering improves label reliability but makes the benchmark diagnostic rather than representative of all game decisions. Tasks are retained when options differ sufficiently in simulated win rate, placement outcomes, or reseeded rule execution. The benchmark consequently emphasizes decisions where one option is measurably better.

The paper does not provide a dedicated contamination audit, train-test contamination checks, canary tasks, private holdouts, or memorization probes. Supplied card rules reduce dependence on memorizing card text, but do not rule out memorization of general Battlegrounds strategy or publicly available examples. The authors state that TavernBench and TavernGym will be open-sourced at [https://github.com/MCG-NJU/AutoTavern](https://github.com/MCG-NJU/AutoTavern), but the supplied paper does not specify a finalized release license, dataset license, or complete release package contents.

The live deployment also involved a human pressing official plug-in shortcuts after TavernAgent named each operation. The human did not make game decisions, but the arrangement is not identical to an entirely autonomous client. Ladder results therefore demonstrate end-to-end capability under the reported deployment arrangement rather than unrestricted autonomous operation under every platform or ruleset.

TavernBench’s central result is that specialized policy learning and simulator-mediated decision systems substantially outperform directly prompted language models on the combined requirements of decision quality, tactical and strategic integration, and real-time response. Its findings also show that LLM research agents can build a strong specialized game-playing system without themselves being strong direct players. The results should nevertheless be interpreted as performance on a simulator-anchored, filtered, single-patch benchmark, not as a complete measure of general game intelligence, human-like reasoning, or statistically established superiority across all future Battlegrounds environments.

Source: https://www.emergentmind.com/topics/tavernbench