AutoTavern: A Self-Improving AI for Hearthstone Battlegrounds

• AutoTavern is a self-improving research system where LLM agents develop and iteratively improve an AI for Hearthstone Battlegrounds, focusing on simulator creation, opponent modeling, and reinforcement learning policy.
• The system produced TavernGym, a full-content reproduction of patch 36.6.3, and TavernAgent, which improved in live games from a rating of 6,000 to approximately 15,000 over 20 days, demonstrating a feedback loop of live-game discrepancies influencing both proxy systems and research methods.
• AutoTavern incorporates a reality-anchored feedback loop that uses live-game data to continuously repair and enhance the simulator, opponent models, and policy, leading to a 255% increase in live-game rating over 17 days.

AutoTavern is a self-improving research system in which specialized LLM agents construct, evaluate, deploy, and iteratively repair an artificial intelligence for Hearthstone Battlegrounds. Rather than directly controlling the game through unconstrained language-model reasoning, the system uses LLM agents to develop a verified simulator, calibrated opponent models, benchmark tasks, and a reinforcement-learning policy. Its central mechanism is a reality-anchored feedback loop: live-game discrepancies modify not only the simulator, opponents, and policy, but also the research procedures used to improve them. Over 20 days, AutoTavern produced TavernGym, a full-content reproduction of patch 36.6.3, 23 delivered versions of TavernAgent, and a live-playing system whose rating increased from 6,000 to approximately 15,000 (Ma et al., 24 Oct 2026).

1. Research objective and system concept

AutoTavern addresses the problem of developing a competent game-playing system for a complex, stochastic, partially observable environment. Hearthstone Battlegrounds is an eight-player auto-battler characterized by stochastic tavern shops and combat outcomes, hidden opponent information, a shared minion pool, hundreds of sequential decisions per game, and nearly 1,000 interacting cards, heroes, spells, and trinkets.

The system’s objective is not for LLMs to play the game directly. Instead, specialized research agents develop the infrastructure and procedures required by a conventional game-playing policy. The resulting policy, TavernAgent, is trained with proximal policy optimization (PPO) and operates through a structured game-state representation.

AutoTavern’s central design principle is described as reality-anchored self-improvement. The agents optimize proxy systems—TavernGym, opponent libraries, exams, and training procedures—but repeatedly compare those proxies with live ladder behavior. Errors discovered during live play are used to repair the current system and to create persistent methodological changes, such as stronger verification gates or new research lessons.

The project reported the following outputs:

Component	Reported result
Project duration	20 days
Registered experiments	Approximately 10,000
Delivered TavernAgent versions	23
TavernGym content	All 978 objects in patch 36.6.3
Engine implementation	Approximately 63,000 lines of Rust
Rule-verification records	375
Confirmed rule decisions	220
Fixes caused by live games	184
Main-line TavernAgent training	Approximately 1,900 H200 GPU-hours
All training runs	Approximately 16,700 GPU-hours
Research-agent output	Approximately 100 million tokens
Human messages	418

A human researcher set goals, resolved mechanics that could not be inferred reliably from data, monitored progress, and pressed the shortcut key corresponding to the action selected by the deployed policy. The human did not choose game actions in the counted live games.

2. Research-agent organization and verification

Research state

At iteration $t$, AutoTavern represents its research state as

$(E_t,O_t,X_t,\pi_t,M_t,G_t),$

where $E_t$ is the current simulator, $O_t$ is the opponent model or opponent library, $X_t$ is the collection of exams and benchmark tasks, $\pi_t$ is the current game-playing policy, $M_t$ is typed research memory, and $G_t$ is the set of verification gates and procedural checks.

The system distinguishes between the true objective, the policy’s live-game performance $J(\pi)$, and a proxy objective,

$\hat J(\pi;E_t,O_t,X_t),$

computed using the simulator, modeled opponents, and internal exams. This distinction exposes two failure modes. Proxy drift occurs when $(E_t,O_t,X_t,\pi_t,M_t,G_t),$0 improves because the policy exploits errors in the simulator or opponent model while live performance does not improve. False findings occur when noisy, leaked, or selection-biased experiments are recorded as reliable conclusions.

Specialized agents

AutoTavern uses long-running role-specific LLM agents and short-lived workers:

Agent	Responsibility
Research lead	Reads memory, selects problems, dispatches work, records findings
Environment agent	Implements rules and content, verifies mechanics, repairs discrepancies
Training agent	Designs training recipes, curricula, and runs
Opponent-modelling agent	Builds and recalibrates opponent libraries
Evaluation agent	Performs comparisons, unseen-seed checks, and audits
Deployment agent	Maintains the controller, encodings, and live-game logging
Short-lived workers	Run probes, isolated coding tasks, and analyses
Human researcher	Sets goals and resolves ambiguous mechanics

Agents do not coordinate through unrestricted conversation. Their shared coordination medium is typed research memory.

Typed research memory

The memory contains a numbered finding ledger, correction entries, confirmed rule decisions, unresolved mechanics, handover notes, an experiment registry, and persistent lessons. Corrections point back to earlier findings rather than overwriting them. This allows the system to retain the history of changing conclusions and the reasons for those changes.

The project recorded 220 confirmed rule decisions. Persistent lessons are loaded at the beginning of later sessions, allowing methodological failures to influence subsequent research behavior.

Verification gates

Every important change passes five gates:

1. G1 — Build gate: code changes must pass the full test suite before entering the shared build.
2. G2 — Contract gate: a model must correspond to the exact simulator and rule build on which it was trained; the controller rejects incompatible models.
3. G3 — Comparison gate: models are compared only within one simulator build and across several seeds.
4. G4 — Delivery gate: a candidate is reevaluated on seeds not used for selecting it.
5. G5 — Reality gate: delivered models play live ladder games, which are replayed in TavernGym.

These gates were introduced or strengthened in response to observed failures. For example, future leakage caused by shared random-number states led to permanent reseeding rules; selection bias motivated unseen-seed evaluation; and input-encoding drift led to stronger comparisons between simulator and controller encodings.

Iteration cycle and human oversight

A typical iteration proceeds through observation, diagnosis, planning, implementation, training, evaluation, deployment, and reflection. Training operates over seconds to minutes, research and evaluation over hours, and live-ladder feedback over days.

The human researcher primarily set or changed project goals, resolved mechanics for which available data were insufficient, and pressed the live-game shortcut key. The 418 recorded human messages were approximately distributed across progress or report requests, goal setting, resource issues, approvals, corrections, mechanics rulings, and live-game reports.

3. TavernGym simulator

TavernGym is a Rust simulator with thin Python bindings. It reproduces the full content of Hearthstone Battlegrounds patch 36.6.3, including minions, heroes and Hero Powers, tavern spells, trinkets, other cards and effects, recruit and combat mechanics, shared card-pool behavior, triples and golden minions, discoveries, Battlecries, Deathrattles, auras, Magnetic attachments, positioning, random combat targeting, and start- and end-of-turn effects.

The simulator uses a common effect context for recruit and combat phases. Cards are represented as event hooks, enabling effects to operate consistently across phases. State is stored as an arena of entities with integer identifiers, making game states inexpensive to copy and serialize. Card data are generated from game data files, while later patches are represented as overlays. A source hash identifies a software build and a rule hash identifies the rule set.

Deterministic stochasticity

All stochasticity is drawn from one seeded random-number generator. A seed and action sequence therefore reproduce a game bit-for-bit. This supports exact replays, deterministic debugging, controlled comparisons, save and restore, lookahead simulation, fuzz testing, and reproducible benchmark labels.

A major methodological failure occurred when copied simulation games shared the random-number state. Search appeared to gain 23.6 percentage points in first-place performance because it could effectively see future shops and combat results. After each sample was reseeded, the gain became

$(E_t,O_t,X_t,\pi_t,M_t,G_t),$1

percentage points. This result became a permanent methodological lesson.

Performance and interface

On one CPU core, scripted bots can complete approximately 190–280 complete eight-player games per second and approximately 52,000–76,000 late-game combats per second. A complete game takes approximately 3.6–5.3 milliseconds, while a late combat takes approximately 13–19 microseconds.

The reinforcement-learning interface provides 245 discrete action types, an exact legal-action mask, batched environments, player-view observations, and exact save and restore. Actions include ending the turn, refreshing, freezing, upgrading the tavern, using the Hero Power, buying and selling, playing or magnetizing hand cards, moving minions, and choosing among offers.

Observations expose only information available to the player. The agent sees its own hero, health, board, hand, tavern, gold, and recruit options, as well as public leaderboard information and each opponent’s warband as last observed in combat. It does not receive opponents’ hidden hands, hidden choices, or current private states.

Validation

TavernGym was validated using unit tests, fuzzing with random legal actions, invariant checks after every action, an exact legal-action-mask guard, replay of live recruit phases and combats, comparison of simulator and controller state encodings, 375 recorded rule-verification checks, and live-game discrepancy analysis.

Live games produced 184 fixes:

Defect category	Fixes
Game rules and card behavior	35
Card data and offer rules	11
Input encoding	73
Client controller	51
Opponent or exam recalibration	14
Total	184

The system retains human intervention for ambiguous mechanics that cannot be uniquely determined from game records or external sources.

4. TavernAgent policy and training

Game formulation

Battlegrounds is treated as a partially observable stochastic multiplayer game with eight players. A player’s placement is

$(E_t,O_t,X_t,\pi_t,M_t,G_t),$2

where 1 denotes first place and 8 denotes last place. The policy maps player-visible observations to an action distribution,

$(E_t,O_t,X_t,\pi_t,M_t,G_t),$3

with illegal actions removed by the legality mask.

The environment includes stochastic shops, stochastic combat, hidden opponent state, and shared card-pool effects. The policy’s objective is expected placement-related utility, although the paper does not provide a separate explicit numerical mapping from placement to reward. It reports average placement, first-place rate, and top-four rate.

The final training configuration used

$(E_t,O_t,X_t,\pi_t,M_t,G_t),$4

where $(E_t,O_t,X_t,\pi_t,M_t,G_t),$5 is the discount factor and $(E_t,O_t,X_t,\pi_t,M_t,G_t),$6 is the generalized advantage-estimation parameter. The undiscounted setting preserves terminal placement information across approximately 450 atomic actions per player.

Entity-transformer architecture

TavernAgent is an entity transformer with up to 56 entity tokens and one global token. Tokens represent the player’s minions, hand cards, tavern offers, pending choices, and opponents.

A card token combines an identity embedding, a projection of static features such as type, tier, attack, health, tribe, keywords, and mechanics, and a projection of an embedding of the card’s rule text. Rule-text embeddings provide useful initialization for rare or unfamiliar cards. Attachments such as Magnetic minions and Dark Gifts are folded into the corresponding minion token. Persistent effects, trinkets, and global counters are pooled into the global token.

Opponent tokens contain public leaderboard information, a pooled representation of the last-seen warband, and the result of the corresponding combat. Inputs are compressed with a symmetric logarithm.

The final model has:

• hidden dimension $(E_t,O_t,X_t,\pi_t,M_t,G_t),$7;
• eight transformer layers;
• eight attention heads;
• approximately 18 million parameters.

The shared transformer trunk feeds a pointer/action head for the 245 masked actions, a value head for expected return, and an auxiliary placement head.

Model-size comparisons favored the final-sized model:

Model	Parameters	Main-exam placement	Main first	Main top-4
Small	3.5M	3.84	0.3%	73.9%
Medium	8M	3.77	0.5%	75.6%
Final-sized	18M	3.67	0.5%	78.5%

PPO training

TavernAgent is trained with PPO. The final-stage configuration uses a clipped policy objective with clipping parameter $(E_t,O_t,X_t,\pi_t,M_t,G_t),$8, value coefficient $(E_t,O_t,X_t,\pi_t,M_t,G_t),$9, entropy coefficient $E_t$0, auxiliary-placement coefficient $E_t$1, target KL $E_t$2, and gradient norm clipping at 1.0.

The final-stage training configuration included:

Quantity	Value
Games per member × steps	$E_t$3
Decisions per iteration	Approximately $E_t$4
Iterations per stage	300
Minibatch	8,192
Epochs	1
Learning rate	$E_t$5
PPO clip	0.2
Target KL	0.05
Gradient norm clip	1.0
Discount factor	1.0
GAE parameter	0.995
Entropy coefficient	0.002
Value coefficient	0.5
Auxiliary-placement coefficient	0.1
KL anchor	0.1 for the first 100 iterations

Each later stage inherited the preceding checkpoint and used a KL anchor to prevent abrupt policy drift. A 300-iteration stage represented approximately 310 million decisions and 570,000 games.

Opponent modelling

TavernAgent does not rely solely on self-play. It plays against seven library seats constructed from top-human-player statistics, recorded games, tribe shares, tavern-upgrade curves, purchase rates, warband strength by turn, hidden counters, and hand states.

A library seat selects a board matching a target strength curve. Later libraries were recalibrated to opponents encountered in live games and incorporated graded difficulty, ranging from aligned opponents to harder late-game boards. Exam libraries were not used for training; training used separate twin libraries constructed from other sources.

Opponent calibration produced the following comparison:

Opponent libraries	Main first	Main place	Hard first	Hard place
Uncalibrated	20.9%	2.74	10.3%	3.64
Reality-calibrated	27.2%	2.65	13.5%	3.59

The paper identifies opponent calibration as the largest single training gain in that comparison.

Curriculum

Training proceeded through a curriculum involving a simplified rule set, full content, player-view observations, a first-place objective against human-statistics opponents, calibrated opponent libraries, and graded libraries with successive corrected environments. This progression allowed the policy to acquire basic mechanics before facing the full partially observable and stochastic setting.

5. Iterative development and empirical findings

AutoTavern delivered 23 versions, designated TA-1 through TA-23. Changes included new opponent libraries, environment updates, recalibration to live games, a play-order fix, and merged specialist models.

On one simulator build, one exam, and three unseen seeds, using 24,576 games per row, the progression was:

Version	Average place	First	Top-4
TA-1	3.40	1.9%	83.4%
TA-11	2.83	18.5%	85.5%
TA-23	2.33	41.5%	89.0%

On the harder exam:

Version	Average place	First	Top-4
TA-1	4.07	1.1%	65.4%
TA-11	3.75	8.9%	67.8%
TA-23	3.28	23.8%	73.2%

A same-lobby version league also showed improvement. TA-23 achieved an average placement of 3.95 and a first-place rate of 26.2%, compared with 5.10 and 0.6%, respectively, for TA-1.

Methodological failure cases

Several failures shaped the research process:

• Future leakage: shared random-number states created an artificial search advantage and resulted in permanent reseeding.
• Error cancellation: an overly strong late-game opponent model and overstated agent growth initially offset one another, producing misleading agreement with live results.
• Winner’s curse: selecting the best snapshot among many inflated selection-seed gains by 1.2–2.1 points, while the selected model lost 0.7 points on unseen seeds.
• Opponent mismatch: uniformly sampled libraries failed to represent the opponents actually encountered by the learner.
• Ladder–exam mismatch: an exam improvement from 12.8% to 24.2% first places was accompanied by a live rating decline from 8,968 to 7,658 over 32 games.
• Input-encoding drift: a difference between simulator and live-controller encoding caused the deployed agent to sell its entire warband more often than in simulation.

Decision-value and hero analyses

Ablations found that removing or randomizing Dark Gifts, Discovery choices, and targets cost approximately 1.4–1.9 placements. Trinket choices cost approximately 0.5 placement, while Hero Power decisions cost approximately 0.25 placement.

Across 111 heroes, the correlation between TavernAgent’s hero-specific placement and public human statistics was only

$E_t$6

The agent was relatively strong with heroes that human players rated weak, including Fungalmancer Flurgl, Trade Prince Gallywix, and Guff Runetotem, and relatively weak with planning-intensive heroes such as The Jailer, Alexstrasza, and Captain Eudora.

6. Live deployment, TavernBench, and limitations

Live-ladder deployment

The live controller reads the game, converts it into TavernAgent’s simulator encoding, obtains an action from the policy, names that action, and has a human press the corresponding official-plugin shortcut. Games with illegal client proposals or controller errors were abandoned, diagnosed, and repaired.

Across days 5–18, 392 games were recorded. The rating rose from 6,000 to 14,007 and reached approximately 15,000 by the end of the study, placing the agent in the server’s top 100. Of 388 games with recorded placements, 96 contained at least one client failure.

Excluding failures, the overall results were an average placement of 3.22, a first-place rate of 16.4%, and a top-four rate of 77.1%. For TA-19 through TA-23, excluding failures, the results were an average placement of 2.37, a first-place rate of 32.6%, and a top-four rate of 91.3%.

The last 20 public leaderboard records were compared with the last 20 records of the server’s ten highest-rated players:

Group	Average place	First	Top-4	Rating change
Server top ten together	2.34	59%	81%	+1,425
TavernAgent	1.60	80%	90%	+1,282

The comparison is limited by the short 20-record window, estimated leaderboard placements, unequal sampling, client failures, one patch, one account, and the absence of confidence intervals or formal significance tests.

TavernBench

TavernBench contains 960 tasks in 40 scenarios and eight categories. Its tasks provide a player-visible state in text, rule text for displayed cards, an objective, and a required answer format. The benchmark includes rule execution, economy, combat, positioning, ordering, targets, selection, and strategy.

Combat labels use up to 4,096 simulations, positioning uses 2,048 combats per arrangement, short-term decision labels use 1,024 combats per option, and strategy tasks compare four plans—upgrade, buy, refresh, or save—by playing each 128 times.

TavernAgent receives the engine state in its standard policy representation rather than the textual benchmark state. For combat and short-term tasks, it performs one-step lookahead using TavernGym.

On the common comparison set, TavernAgent achieved 91% play accuracy, 90% real-time accuracy, and a median decision time of 0.03 seconds:

System	Play accuracy	Real-time accuracy	Median time
TavernAgent	91%	90%	0.03 s
GPT-6 Astra	63%	0%	2.4 s
Claude Sonnet 5.5	53%	0%	1.4 s
Claude Opus 5.5	52%	0%	6.2 s
GPT-6.1 Sol	46%	0%	2.0 s
Gemini 3.8 Flash	44%	0%	9.9 s
GLM-5.3	40%	0%	26 s
Intern-S2-Preview	32%	0%	2.8 s
DeepSeek-V4.1-Flash	26%	26%	0.91 s
Kimi-K2.6	28%	0%	16 s
Grok 4.7	22%	0%	4.8 s
Jev	37%	37%	0.26 s

TavernAgent’s category results were 100% in combat, 98% in positioning, 100% in ordering, 98% in targets, 64% in selection, and 75% in strategy. On the full 960-task set, it achieved 92%.

When LLMs were given TavernGym tools, their short-term decision accuracy improved substantially. GPT-6 Astra increased from 62% to 97%, GLM-5.3 from 38% to 81%, Kimi-K2.6 from 26% to 65%, Intern-S2-Preview from 32% to 41%, and DeepSeek from 24% to 64%. Remaining failures involved illegal or mis-indexed actions, shifting hand positions, incomplete candidate search, and large purchase-sequence spaces.

Limitations and reproducibility

The reported evidence is constrained by a single patch, one game and account, approximately two weeks of live ladder play, client failures, calibrated rather than independently learning opponents, human intervention for ambiguous mechanics, and limited transfer evidence.

The live comparison with top server players uses estimated placements from a public leaderboard tracker and only 20 recent records per player. The paper reports no confidence intervals or formal significance tests for the ladder claims. TavernBench labels and some strategy judgments depend on TavernGym and earlier TavernAgent versions, particularly TA-21, creating a potential benchmark-circularity concern.

The system’s performance is inseparable from the correctness of TavernGym and the controller’s input encoding. The supplied material does not specify a complete numerical placement-reward formula, every prompt, every random seed, all card scripts, or the complete configuration of every experiment.

TavernGym and TavernBench are stated to be open-sourced at https://github.com/MCG-NJU/AutoTavern. The reported system therefore combines reproducible interfaces, simulator architecture, model specifications, training hyperparameters, evaluation procedures, and release history, while a complete reproduction of the 20-day research process would require additional implementation and configuration details.

AutoTavern’s principal contribution is an architecture for using LLM agents to construct and repair the infrastructure required by a strong policy:

$E_t$7

Its empirical significance derives from the interaction of these components rather than from policy training alone. TavernGym provides a fast and deterministic environment, TavernAgent provides a low-latency structured policy, calibrated opponents reduce proxy mismatch, TavernBench measures game competence and real-time feasibility, and live deployment supplies evidence that can change both the artifacts and the research process. The resulting system substantially outperforms the LLMs that built it when they are asked to make the same game decisions, while the strongest live-ladder claims remain subject to the stated sampling, deployment, benchmark, and statistical limitations.