Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players

Abstract: Can LLM agents carry a hard real-world problem from raw data to a system that holds its own against people? We study this in Hearthstone Battlegrounds, an eight-player auto-battler with stochastic shops and battles, hidden information and nearly a thousand interacting cards. We present AutoTavern, a self-improving research system: specialised LLM research agents share a typed research memory, every claim passes verification gates, and live games close the loop, so the agents improve the simulator, opponents and policy and, from the same feedback, their own research process; a human sets goals and rules on ambiguous mechanics. In 20 days AutoTavern built TavernGym, the first faithful, full-content reproduction of the live game, and delivered 23 versions of TavernAgent, trained with PPO against opponents calibrated to real games. TavernAgent plays complete games on the live ladder by itself: its rating rose from 6,000 to about 15,000 and, on the public leaderboard's measure, its last 20 records average a placement of 1.60 with 80% first places, better than the server's ten highest-rated players on every statistic (2.34, 59%). TavernBench, 960 tasks in 40 scenarios, shows that the system the agents built is the strongest player: TavernAgent answers 91% of the play tasks against at most 63% for frontier LLMs, and decides in 0.03 s where they need up to 26 s; within the second a live turn allows, it keeps 90% and no LLM more than 26%. LLM agents build far better than they play. We will open-source TavernBench and TavernGym at https://github.com/MCG-NJU/AutoTavern.

• The paper demonstrates an LLM-based system, AutoTavern, that autonomously develops an AI for Hearthstone Battlegrounds, outperforming top players and playing competitively in a live game environment.
• AutoTavern's primary method consists of specialized research roles within six types of LLM agents, coupled with a typed, versioned research memory, verification gates, and automated experimentation, noting the agent required 0.6% human input post initial 20 days creation.
• The live-deployment loop in which TavernAgent was tested revealed 184 defects in the TavernGym simulator, highlighting the necessity of live game testing to maintain simulation fidelity, the system only needed 0.6% human involvement later on to input objectives or resolve ambiguous mechanics; this further supports the paper's central claim of autonomous system improvement.

Research objective and central claim

The paper investigates whether LLM-based research agents can autonomously carry a technically demanding game-AI project from environment construction to deployment in a live competitive setting. The target is Hearthstone Battlegrounds, an eight-player auto-battler characterized by stochastic shops and combat, partial observability, shared card pools, long-horizon credit assignment, and nearly one thousand interacting game objects. The central claim is deliberately divided into two parts: LLM agents can construct a highly capable specialized system through an iterative research process, but the same LLMs are substantially weaker when asked to make the system’s decisions directly.

The proposed system, AutoTavern, combines six specialized LLM research roles with a typed, versioned research memory, verification gates, automated experimentation, and a reality loop connecting simulation to live games. Over 20 days, the system registered approximately 10,000 experiments, produced 23 delivered TavernAgent versions, implemented 63,000 lines of Rust, and required 418 human messages, reported as 0.6% of all messages. The human remained responsible for setting objectives, resolving mechanics that could not be inferred unambiguously, and pressing the shortcut key corresponding to the agent’s selected action during live deployment. Thus, the claimed autonomy is substantial but not equivalent to unsupervised deployment.

The empirical contribution is supported by three artifacts: TavernGym, a full-content simulator for patch 36.6.3; TavernAgent, a PPO-trained policy operating through a transformer architecture; and TavernBench, a 960-task benchmark for evaluating Battlegrounds decision-making. The results are reported in “Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players” (2610.33845).

AutoTavern as a reality-anchored research process

AutoTavern models research as an evolving collection of artifacts: the simulator, opponent models, evaluation exams, policy, research memory, and verification gates. The system’s defining design choice is that improvements to the proxy objective are not accepted as evidence of progress unless they survive independent checks and, ultimately, comparison with live-game behavior.

The research agents do not communicate through unconstrained conversational exchange. Instead, they write to typed and versioned stores containing findings, rule decisions, open questions, experiment specifications, handover notes, and persistent lessons. Corrections are appended rather than overwriting previous findings. This creates an auditable lineage of conclusions and makes methodological errors part of the system’s persistent state.

Five gates regulate changes to the shared research state:

1. Build validation requires code changes to pass the test suite.
2. Contract validation prevents a model trained on one simulator build from being used with another.
3. Within-build comparison restricts model comparisons to a common environment and multiple random seeds.
4. Unseen-seed rechecking prevents delivery based solely on selection-seed performance.
5. Reality validation requires delivered models to play live games whose records are replayed in the simulator.

The fifth gate is especially important because the simulator and opponent models are themselves learned engineering artifacts rather than given infrastructure. Live games exposed 184 defects across game rules, card data, input encodings, client control, and opponent calibration. This result establishes that live deployment was not merely a final demonstration; it was an active source of environment correction.

The system’s process-level self-improvement is narrower and more concrete than unrestricted self-modification. The agents did not rewrite their own underlying LLM code. Instead, methodological failures were converted into persistent lessons, new gates, or standing rules. For example, a copied-game search appeared to improve first-place rate by 23.6 percentage points because the copied trajectories retained the simulator’s random-number state and therefore leaked future shop and combat outcomes. After reseeding each sample, the apparent improvement became $-0.65 \pm 1.14$ points. Similarly, winner’s-curse effects from selecting the best snapshot on evaluation seeds motivated the unseen-seed delivery gate. The implication is that the principal mechanism of self-improvement is institutional memory plus procedural constraint, not autonomous invention of a new research algorithm.

Figure 1: AutoTavern’s six-agent architecture and its training, research, and live-ladder loops.

TavernGym and the problem of environment fidelity

TavernGym is presented as the first full-content reproduction of the live Battlegrounds game suitable for training and evaluation. It implements all 978 game objects in patch 36.6.3, including minions, heroes, Hero Powers, trinkets, spells, and other cards. Its Rust engine exposes Python bindings, batched environments, exact save-and-restore, a player-view observation interface, and 245 discrete actions with an exact legality mask.

The simulator uses one seeded random generator, making a seed and action sequence exactly reproducible. Recruit and combat phases share an effect context, and card behaviors are represented through event hooks. This architecture is technically consequential because many Battlegrounds mechanics span phases and interact through triggers, attachments, deathrattles, auras, and persistent effects. The engine also uses an arena of serializable entities, facilitating state copying and lookahead.

Performance is high enough to support extensive PPO training and simulated evaluation: a single CPU core runs approximately 190–280 complete eight-player games per second, while late-game combats run at roughly 52,000–76,000 per second. The paper reports 375 rule-verification records and 220 confirmed rule decisions. Fuzzing checks state invariants after every random legal action, and live-game replay identifies discrepancies between the client and the simulator.

Figure 2: TavernGym’s Rust engine, Python interface, derived programs, and shared client encoding.

The paper’s strongest methodological point is that simulator fidelity cannot be established solely through internal tests. The live loop found 35 rule and card-behavior defects, 11 card-data or offer-rule defects, 73 input-encoding discrepancies, 51 client-controller defects, and 14 opponent or exam-calibration problems. Consequently, “faithful reproduction” should be interpreted as the result of repeated alignment against one live patch, not as a formally complete proof of semantic equivalence.

TavernAgent architecture and training

TavernAgent is an 18-million-parameter entity transformer. The observation is tokenized into the player’s minions, hand cards, tavern offers, pending choices, and opponents, with a maximum of 56 entity tokens plus a global token. Card representations combine identity embeddings, static attributes, and an embedding of the card’s rule text. The rule-text component allows rare or newly encountered cards to receive a semantically informed initialization rather than relying only on identity-specific data.

An eight-layer transformer with hidden dimension 384 and eight attention heads feeds three output heads: a pointer head over the 245 masked actions, a value head, and an auxiliary placement head. The policy is trained with PPO using terminal placement utility. Each training stage involves approximately 310 million decisions and 570,000 games, beginning from the preceding stage under a KL anchor. Training progresses from simplified rules to full content, player-view observations, first-place objectives, and increasingly calibrated opponent libraries.

The most important training intervention is opponent calibration. Rather than relying on self-play, the policy faces seven library seats whose boards are sampled from statistics of strong human players and recalibrated to resemble the opponents encountered in live deployment. From the same checkpoint and training budget, calibrated opponents increase main-exam first-place rate from 20.9% to 27.2%, and harder-exam first-place rate from 10.3% to 13.5%. This result supports the paper’s broader argument that opponent-model misspecification is a major source of proxy failure in multiplayer environments.

The delivered sequence of 23 versions improves substantially on the fixed simulator exams. TavernAgent’s average placement improves from 3.40 to 2.33 on the main exam, while first-place rate increases from 1.9% to 41.5% and top-four rate reaches 89.0%. On the harder exam, the final version reaches an average placement of 3.28, a 23.8% first-place rate, and a 73.2% top-four rate.

System or version	Main-exam average placement	First place	Top four
Random legal actions	7.97	0.0%	0.0%
Scripted bot	7.54	0.0%	0.5%
TA-1	3.40	1.9%	83.4%
TA-11	2.83	18.5%	85.5%
TA-23	2.33	41.5%	89.0%

The improvement is not attributable only to model scaling. An early comparison of 3.5M-, 8M-, and 18M-parameter networks shows a consistent but moderate benefit from larger models. The research process, simulator corrections, opponent calibration, curriculum, and deployment feedback account for the dominant longitudinal improvement.

Figure 3: TavernAgent’s entity-token representation, rule-text-enhanced card embeddings, transformer trunk, and three output heads.

Live-ladder performance

TavernAgent was deployed on the live ranked ladder through a controller that converted the client state into the simulator’s representation and executed the policy’s decisions. A human pressed the corresponding shortcut key, but the paper states that the human made no game decisions. The deployment covered one account, one patch, and 392 games across days 5–18.

The rating increased from 6,000 to 14,007 during the logged sequence and reached approximately 15,000 shortly afterward. Across games without client failures, the agent achieved an average placement of 3.22, a 16.4% first-place rate, and a 77.1% top-four rate. The latest versions, TA-19 through TA-23, achieved an average placement of 2.37, 32.6% first places, and 91.3% top-four finishes.

Figure 4: TavernAgent’s live-ladder rating trajectory over 392 games, including version intervals and deployment-induced pauses.

The paper makes a stronger comparison using the public leaderboard’s estimated-placement metric. In the final 20 leaderboard records, TavernAgent averages placement 1.60, with 80% first places and 90% top-four finishes, compared with 2.34, 59%, and 81% for the server’s ten highest-rated players collectively. The paper therefore claims that the system plays at the level of the server’s best players.

This comparison requires qualification. The leaderboard placements are inferred from rating changes rather than directly observed game-by-game outcomes, and the agent’s sample is only 20 records. The rating is also accumulated over a short period and depends on the number of games played. In addition, 96 of 388 games with recorded placements contained client failures and are excluded from the primary placement statistics. These limitations do not invalidate the live result, but they make “rivals top players” a claim about a short, deployment-specific evaluation window rather than a definitive estimate of long-run population strength.

The ladder nevertheless revealed failures not captured by the simulator. TA-10 nearly doubled the exam first-place rate of its predecessor, from 12.8% to 24.2%, yet its rating declined from 8,968 to 7,658 over 32 games. Investigation identified client failures and an early- to mid-game weakness hidden by the exam. The live loop consequently acted as a falsification mechanism for simulated progress.

TavernBench and the build-versus-play distinction

TavernBench evaluates whether general-purpose LLMs can directly perform the decisions made by TavernAgent. It contains 960 tasks across 40 scenarios and eight categories, including economy, combat, positioning, ordering, targeting, selection, and strategy. Labels are generated through TavernGym simulation, repeated combat rollouts, or play-out comparisons. All models receive textual game states and card rule descriptions, while TavernAgent receives its native structured engine state.

On the 368 tasks shared by all systems, TavernAgent achieves 91% accuracy, compared with 63% for the strongest directly prompted LLM, GPT-6 Astra. Claude Opus 5.5 and Claude Sonnet 5.5 reach 52% and 53%, respectively. TavernAgent is near-perfect on combat, positioning, ordering, and targeting, while its lower selection score of 64% and strategy score of 75% indicate that high-level choice among uncertain alternatives remains more difficult even for the specialized policy.

System	Play accuracy	Real-time accuracy	Median time
TavernAgent	91%	90%	0.03 s
GPT-6 Astra	63%	0%	2.4 s
Claude Sonnet 5.5	53%	0%	1.4 s
Claude Opus 5.5	52%	0%	6.2 s
GPT-6.1 Sol	46%	0%	2.0 s
Jev	37%	37%	0.26 s

Latency produces an even sharper separation. TavernAgent answers in a median of 0.03 seconds and retains 90% accuracy under the paper’s one-second decision budget. Every evaluated LLM except DeepSeek-V4.1-Flash falls to 0% real-time accuracy under that threshold; DeepSeek reaches 26%. Since Battlegrounds turns chain many dependent operations, per-decision errors compound rather than average independently. The practical result is therefore not merely that LLMs are slower, but that their latency and error rates are jointly incompatible with the action cadence of live play.

The benchmark also separates rule execution from strategic play. Frontier LLMs perform well on rule tasks—GPT-6 Astra reaches 95%, Claude Opus 92%, and Gemini 3.8 Flash 90%—but perform much worse on play-out decisions. This supports the paper’s explicit claim that reading a card’s rules is not equivalent to selecting the action that maximizes downstream placement.

Providing TavernGym as a tool closes much of the gap. GPT-6 Astra improves from 62% to 97% on the relevant short-term tasks, while other models also improve substantially. The remaining errors arise from illegal or mis-indexed actions, incomplete candidate search, and the combinatorial size of purchase sequences. This ablation suggests that a large component of the direct-LLM deficit is the absence of a fast, exact world model and action evaluator. However, the tools do not establish that an LLM can independently reproduce TavernAgent’s long-horizon policy: the tool calls remain budgeted, task-specific, and evaluated on constrained scenarios.

Figure 5: TavernBench accuracy by capability and the trade-off between play accuracy and per-decision latency.

What the ablations reveal about the learned policy

The paper measures the value of decision classes by randomizing, removing, or hiding one class at a time. Discover choices, Dark Gifts, and targeting decisions cost approximately 1.4–1.9 average placement positions when disrupted; trinket decisions cost about half a placement, and Hero Power decisions approximately one quarter. This result indicates that the policy’s performance is concentrated in a small number of high-leverage branching decisions rather than uniformly distributed across all actions.

The policy also exhibits a distinct hero profile from human players. Across 111 heroes, the correlation between TavernAgent’s hero-specific placement and public human statistics is only $r = 0.13$. The agent performs relatively well with heroes that humans rate as weak, such as Fungalmancer Flurgl, Trade Prince Gallywix, and Guff Runetotem, but underperforms relative to humans with heroes requiring multi-turn planning, including The Jailer, Alexstrasza, and Captain Eudora.

This weak correlation is important because it contradicts a simple interpretation of the agent as a general imitation of top human play. TavernAgent achieves strong aggregate performance while using a policy distribution that differs materially from human hero preferences and strengths. The authors also caution that the absolute comparison is confounded by the agent’s library opponents rather than human opponents, so the correlation measures relative profile similarity, not equivalent head-to-head hero strength.

Figure 6: Hero-specific average placement for TavernAgent compared with public human-player statistics.

Warband-growth analysis provides a related but limited diagnostic. The final agent matches approximately the top 1% of human players in warband strength through turn 9 and exceeds their reported attack-and-health totals by two to eight times from turn 11 onward. Because the curves are survivor curves and TavernAgent is evaluated against library seats rather than actual humans, the result measures growth dynamics rather than direct competitive superiority.

Figure 7: Warband strength across turns for TavernAgent versions and human-player statistics.

Limitations and open questions

The evaluation is concentrated on one game patch, one account, one deployment configuration, and a short live-ladder interval. Hearthstone Battlegrounds is continuously updated, so the simulator’s 978-object fidelity is patch-specific. The paper does not establish how rapidly AutoTavern can reconstruct a substantially changed ruleset or how much of its research memory transfers across patches.

The live comparison also has statistical and measurement limitations. The top-player comparison uses 20 leaderboard records and placements inferred from rating changes. Client failures affect 96 games, and the deployment still includes a human pressing keys, even though the paper reports that the human made no decisions. The calibrated opponent libraries are designed to resemble encountered opponents but remain proxies; consequently, high exam performance cannot by itself establish population-level human strength.

TavernBench is simulator-labelled and partly uses earlier TavernAgent versions as evaluators, particularly for strategic play-out tasks. This creates a potential dependence between the benchmark and the system being evaluated. The authors mitigate this through multiple labeling procedures and public human statistics for some tasks, but independent human-expert evaluation of the full benchmark would strengthen the claim that the benchmark measures game quality rather than agreement with the simulator or an earlier policy.

Finally, the study does not isolate the marginal contribution of LLM research agents from that of conventional engineering and reinforcement learning. AutoTavern uses PPO, a transformer policy, extensive compute, hand-defined interfaces, and human adjudication. The experiments demonstrate that LLM agents can organize and execute this pipeline under the specified framework; they do not show that LLMs independently discovered the complete algorithmic design or that the same process would outperform a comparably resourced human engineering team.

Open questions include whether reality-anchored memory and verification gates remain effective under rapid patch changes, whether independent evaluation environments can replace policy-generated labels, and whether the build-versus-play gap persists when LLMs receive persistent tool access, structured state, and recurrent planning over full turns rather than isolated benchmark tasks.

Conclusion

AutoTavern demonstrates a technically substantive form of LLM-mediated research automation. Its contribution is not a new RL algorithm but an integrated process for constructing, validating, correcting, and deploying a specialized game-playing system under imperfect simulation. The combination of typed research memory, verification gates, opponent calibration, and live-game replay enabled TavernGym and TavernAgent to progress from an initial 1.9% simulated first-place rate to 41.5%, while the deployed agent reached approximately 15,000 ladder rating and reported short-window results superior to the server’s ten highest-rated players.

The central empirical distinction is between constructing a capable policy and executing its decisions through language. TavernAgent reaches 91% on TavernBench play tasks at 0.03 seconds per decision, whereas the best directly prompted LLM reaches 63% and fails the one-second live-play constraint. The paper therefore supports a precise conclusion: under this setting, LLM agents are more effective as components of a tool-using research and engineering process than as direct controllers of a high-dimensional, real-time, stochastic game.