LLM Pokémon League: Benchmarking LLM Battles
- LLM Pokémon League is an emerging benchmark ecosystem where language models act as trainers, battle agents, and planners in diverse Pokémon scenarios.
- The framework spans competitive battling, RPG control, and trading card game challenges, evaluating strategic reasoning under partial observability and long-horizon planning.
- Hybrid approaches combining reinforcement learning, search, and simulation backends demonstrate improved win rates and decision-making in complex, competitive settings.
Searching arXiv for the cited Pokémon/LLM papers to ground the article in current records. First, I’ll look up the exact “LLM Pokémon League” paper and then the main adjacent benchmark/agent papers. LLM Pokemon League denotes an emerging research agenda and benchmark ecosystem in which LLMs are instantiated as Pokémon trainers, battle agents, planners, or multi-agent systems and are evaluated through direct competition, laddering, tournament play, meta forecasting, and long-horizon gameplay. In the narrow sense, the phrase names a specific single-elimination tournament framework in which frontier models build teams, choose actions, and emit natural-language rationales (Yashwanth et al., 3 Aug 2025). In the broader literature, it refers to a family of Pokémon-based evaluation settings—especially Pokémon Showdown battling, emulator-backed RPG control, and Pokémon TCG play—that stress partial observability, strategic reasoning, long-horizon planning, and adaptation under competitive pressure (Karten et al., 16 Mar 2026, Karten et al., 6 Mar 2025).
1. Scope and benchmark rationale
Pokémon has been adopted as a benchmark substrate because it combines several capabilities that are usually evaluated separately. The PokeAgent Challenge frames the domain around two complementary tracks: a Battling Track for strategic reasoning under partial observability and a Speedrunning Track for long-horizon sequential decision-making in the Pokémon RPG (Karten et al., 16 Mar 2026). PokéChamp motivates the same choice from the battle side: Pokémon is two-player, zero-sum, turn-based, partially observable, strategically deep, and stochastic, with a first-turn state space stated to be on the order of in Gen 9 OU (Karten et al., 6 Mar 2025). The PokeAgent Challenge further estimates a Gen 9 OU battle-state space of approximately and a Gen 1 OU battle-state space of approximately , reinforcing the view that Pokémon is a compact but nontrivial decision-making environment rather than a narrow domain toy (Karten et al., 16 Mar 2026).
Within this landscape, “league” has several distinct meanings. It can mean a head-to-head tournament among LLM trainers, as in the eight-model single-elimination system of “A Multi-Agent Pokemon Tournament for Evaluating Strategic Reasoning of LLMs” (Yashwanth et al., 3 Aug 2025). It can mean a public benchmark ecosystem with ladder ratings, tournaments, baselines, datasets, and live leaderboards, as in the PokeAgent Challenge (Karten et al., 16 Mar 2026). It can also mean a broader systems view in which battle agents, RPG agents, and meta simulators are assembled into a common evaluation substrate.
| Domain | Representative systems | Primary emphasis |
|---|---|---|
| Competitive battling | PokeLLMon, PokéChamp, PokeAgent Battling Track | partial observability, tactical play, rating-based comparison |
| Meta forecasting | ABC-Meta | post-ban ecosystem prediction |
| RPG control | PokeRL, PokéAI, PokeAgent Speedrunning Track | long-horizon planning, control, memory, tool use |
| Trading card game | PTCG-Bench | harness-aware full-game decision-making and self-evolution |
This breadth matters because the literature does not treat LLM Pokemon League as only a battle bot problem. It is an umbrella for competitive evaluation, strategic simulation, and agent-systems research across multiple Pokémon modalities.
2. Competitive battling as the canonical league arena
The most mature line of work treats Pokémon Showdown battles as the primary arena for LLM competition. PokeLLMon is the earliest explicit demonstration that a language-model battle agent can operate directly on battle text with competitive intent. Its design combines in-context reinforcement learning, knowledge-augmented generation, and consistent action generation; in eighth-generation random battles on Pokémon Showdown it achieved 48.57% win rate against random ladder humans and 56.00% against an invited experienced human player, while its strongest bot-facing ablation, self-consistency with , reached 64.00% win rate against the heuristic bot (Hu et al., 2024). The system’s main technical lesson is that raw LLM prompting is not enough: battle feedback, external type and move knowledge, and multi-sample action stabilization are what turned GPT-4 from a 26.00% bot-baseline battler into a competitive agent.
PokéChamp moves from prompt-centric play to search-integrated play. It models Pokémon battle as a partially observable Markov game , then uses LLMs to replace three classical search modules: player action sampling, opponent modeling, and value function estimation at search cutoffs (Karten et al., 6 Mar 2025). In Gen 9 OU, PokéChamp with GPT-4o achieved 84% win rate against Abyssal Bot and 76% against the previous best LLM-based bot, with a projected ladder Elo of 1300–1500, corresponding to roughly the top 30% to top 10% of human players. The paper’s importance for a league concept lies less in any single result than in its architecture: Pokémon becomes a benchmark for competitive search agents whose world knowledge and hidden-state inference are supplied by LLMs rather than hand-coded heuristics alone.
A more stripped-down line appears in “LLMs as Pokémon Battle Agents: Strategic Play and Content Generation,” which uses a custom simplified simulator rather than Pokémon Showdown. In that setup, Gemini-Flash (OFF) reached a 62% win rate and Gemini-Pro (OFF) a 71% win rate over 50 battles against a random player, while the paper reports that Gemini 2.5 Flash with chain-of-thought enabled maintained type-aligned move selection in 78% of decisions (Jain et al., 19 Dec 2025). These results are weaker as evidence about canonical competitive Pokémon, but they are useful for league design because they foreground structured JSON actions, latency, token use, and action-format compliance.
Taken together, these systems establish the battling side of LLM Pokemon League as a spectrum. At one end are prompt-engineered language agents. At the other are hybrid search agents in which LLMs supply priors and hidden-state reasoning inside an explicit game-theoretic scaffold. The literature consistently treats the second class as stronger.
3. League infrastructure, tournaments, and ratings
The strongest operationalization of a full league is the PokeAgent Challenge. It ran a NeurIPS 2025 competition with 100+ teams, 150+ submissions, 650+ Discord members, and 100K+ battles on the competition server, and then transitioned into a living benchmark hosted at pokeagentchallenge.com (Karten et al., 16 Mar 2026). Its Battling Track uses a dedicated Pokémon Showdown server for AI benchmarking, focuses on Gen 1 OU and Gen 9 OU, and supports both normal timers and an “Extended Timer” mode to separate strategic quality from inference latency. Its ranking methodology combines online metrics such as Glicko-1 and GXE with a primary benchmark metric, Full-History Bradley–Terry (FH-BT), fit over the full match history. Qualification in the competition required at least 250 battles for FH-BT-based advancement, and the finals were single-elimination best-of-99 tournaments.
The narrower “LLM Pokémon League” tournament paper offers a more interpretable but less statistically robust prototype. It uses eight zero-shot models—GPT-4.1, o4-mini, o3, Claude Sonnet 3.5, Claude Sonnet 3.7, Claude Sonnet 4, Gemini 2.5 Pro, and Gemini 2.5 Flash—in a single-elimination bracket, records team-building rationales and turn-by-turn justifications, and reports o4-mini as champion with a 3–0 record (Yashwanth et al., 3 Aug 2025). Its main contribution is not ladder calibration but comparative strategic phenomenology: most models converged on balanced human-style teams, whereas the winning model selected Kyogre, Groudon, Rayquaza, Lugia, Magnezone, and Ho-Oh and was described as exploiting weather synergy and legendary base-stat concentration.
PTCG-Bench extends league logic beyond mainline video-game battles. It formalizes agent behavior as
with expected performance depending jointly on model backbone and harness , and evaluates both fixed agents and self-evolving agents through full Pokémon Trading Card Game matches (Hua et al., 28 May 2026). Its primary metric is Glicko-2, supplemented by invalid action rate and tool calls. Among LLM agents, the best reported rating is Gemini 3.1 Pro at 1854 and the worst is GPT-5.4 Nano at 1237, a 617-point spread. Equally important are its harness ablations: the full harness achieves a 1726 rating mean with 3.3% invalid action rate, while the minimal harness falls to 1575 with 27.3% invalid action rate. For league methodology, this result is foundational: a leaderboard that does not standardize harness design may be ranking scaffolds as much as models.
4. Meta discovery and ecosystem forecasting
A distinctive strand of the literature treats league operation not as match playing but as ecosystem prediction. “A Framework for Predicting the Impact of Game Balance Changes through Meta Discovery” introduces the task ABC-Meta, “Analyzing Balance Changes on the Metagame,” in which the input is an approximation of the pre-change metagame and the output is the predicted top-40 Pokémon by usage after a simulated post-ban adaptation period (Saravanan et al., 2024). The framework has three components—a battle agent, a team-builder, and a simulation environment—and is evaluated on Smogon singles tiers OU, UU, NU, and PU. Its core insight is that metagame prediction is an iterative ecosystem simulation problem rather than a one-shot supervised prediction problem.
The system is technically modest on the battle side. The RL battle agent uses PPO with Generalized Advantage Estimation, a discrete action space of size 22, action masking, and the default poke-env state vector of size 10. The team-builder is more consequential. In the ABC-Meta setting it uses only information publicly available from Showdown data and, after grid search, settles on the scoring function
The environment then repeatedly generates 2500 teams, updates statistics every 1000 battles, and simulates 150,000 battles per month for 3 months, or 450,000 battles per scenario.
The headline result is strong on membership recovery and weak on fine-grained ordering. For Overlap between the discovered top-40 and the true post-ban top-40, the framework reaches 0.975 in OU, 1.000 in UU, 0.950 in NU, and 0.925 in PU, yielding 92.5% to 100% recovery of actual post-ban meta membership (Saravanan et al., 2024). Edit-distance deltas from the true meta are also substantially better than the naive baseline. By contrast, Spearman’s rho is low and statistically non-significant in all tested cases. The practical significance for an LLM Pokemon League is direct: if the goal is season forecasting, suspect-test analysis, or explaining likely consequences of bans, simulator-backed meta discovery is more reliable as a predictor of which Pokémon remain central than as a precise forecast of ranking order or exact usage shares.
The same paper is also explicit that this is not an LLM-native method. A plausible implication is that meta-aware league assistants will need a non-LLM simulation backend for ecosystem roll-forward and an LLM layer for explanation, reporting, and intervention proposal rather than relying on language modeling alone.
5. Long-horizon RPG agency and the non-battle league problem
The RPG side of LLM Pokemon League is organized around a different difficulty profile: low-level control, sparse rewards, memory, and subgoal persistence. PokeRL makes this explicit in Pokémon Red. It wraps the PyBoy emulator in a custom Gymnasium environment, adds per-map visited masks, anti-loop and anti-spam machinery, and a dense hierarchical reward design, and trains PPO agents on three early tasks: house exit, exploration to grass, and the first rival battle (Mudireddy et al., 12 Apr 2026). Its empirical findings are largely about substrate engineering. The visited-mask ablation increases average unique positions per episode from 34.2 to 48.1 and exploration coverage in Pallet Town from 12% to 41%. Anti-loop handling reduces loop episodes from 41.2% to 4.7%. Reported task performance reaches roughly 65% house-exit success after 150k timesteps, about 60% exploration-to-grass success by 500k timesteps, and around 50% win rate in the first rival battle after 500k timesteps. The paper’s larger claim is that loop handling, action restriction, movement abstraction, and explicit spatial memory are prerequisites for any more ambitious Pokémon agent.
PokéAI introduces a text-based multi-agent architecture for Pokémon Red with a Planning Agent, an Execution Agent, and a Critique Agent, each with its own memory bank and role (Liu et al., 30 Jun 2025). Only the battle module inside the Execution Agent is fully implemented and evaluated, but it already illustrates the league-relevant closed loop: planning from persistent context, execution via tools and emulator control, and verification before proceeding. In a fixed Mt. Moon wild-encounter setting, the battle AI achieves an average win rate of 80.8% across 50 encounters, compared with 86% for one experienced human player. The ablations are revealing: removing switching drops performance to 58.8%, and removing item use drops it to 32.6%. The paper also uses memory address 0xD057 to detect battle onset and reports model-specific playstyles, indicating that league standings could in principle be complemented by behavioral style signatures.
The PokeAgent Challenge turns this into a standardized competitive track. Its Speedrunning Track evaluates progress from the start of Pokémon Emerald to defeating Roxanne across 15 milestones, with completion percentage as the primary score and completion time as the secondary score among 100% runs (Karten et al., 16 Mar 2026). In the NeurIPS competition, 22 valid submissions were recorded and 6 achieved 100% completion. The winner, Heatz, used Scripted Policy Distillation and finished in 40:13; the second-place Hamburg PokéRunners finished in 01:14:43. One of the paper’s most important negative results is that without a harness, raw frontier VLMs achieve effectively 0% task completion. On the RPG side, therefore, the “league” is less a ladder of raw model intelligence than a comparison of orchestrated agent systems.
6. Limitations, misconceptions, and research trajectory
A recurring misconception is that LLM Pokemon League is primarily about prompting a single model harder. The strongest evidence runs in the opposite direction. In the PokeAgent Challenge Battling Track, all top-performing submissions employed RL or search-based methods rather than pure LLM approaches; in the Speedrunning Track, the top two submissions were RL-based, and the winning Scripted Policy Distillation system used LLMs to generate priors that were then distilled into neural policies and further improved with RL (Karten et al., 16 Mar 2026). This does not make LLMs irrelevant. It suggests that, in current practice, they function best as planners, priors, reflectors, or hidden-state reasoners inside a broader competitive system.
A second misconception concerns published strength claims. PokeLLMon’s “human-parity” result refers to 48.57% win rate on an eighth-generation random-battle ladder and 56.00% against one invited experienced human player, not elite tournament equivalence (Hu et al., 2024). PokéChamp is stronger, but its own framing is careful: projected Elo 1300–1500 in Gen 9 OU places it among roughly the top 30% to top 10% of human players rather than at championship level (Karten et al., 6 Mar 2025). PokéAI’s 80.8% battle result is likewise confined to 50 wild encounters in Mt. Moon, not adversarial PvP or full-game completion (Liu et al., 30 Jun 2025).
A third issue is benchmark fragility. The single-elimination LLM Pokémon League paper does not report repeated matches per pairing and contains an unresolved discrepancy between a curated pool of 60 Pokémon in the methodology and 30 species in the results section (Yashwanth et al., 3 Aug 2025). The simplified battle-and-content-generation paper uses a non-canonical three-Pokémon simulator and leaves several mechanics, including full damage specification and STAB modeling, unconfirmed or underspecified (Jain et al., 19 Dec 2025). PTCG-Bench makes this concern systematic by showing that harness variation alone can shift ratings by more than adjacent-model gaps (Hua et al., 28 May 2026).
Finally, simulator-backed ecological models should not be overinterpreted. ABC-Meta was tested only on four historical single-ban scenarios and shows low, statistically non-significant Spearman correlations even when overlap with post-ban top-40 membership is high (Saravanan et al., 2024). The paper explicitly cautions that it models a simulator-discovered ladder-like population meta rather than the full sociology of tournament preparation, scouting, imitation, and innovation.
Taken together, the literature suggests that LLM Pokemon League is best understood not as a single benchmark or a single agent architecture, but as a layered research program. Its enduring components are likely to be simulator-backed environments, explicit legality and harness control, rating-based public evaluation, trajectory logging, and hybrid agent designs in which LLMs contribute reasoning, memory, and explanation while search, RL, and domain-specific middleware carry much of the execution burden.