The PokeAgent Challenge: Competitive and Long-Context Learning at Scale

Abstract: We present the PokeAgent Challenge, a large-scale benchmark for decision-making research built on Pokemon's multi-agent battle system and expansive role-playing game (RPG) environment. Partial observability, game-theoretic reasoning, and long-horizon planning remain open problems for frontier AI, yet few benchmarks stress all three simultaneously under realistic conditions. PokeAgent targets these limitations at scale through two complementary tracks: our Battling Track, which calls for strategic reasoning and generalization under partial observability in competitive Pokemon battles, and our Speedrunning Track, which requires long-horizon planning and sequential decision-making in the Pokemon RPG. Our Battling Track supplies a dataset of 20M+ battle trajectories alongside a suite of heuristic, RL, and LLM-based baselines capable of high-level competitive play. Our Speedrunning Track provides the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system for modular, reproducible comparisons of harness-based LLM approaches. Our NeurIPS 2025 competition validates both the quality of our resources and the research community's interest in Pokemon, with over 100 teams competing across both tracks and winning solutions detailed in our paper. Participant submissions and our baselines reveal considerable gaps between generalist (LLM), specialist (RL), and elite human performance. Analysis against the BenchPress evaluation matrix shows that Pokemon battling is nearly orthogonal to standard LLM benchmarks, measuring capabilities not captured by existing suites and positioning Pokemon as an unsolved benchmark that can drive RL and LLM research forward. We transition to a living benchmark with a live leaderboard for Battling and self-contained evaluation for Speedrunning at https://pokeagentchallenge.com.

• The paper introduces a benchmark that jointly stresses partial observability, adversarial reasoning, and long-horizon planning using two tracks in Pokémon games.
• The study employs diverse methodologies including RL, LLM-based approaches, and hybrid systems, revealing that specialized RL and search methods outperform LLMs in competitive scenarios.
• The research underscores the importance of modular harness design and efficiency metrics, offering critical insights for advancing embodied, long-context AI systems.

The PokeAgent Challenge: A Large-Scale Benchmark for Competitive, Partially Observable, and Long-Horizon AI Reasoning

Motivation and Benchmark Design

The PokeAgent Challenge (2603.15563) addresses a central gap in the current AI ecosystem: the absence of standardized benchmarks that jointly stress partial observability, strong game-theoretic reasoning, and long-horizon planning in realistic environments. Existing benchmarks typically isolate these axes—e.g., games emphasize adversarial reasoning but in short or fully observable settings, while open-ended environments test exploration in the absence of strategic adversaries. Pokémon, with its exponentially large, partially observed state space, dynamic competitive metagames, and visual RPG environments, uniquely combines these challenges.

The challenge introduces two complementary tracks:

1. Competitive Battling Track: Two-player, zero-sum Pokémon battles on Pokémon Showdown, invoking partial observability (hidden information in team construction and movesets), stochastic transitions, and adversarial reasoning.
2. Speedrunning Track: Long-horizon, sequential decision-making in Pokémon Emerald, evaluated via completion rates and wall-clock time across decomposed game milestones, demanding long-context memory, robust planning, and visual perception.

Both tracks deploy standardized infrastructure, harmonized baselines (RL, LLM, hybrid), and large curated datasets. Importantly, the evaluation is designed to decouple agent architecture/model capability from harness/es scaffold, allowing for controlled comparative experiments, especially critical for LLM-based agents in embodied tasks.

Environment Complexity and Evaluation Protocols

The battleground for agent evaluation is both theoretically and practically challenging. The combinatorial size of the Pokémon battle state space is extreme, with the paper providing a detailed derivation showing, for example, $\sim 10^{564}$ possible states for Gen 9 OU (Figure 1 and state-space tables).

Figure 1: Cumulative usage distributions reveal that a small set of Pokémon/moves dominate competitive play, yet the combinatorial team space remains formidable.

Agents are evaluated with experiment-trackable, reproducible leaderboards. In the battling track, a public leaderboard tracks Full-History Bradley-Terry (FH-BT) skill ratings, Glicko-1, and GXE (expected win rate versus random opponents) on a private agent-only Showdown server, supporting multiple competitive formats (Gen 1 OU, Gen 9 OU), and time-constrained as well as 'extended timer' variants for LLMs.

Figure 2: RL baselines approach strong human performance, LLMs lag except when scaffolded with specialized harnesses.

For the speedrunning track, agents are evaluated based on progression through 15 milestones, wall-clock time, and sample efficiency (number of agent steps), formalized as an episodic MDP. The environmental setup provides only limited, noisy vision-state, requiring agents to handle perception and partial observability akin to real human players.

Figure 3: Milestone-based breakdown for Pokémon Emerald speedrunning, emphasizing the nonlinear exploration dependencies.

Baseline Architectures and Agent Capabilities

Competitive Battling Track:

Baselines span heuristic bots, RL agents (Metamon), and LLM-based agents with search harnesses (PokeChamp). The RL baselines (e.g., Transformers trained via large-scale offline RL) leverage >22M battle trajectories (4M human, 18M synthetic self-play), representing skill levels up to near-top human players. LLM-based agents employ structured text generation of game state, domain knowledge enrichment (usage statistics, opponent prediction), and protocolized depth-limited minimax search.

Advanced search-based systems (e.g., Foul Play) integrate root-parallelized MCTS with custom battle engines, employing innovations like "damage roll grouping" to achieve tractable lookahead in the face of combinatorial complexity.

Figure 4: Foul Play's MCTS architecture with damage roll clustering enables efficient deep search.

Figure 5: Set prediction pipeline leverages observed battle signals to infer hidden opponent configuration.

Speedrunning Track:

RL, VLM-LLM harnesses, and hybrid methods are compared under common instrumentation and evaluation. Baselines include:

• A multi-agent orchestration system coordinating subagents for context management, pathfinding (A*), memory, battle reasoning, and gym-specific puzzles
• LLM and VLM backends for perception and tool invocation
• Harnesses for external tool use, memory compaction, and reflection.

Rigorous evaluation reveals that, absent sophisticated harnesses, even frontier VLMs (including GPT-5, Gemini variants, Claude) fail to achieve nontrivial progression.

Figure 6: Model families' wall-clock completion times and cost for baseline harnesses; Gemini 3 Flash fastest, but cost/sample efficiency varies.

Empirical Results and Notable Algorithms

Battling Track

Competition and baseline results demonstrate:

• RL and search-based agents decisively outperform LLMs in partial information, adversarial settings. RL agents trained via large-scale offline data and fine-tuning (e.g., PA-Agent) and search agents (Foul Play; root-parallel MCTS) consistently secured top leaderboard positions and tournament wins.

  Figure 7: Organizer and participant ratings, showing clear stratification between specialist and generalist methods.

  

  Figure 8: Tournament standings reveal the edge of specialist approaches in both formats.

• Key architectural innovations included iterative offline RL with dynamic data weighting, curriculum-based two-phase training (Team Q), and tailored optimizer/activation function pairings (4thLesson's Kron optimizer + AID for plasticity).

  Figure 9: The Kron optimizer plus AID mitigates plasticity loss/instabilities in RL training.

Speedrunning Track

• Pure RL remains sample-inefficient for true long-horizon RPG play, but hybrid methods unlock scalability. The leading method, Scripted Policy Distillation (SPD), synthesizes LLM-generated subgoals and policies as exploration priors, followed by distillation into neural networks and RL refinement, yielding the only near-human-score run (40:13).

  Figure 10: SPD pipeline—LLM task decomposition, scripting, and policy distillation via expert action supervision.

• Harness design is paramount: Model performance is bottlenecked by harness sophistication—CLI-coding agent scaffolds are unable to maintain task coherence over thousands of steps, whereas specialized modular harnesses (as in PokeAgent) enable successful runs.
• Action sample efficiency versus wall-clock tradeoff: The Deepest agent completed with the fewest steps (most sample-efficient), but slower reasoning increased overall wall time.

  Figure 8: Efficiency-frontier; Deepest achieves action-minimal run.

Failure Modes and Benchmark Orthogonality

PokeAgent exposes failure modes rarely observed in standard LLM evaluation: "panic behavior," memory corruption cascades, computational paralysis (recursive reasoning loops blocking timely action), and strategic overcommitment. Chain-of-thought (CoT) visualizations synchronize agent reasoning with game state and uncover LLM pathologies.

Figure 11: Qwen model entering a computational paralysis loop during competitive play, evident in reasoning traces.

Critically, empirical BenchPress matrix analysis confirms that Pokémon battling skill is nearly orthogonal to summary performance on established QA/math/coding benchmarks, as no combination of standard LLM benchmarks predicts Battling GXE scores (max Spearman ρ ≈ 0.77, but mean |ρ| = 0.45, and SVD rank-2 structure fails to explain >70% of GXE variance seen).

Practical and Theoretical Implications

Practical:

• The large, open-source datasets (20M+ battles, 200K+ teams), modular harnesses, and multi-agent orchestration systems directly support future research in competitive, partially observed, and embodied AI.
• Standardization of evaluation protocols and task decomposition is poised to drive reproducible, comparable progress across LLM, RL, and hybrid agents.
• Action and wall-clock efficiency measurements are relevant for deploying real-time and time-critical AI agents.

Theoretical:

• The near-orthogonality of Pokémon performance to canonical NLP and contextual reasoning benchmarks demonstrates the inadequacy of current model benchmarks for measuring strategic sequential reasoning with latent state and adversarial response.
• The evidence supports the "LLMs as prior, RL as refinement" paradigm for embodied, long-context tasks.
• The necessity and impact of harnesses—versus raw model capability—on embodied OOD environments offers insight for future agentic evaluation and safety discussions.

Future Directions

Open challenges highlighted include:

• VLM-SLAM: Robust spatial consistency and action localization demands integration of vision-LM outputs with SLAM-like perceptual frameworks, a major bottleneck for RPG/ALE-style environments.
• LLM–RL performance gap: Designing hybrid or improved LLM architectures to approach RL sample efficiency in partial-observation, adversarial domains is a priority.
• Open-source full-game completion: Achieving human-level speed and robustness in long-context, open-source models would democratize research and evaluation.
• Planning efficiency: Sample efficiency, navigation, and obstacle avoidance are under-explored in current agent designs.

Conclusion

The PokeAgent Challenge establishes a scalable, actively maintained benchmark that exposes fundamental weaknesses and research opportunities for both RL and frontier LLM-based agents on challenging, high-dimensional sequential decision-making tasks. Empirically, specialist RL and search methods continue to dominate in competitive, partially observed domains, while LLMs play an essential role as priors and for task decomposition. The benchmark's living deployment ensures continued relevance and supports rigorous, reproducible advancement in long-context agentic AI research.

(2603.15563)