Gemini Plays Pokémon (GPP) Framework

Updated 3 July 2026

Gemini Plays Pokémon (GPP) is a framework that employs advanced foundation models and modular memory to achieve autonomous gameplay across various Pokémon game modes.
It integrates dynamic sub-agent architectures and human-in-the-loop refinement, transitioning towards emergent self-improvement for complex strategic tasks.
Empirical results demonstrate lossless game completions and competitive performance, establishing a blueprint for continual learning in high-complexity environments.

Gemini Plays Pokémon (GPP) refers to a framework and set of experiments leveraging foundation models—centered on the Gemini series of LLMs—for embodied, high-skill play of Pokémon games across mainline RPGs, competitive battle simulators, and card games. GPP incorporates foundation model harnesses, modular memory systems, and adaptive sub-agent architectures to achieve milestone-firsts in autonomous single-player game completion (Blue, Yellow, Crystal), competitive human-level play (Showdown/OU), and emergent self-improvement. The GPP research program unifies advances in harness modularity, continual agent refinement, and large-scale representation learning, forming a generalizable blueprint for next-generation, adaptively self-improving artificial agents in high-complexity, long-horizon environments (Karten et al., 11 May 2026).

1. Foundation Model Harnesses and Agentic Architectures

At its core, GPP employs a foundation model (Gemini 2.5 Pro/3 Pro variants) interfaced with minimal, pixel-based and symbolic state extractors—image frames and tile maps for mainline RPGs, JSON state for competitive battles. The agent harness $\mathcal{H}$ organizes the interaction loop, wrapping the model with dynamically editable components:

System Prompt $p_t$ : Task-level plan guiding the model's decision process.
Sub-agents $\mathcal{G}$ : Named modules for specialized reasoning (e.g., battle strategy, navigation).
Skills $\mathcal{K}$ : Reusable routines or code primitives (e.g., press_buttons, pathfinders).
Memory $\mathcal{M}$ : Persistent, timestamped entries recording trajectory, hints, or state.

The harness exposes a meta-tool API that enables both humans and the agent to define, delete, and update agents, skills, and memory in real time. At each time step, the orchestrator selects between direct action, skill invocation, or sub-agent delegation, with sub-agent calls managed via concise schema-prompts to minimize inference overhead and maximize actionable context per API invocation (Karten et al., 11 May 2026).

Initial GPP deployments relied on live human-in-the-loop refinement: researchers monitored streams (e.g., Twitch), inspecting logs and performance for failure signatures (e.g., navigation stalls, illogical battle decisions). Refinement cycles occurred at regular intervals (e.g., every 2,000 turns), where CRUD (create, read, update, delete) operations were applied to sub-agents, skills, and memory modules via the meta-tool API. Typical refinements included specializing the battle strategy prompt, segmenting logic into master-agents, and codifying repetitive navigational or puzzle routines.

As the harness matured, the need for human curation diminished—late-stage runs saw the Gemini agent itself defining new skills (e.g., reusable menu navigator routines) and tactical memory facts (e.g., puzzle truth tables) based on long-context recall. These emergent self-improvement signals—independent modularization, memory update rules, prompt optimization—served as foundational evidence for formalizing the Continual Harness architecture: an automated, reset-free refinement loop where Gemini alternates between environment interaction and prompt/harness self-editing using its own trajectory data (Karten et al., 11 May 2026).

3. Technical Description of Key Agent Modules and Learning Protocols

GPP generalizes core advances from recent LLM and RL-based Pokémon agents, formalizing them as follows:

Multi-agent skeleton: Inspired by the PokéAI architecture, a three-layered framework (Planning, Execution, Critique) operates in closed-loop, with memory-augmented LLM agents exchanging JSON-encoded state, task decomposition, execution logs, and post-hoc validation. This organization supports robust progression through RPG environments as well as real-time interaction with vision-based state extraction (Liu et al., 30 Jun 2025).
Battle micro-agent: Within Execution, specialized modules automate turn-level battle decisions, combining LLM-based move/action selection with hard-coded heuristics (type-, HP-, item-based rules) and stochastic action exploration. Empirical win rates approach human baselines (80.8% vs. 86.0%), and a strong positive correlation ( $r\approx 0.78$ ) is observed between language modeling ability (LLM Arena score) and tactical performance (Liu et al., 30 Jun 2025).
Retrieval-augmented reasoning: Drawing from PokéLLMon, state context is supplemented with top- $k$ knowledge base entries (type charts, move effects) indexed via dense vector embeddings, enhancing both strategic coherence and factual correctness (Hu et al., 2024). Retrieved information is injected at each decision point.
In-context Reinforcement Learning (ICRL): The agent incorporates recent examples of (state, action, feedback) as few-shot prefixes, using heuristically assigned rewards (e.g., HP delta, status effectivity) to simulate a policy gradient update in the LLM context window. This process biases action generation toward historically high-reward choices without requiring full model fine-tuning (Hu et al., 2024).
Consistent action generation: To mitigate degenerate oscillatory behavior (e.g., panic-switching), GPP employs self-consistency voting over multiple LLM samples per decision step, returning the modal action as the final output (Hu et al., 2024).
Three-stage RL pipeline: For competitive Showdown play, a transformer-based policy is trained via behavior cloning on reconstructed first-person trajectories, transitioned to offline RL with advantage-filtered actor-critic updates, and fine-tuned with synthetic self-play data to surpass baseline LLM-agent benchmarks (Grigsby et al., 6 Apr 2025).

4. Empirical Results and Analysis

Mainline RPGs

GPP achieved complete, lossless runs on Pokémon Blue, Yellow (Hard, Legacy variant), and Crystal using iterative harness refinement. Button-press cost and wall-clock efficiency improved by 30–50% across runs strictly from improvements in the modular harness without curated knowledge or hand-crafted algorithmic tools (Karten et al., 11 May 2026). Table: Key results.

Game	Model Variant	Lost Battles
Pokémon Blue	Gemini 2.5 Pro	0
Yellow Legacy	Gemini 3 Pro + tools	0
Crystal	Gemini 3 Pro + tools	0

Battle and Card Simulations

In battle simulations (Showdown OU/Gen 1–4), sequence models outperform recent LLM-agent and heuristic baselines, climbing to the top 10% of ladder players (GXE up to 80%, peak global rank #31 Gen 1 OU), validating the efficacy of offline RL with continual replay retraining (Grigsby et al., 6 Apr 2025).
For the Pokémon Trading Card Game, modular harness ablation reveals strong sensitivity to legal-action masking and history window design. Gemini 3.1 Pro massively outperforms smaller LLMs (Glicko-2 $\mu=1854$ vs. $1237$ for GPT-5.4 Nano). However, no tested self-evolution harness upgrade yielded monotonic policy improvements, highlighting the challenges of stable, strategy-level self-adaptation (Hua et al., 28 May 2026).

5. Policy Generalization and Methodological Challenges

Benchmarking in the competitive VGC domain reveals the growth in combinatorial complexity of team configuration space ( $|\mathcal{C}|\approx10^{139}$ ), inducing extreme strategic context dependence and cyclic metagame phenomena. Despite strong single-team expert performance via BC and RL (e.g., BCFP ELO 1768), robustness degrades rapidly with team set size, and current RL and LLM-hybrid agents fail to generalize to out-of-distribution team compositions with high confidence. The branching factor per turn ( $p_t$ 0) and partial observability ( $p_t$ 1 information sets) compound the need for continual, context-aware adaptation (Angliss et al., 12 Jun 2025).

6. Future Directions and Open Challenges

Key recommendations for advancing the GPP architecture and proximal research include:

Meta-learning and contextual policy adaptation to dynamically embed team archetypes and strategy clusters.
Hybrid symbolic–LLM approaches for marrying high-level planning (weather, speed control) with low-level reactive policy.
Continual Harness automation where episode resets are obviated, and process-reward co-learning directly couples model and harness updates in situ (Karten et al., 11 May 2026).
Curriculum-driven and regularized multi-task RL to progressively expose the agent to increasingly diverse strategic environments, mitigating brittleness to unseen states.
Empirical game-theoretic augmentation employing scalable population-based methods (e.g., PSRO, fictitious play) to sample policy and counter-policy dynamics in high-dimensional meta-games.

A plausible implication is that tight integration of memory, modular skill/harness editing, and prompt-driven LLM orchestration will remain foundational for scaling general-purpose embodied agents to the full complexity of Pokémon and analogous interactive environments. While GPP has set empirical benchmarks in lossless RPG completion and hybridized LLM–RL agent play, fully automated continual self-improvement and robust, out-of-distribution generalization remain active technical frontiers (Karten et al., 11 May 2026, Liu et al., 30 Jun 2025, Angliss et al., 12 Jun 2025, Hu et al., 2024, Grigsby et al., 6 Apr 2025, Hua et al., 28 May 2026, Sarantinos, 2022).