Continual Harness: Online Adaptation for Self-Improving Foundation Agents

Abstract: Coding harnesses such as Claude Code and OpenHands wrap foundation models with tools, memory, and planning, but no equivalent exists for embodied agents' long-horizon partial-observability decision-making. We first report our Gemini Plays Pokemon (GPP) experiments. With iterative human-in-the-loop harness refinement, GPP became the first AI system to complete Pokemon Blue, Yellow Legacy on hard mode, and Crystal without a lost battle. In the hardest stages, the agent itself began iterating on its strategy through long-context memory, surfacing emergent self-improvement signals alongside human-in-the-loop refinement. Continual Harness removes the human fully from this loop: a reset-free self-improving harness for embodied agents that formalizes and automates what we observed. Starting from only a minimal environment interface, the agent alternates between acting and refining its own prompt, sub-agents, skills, and memory, drawing on any past trajectory data. Prompt-optimization methods require episode resets; Continual Harness adapts online within a single run. On Pokemon Red and Emerald across frontier models, Continual Harness starting from scratch substantially reduces button-press cost relative to the minimalist baseline and recovers a majority of the gap to a hand-engineered expert harness, with capability-dependent gains, despite starting from the same raw interface with no curated knowledge, no hand-crafted tools, and no domain scaffolding. We then close the loop with the model itself: an online process-reward co-learning loop, in which an open-source agent's rollouts through the refining harness are relabeled by a frontier teacher and used to update the model, drives sustained in-game milestone progress on Pokemon Red without resetting the environment between training iterations.

• The paper presents a reset-free, online in-context learning framework that allows agents to continually refine their harness without external resets.
• It details a dual-loop methodology where an inner agent loop and an outer refiner loop collaboratively update system prompts, sub-agents, skills, and memory in real time.
• Experimental results demonstrate that this approach nearly matches hand-engineered performance in complex, partially observable environments while reducing cumulative costs.

Continual Harness: Reset-Free Online Adaptation for Self-Improving Foundation Agents

Introduction and Motivation

While agentic harnesses—scaffolding layers that wrap foundation models (FMs) with tools, memory, and modular controllers—have become standard for autonomous software agents, embodied agents in long-horizon, partially observable settings have lacked a comparable paradigm. Previous solutions such as human-in-the-loop refinement for LLM agents playing complex games (notably Gemini Plays Pokémon, GPP) were able to achieve the first documented FM completions of entire Pokémon RPGs by iteratively updating the harness structure based on observed failures. However, such processes were labor-intensive and fundamentally episodic, requiring human oversight and frequent environment resets, limiting the scalability and autonomy of agentic improvement.

The paper "Continual Harness: Online Adaptation for Self-Improving Foundation Agents" (2605.09998) addresses this by introducing Continual Harness (CH): a reset-free, automated, online in-context learning framework that enables embodied FMs to self-improve their harness mid-episode, without resets or external engineering. This work fills a critical gap, allowing agent architectures developed for coding and software tasks (e.g., Claude Code, OpenHands) to extend to embodied domains with rich, multi-stage, partially observable dynamics such as RPGs.

Figure 1: Continual Harness automates agentic harness refinement, enabling joint adaptation of model weights and harness state in online, reset-free mixed-initiative settings.

Continual Harness: System and Methodology

Harness Components

CH formalizes the agentic harness $\mathcal{H}$ as a tuple $(p,\,\mathcal{G},\,\mathcal{K},\,\mathcal{M})$, comprising:

• System prompt ($p$)
• Sub-agents ($\mathcal{G}$): task-specific controllers (e.g., navigation, battle)
• Skills ($\mathcal{K}$): reusable action primitives and executable routines
• Memory ($\mathcal{M}$): persistent knowledge and episodic facts

Harness updates occur via meta-tools exposed within the agent's context (e.g., define_agent, run_code, process_memory).

Reset-Free Alternating Refinement

CH operates in two coupled online loops:

1. Inner (Agent) Loop: The FM agent observes environment input, reasons over the current harness, and acts.
2. Outer (Refiner) Loop: Every $F$ steps (after a warmup), a Refiner analyzes the recent trajectory for failure signatures (e.g., tool-call errors, navigational loops, idle objectives) and applies composite CRUD-style edits to $p$, $\mathcal{G}$, $\mathcal{K}$, and $(p,\,\mathcal{G},\,\mathcal{K},\,\mathcal{M})$0. Importantly, both the agent and Refiner share the same core FM and issue edits via the same meta-tool API.

Unlike prior methods (e.g., GEPA) that require episode resets between prompt or harness updates, CH updates the harness in situ, so long-horizon information is retained and harness improvement compounds throughout the episode.

Figure 2: Methodology overview: the agent and refiner alternately update the harness mid-trajectory, and CH can be coupled with a process-reward-driven model update for co-learning.

Continual Model-Harness Co-Learning

For open-source models, CH supports a joint online learning regime:

• DAgger+PRM Training: At each iteration ($(p,\,\mathcal{G},\,\mathcal{K},\,\mathcal{M})$1 steps), a paired process reward model scores trajectory windows; low-reward shards are relabeled by a strong FM teacher (e.g., Gemini-3.1-pro) for soft SFT, driving model policy improvement. The emulator state, harness state, and model weights persist across iterations—yielding true reset-free adaptation.

Experimental Results and Quantitative Analysis

GPP Human-In-the-Loop Performance and Emergent Harness Refinement

The Gemini Plays Pokémon project established the feasibility of LLMs completing complex RPGs using manually iterated harnesses. Notably, emergent behaviors included skill composition, adaptive sub-agent construction, and dynamic prompt rewriting. Instrumentation showed that persistent harness refinement, not static engineering, was the key to overcoming hard exploration and multi-step reasoning bottlenecks in late-game content.

Figure 3: CRUD operations on skills and sub-agents in Yellow Legacy—updates are highly concentrated, recurrent, and non-uniform, indicating active, online refinement and specialization.

Figure 4: Decision graph metrics for a key battle-agent prompt over time during Elite Four—structural complexity cycles, with rewrites absorbing logic into new sub-agent compositions.

Continual Harness Brings Hand-Engineered Expert Performance Within Reach

Direct comparison on Pokémon Red and Emerald demonstrates that from-scratch CH, starting with no domain knowledge, recovers a majority of the efficiency gap with a competition-grade hand-engineered harness, drastically reducing cumulative button-press cost to milestones across diverse environments.

Figure 5: Milestones reached versus button presses in Red and Emerald. CH runs close the gap to hand-engineered harnesses, with compound gains when building on refined harnesses.

Notably, continuous refinement outperforms frozen or reset baselines, demonstrating that in-episode learning compounds and bootstraps future runs.

Figure 6: Pareto frontier on Emerald: CH achieves strict cost-completion Pareto dominance with sufficiently capable models (Pro), but is ineffective below a capability threshold (Flash-Lite).

Open-Source Co-Learning: Reset-Free, Data-Efficient Policy Improvement

Open-source models (Gemma-4 variants) seeded with SFT and offline GRPO (process reward supervision) show no milestone gains in isolation. Only with the full online CH loop (DAgger+PRM, trajectory relabel and SFT) does in-game performance advance, even from mid-game checkpoints, confirming that reset-free, self-improving harnesses translate into sustained, data-efficient policy optimization.

Figure 7: Reset-free DAgger+PRM training drives milestone progression in open-source models, both from game start and mid-game initialization, confirming the power of live model-harness co-adaptation.

Mechanistic Attribution: Online Skill and Harness Self-Improvement

Direct measurement of navigational skill cost relative to a Dijkstra oracle documents that CH closes in on optimal path costs as trajectories progress, showing tangible in-loop skill repair and retention effects that are impossible under reset-based regimes.

Figure 8: Pathfinding skill mechanism—path cost relative to Dijkstra oracle improves steadily during online refinement, with CH conditions invoking skills far more frequently and effectively.

Implications, Limitations, and Future Directions

Continual Harness delivers reset-free, automated harness refinement for embodied agents, matching most hand-created expert harness efficiency without domain-specific engineering, curated tools, or task-specific scaffolding. This positions CH as a foundational paradigm for scalable, long-term, self-improving embodied FMs, with two-tiered online loops—agent/refiner and model/harness—enabling continual adaptation.

Practically, CH will be indispensable for long-running simulators, real-world robotics, interactive agents, and any domain where environment resets are costly or infeasible. It enables structured, curriculum-free transfer and continual lifelong learning with realistic, non-stationary distributional shifts.

There is a clear capability floor: insufficiently powerful models (e.g., Gemini Flash-Lite, undertrained open-source checkpoints) cannot yet bootstrap effectively, and supervisory signals must be strong and contextually rich. Moreover, the effectiveness of harness reuse across episodes depends on the agent's willingness to leverage inherited memory and sub-agents, suggesting that more explicit transfer priors and deletion/retention heuristics are important future refinements.

Theoretical extensions will likely formalize the interaction between online harness adaptation and long-horizon credit assignment, connect CH with developments in in-context RL, and spawn a new class of self-improving, tool-building, life-long learning agents.

Conclusion

Continual Harness (2605.09998) constitutes a significant advance in agent infrastructure for embodied FMs, demonstrating that reset-free online harness refinement with coupled process-reward model co-learning yields near-expert performance in highly complex, partial information domains without human supervision or domain-specific code. By automating both agentic harness construction and continual adaptation, CH establishes a paradigm for scalable, self-improving, long-lived agents—paving the way for robust, flexible, multi-domain embodied AI.