Jericho Test-Time Learning (J-TTL)

• J-TTL is a framework that challenges language agents to adapt internally across consecutive game episodes in text-based adventures.
• It integrates modular adaptation mechanisms like prompt mutation, memory updates, hyperparameter tuning, and evolutionary strategies without persistent external state.
• Empirical findings show that methods like EvoTest and JitRL offer superior sample efficiency and robustness compared to traditional RL and static prompt techniques.

Jericho Test-Time Learning (J-TTL) is a benchmark and framework for evaluating and driving progress in continual, test-time adaptation of language agents in interactive-fiction (IF) text games. The J-TTL paradigm places an agent in a single environment for multiple consecutive episodes, requiring episodic improvement through internal adaptation alone, as environment resets eliminate persistent external state. This setup exposes and quantifies the limitations of traditional adaptation, memory-based, and reinforcement learning approaches, and has driven the development and comparison of new modular, evolutionary, and meta-optimization-based test-time learning algorithms.

1. Problem Definition and Motivation

J-TTL formalizes the challenge of on-the-fly adaptation in partially observable Markov decision processes (POMDPs), as instantiated by text-based adventure games in the Jericho suite. At each step, the agent receives an observation, issues a natural-language command, and receives a reward. A session consists of $K$ back-to-back episodes; after each episode, the environment resets to its initial state. All cross-episode learning must be internal: the agent's configuration $\theta^{(e)}$ is updated via an adaptation rule $U$ based on the observed trajectory $\tau^{(e)}$, with $\theta^{(e+1)} = U(\theta^{(e)}, \tau^{(e)})$. The key learning objective is to maximize total or per-episode reward improvement across the sequence of episodes, as measured by metrics such as area under the curve (AUC), per-episode gain $\Delta R(e)$, and final score fractions (He et al., 15 Oct 2025).

J-TTL exposes two critical bottlenecks in existing adaptation strategies:

• Pure memory augmentation and prompt engineering plateau early, capturing little reward signal beyond initial explorations.
• Gradient-based reinforcement learning is expensive, exhibits high variance in sparse-reward settings, and often suffers catastrophic forgetting (Li et al., 26 Jan 2026, He et al., 15 Oct 2025).

2. Agent Architecture and Adaptation Mechanisms

The agent in the J-TTL paradigm is characterized by a modular configuration $\chi=(p, M, h, u)$, comprising the policy prompt $p$ (system prompt to a frozen LLM), deployment-time memory $M$, inference hyperparameters $h$, and tool-use routines $\theta^{(e)}$0. The adaptation mechanism after each episode utilizes episode transcripts to propose changes across all these axes:

• Prompt mutation: LLM-mediated revisions of $\theta^{(e)}$1 to encode new tactics, block previously failing patterns, or recommend effective schedules.
• Memory update: Logging coarse-grained (state, action, reward) triplets, extracting success and failure modes to guide future action selection.
• Hyperparameter tuning: Adjusting $\theta^{(e)}$2, e.g., sampling temperature or candidate set size, often based on measures of exploration or detected stagnation.
• Tool-use refinement: Updating $\theta^{(e)}$3, such as memory retrieval heuristics and feature extractors (He et al., 15 Oct 2025).

This holistic adaptation is realized by various frameworks—most prominently EvoTest and Meta-TTL—which evolve or meta-learn these components without weight updates to the underlying LLM. The adaptation rule $\theta^{(e)}$4 can thus be seen as an evolvable or learnable policy that proposes agent configuration updates based on a given trajectory (or history of trajectories).

3. Representative Algorithms: EvoTest, JitRL, and Meta-TTL

EvoTest

EvoTest employs a bi-level evolutionary process, dividing the agent into an Actor (executing the game) and an Evolver (analyzing the transcript and proposing new configurations). After each episode, the Evolver proposes $\theta^{(e)}$5 mutated "child" configurations (via prompt rewrites, memory updates, hyperparameter changes, tool refinements). Selection is performed using upper confidence bound (UCB) over observed returns:

$\theta^{(e)}$6

where $\theta^{(e)}$7 is the configuration's empirical mean return and $\theta^{(e)}$8 its trial count. Mutation, memory, and selection drive improvement over repeated episodes (He et al., 15 Oct 2025).

Just-In-Time Reinforcement Learning (JitRL)

Unlike population-based methods, JitRL maintains a non-parametric episodic memory of $\theta^{(e)}$9 triplets. At each timestep, relevant past transitions are retrieved using hybrid state/history embedding similarity and Jaccard lexical overlap. Action-value estimates $U$0 and baseline state-value $U$1 are computed from neighbors. A KL-constrained objective yields the closed-form policy update:

$U$2

with solution

$U$3

implemented as a logit adjustment on the frozen LLM (Li et al., 26 Jan 2026).

Meta-TTL

Meta-TTL (as instantiated for J-TTL) treats the adaptation rule (in this case, system prompt update) as itself subject to meta-optimization. A meta-agent LLM, parameterized by a meta-prompt $U$4, receives the actor's prompt and past trajectory history, emitting an adapted prompt for the next episode. Meta-training is formulated as a bi-level evolutionary process:

• Inner loop: per-task weighted AUC on game $U$5
• Outer loop: mutation/selection over pools of meta-prompts across diverse training tasks, using performance on both in-distribution and held-out games to shape prompt evolution (Lou et al., 1 Apr 2026).

4. Empirical Evaluation on J-TTL

Extensive experiments on Jericho J-TTL have compared EvoTest, JitRL, Meta-TTL, and other baselines including static prompt agents, memory agents, Reflexion (hand-crafted agent self-critique), AWM, and online gradient RL (GRPO). In a standardized protocol (multiple games, fixed hyperparameters, $U$6 episodes, max 60 steps), JitRL and EvoTest demonstrate superior sample efficiency, learning curves, and final scores.

Method	Library Score	Zork1 Score	Zork3 Score
Static	10.0 / 10	8.5 / 10	0.2 / 0
Memory	13.2 / 14	22.9 / 25	1.0 / 1
Reflexion	15.4 / 18	26.1 / 35	1.4 / 1
AWM	12.7 / 10	38.8 / 44	1.9 / 2
EvoTest	21.5 / 26	46.8 / 54	2.6 / 4
GRPO	13.6 / 11	16.2 / 10	1.1 / 2
JitRL	25.9 / 30	53.0 / 69	3.1 / 5

EvoTest achieves average AUC ≈ 0.47–0.50, outperforming prompt-evolution (≈0.34–0.36) and online RL (≈0.30), while static agents yield AUC ≈ 0.11–0.12. EvoTest also uniquely solves (wins) games such as Detective and Library, while all baselines fail to win any game within 50 episodes (He et al., 15 Oct 2025, Li et al., 26 Jan 2026).

Ablation studies on EvoTest show prompt mutation, UCB selection, and memory update are all necessary for maximal performance. JitRL ablation highlights the superiority of direct logit modulation over prompt-update schemes, and the crucial role of structured state representations for effective experience retrieval (Li et al., 26 Jan 2026, He et al., 15 Oct 2025).

5. Adaptation Policy Insights and Analysis

Learned adaptation policies in both Meta-TTL and EvoTest exhibit emergent structure and task-specific strategies. Key features include:

• Structured prompt templates encoding diagnosis, fact extraction, exploration plans, route scripts, and parser tips; ablations indicate 10–30% performance loss if any module is removed (Lou et al., 1 Apr 2026).
• Explicit credit assignment in memory, logging which actions are responsible for reward changes.
• Disciplined exploration logic, e.g., limiting new untried moves per episode, reverting to fallback plans on repeated failure.
• Tools for memory extraction and selective recall of reward-increasing state-action pairs.
• Evolution of tool routines and hyperparameters in addition to prompt and memory (He et al., 15 Oct 2025).

These findings suggest that effective test-time adaptation in text-based domains is a joint function of strategic scripting, selective credit assignment, and modular system-level evolution, rather than gradient-driven weight adjustment.

6. Strengths, Limitations, and Prospects

J-TTL and its associated frameworks underscore several strengths:

• Data- and compute-efficiency, allowing online adaptation via API calls in seconds, while RL fine-tuning requires minutes on GPUs.
• Robustness ensured by UCB selection, modular mutation, and holistic agent evolution (prompt, memory, tools, hyperparameters).
• Scalability: Both JitRL and EvoTest achieve continual improvement without gradients or fine-tuning, and substantially reduce monetary costs relative to RL (He et al., 15 Oct 2025, Li et al., 26 Jan 2026).

Limitations include:

• Dependence on the reasoning quality of the Evolver LLM in EvoTest; weaker optimizers degrade improvement rates.
• Limited scope of validation to text-adventure games; real-world applications may require integration with additional tools and symbolic planners.
• Single-session learning: most methods do not transfer adaptation policies or memories across games.

Potential extensions include multi-parent crossover, global memory pools for cross-game transfer, meta-evolutionary adaptation of strategy parameters, and application to real-world domains with rich narrative feedback (He et al., 15 Oct 2025).

7. Impact and Future Directions

J-TTL precipitates a shift in the language agent research landscape by quantitatively focusing on agent self-improvement via test-time learning rules rather than only pre-training or offline adaptation. Research in this domain demonstrates that prompt- and memory-evolution, meta-learned strategies, and gradient-free optimization can outperform hand-crafted and RL-based adaptation in challenging, sparse-reward, partially observable settings (He et al., 15 Oct 2025, Li et al., 26 Jan 2026, Lou et al., 1 Apr 2026).

Future directions include:

• Meta-optimization of richer adaptation policies able to generalize across tasks.
• Integration with symbolic, planning, and tool-use components for broader environments.
• Automated discovery of modular genotype spaces beyond prompt and memory.
• Extending the J-TTL paradigm to multi-step tasks in robotics or web environments, leveraging narrative logs as adaptation substrate.

Collectively, J-TTL establishes a rigorous testbed for language agent continual learning and provides a catalyst for the exploration of modular, evolutionary, and meta-learned adaptation strategies that ensure improvement in previously static, "frozen" agent architectures.