ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience

Abstract: A LLM agent solves long-horizon tasks through many reasoning-action turns, with one verification signal at termination. Deployed agents face streams of related tasks, making their trajectories a natural resource for improvement. In-context adaptation agents store reflections, memories, or skills as text, so reuse depends on retrieving the right experience and on a frozen policy executing it. We study Online Agentic Test-Time Training (OaTTT), which trains the LLM's weights on its own execution trajectories during deployment. The agent executes each task once, in one pass over the stream, and the executed trajectory with its verification result is the only learning signal for weight updates that persist across tasks. Directly imitating or reinforcing the generated tokens of this single attempt destabilizes the policy. We introduce ASCENT (Agentic Self-distillation for Cross-task EvolutioN at Test-time), which instead self-distills verified experience. A stable version of the LLM, its frozen initial copy, receives the verified trajectory as privileged information and predicts next-token distributions along it with this hindsight. Distilling them into persistent LoRA fast weights updates the agent for later tasks, without an external reference solution or stronger teacher. By further removing invalid-action turns, ASCENT distills enhanced privileged experience for more efficient execution. We characterize its population target and the limits of sparse outcome selection. Across ALFWorld, WebShop, and AppWorld at varied model scales, ASCENT improves task success and interaction efficiency as experience accumulates, outperforms online adaptation methods, and transfers to held-out scenes, showing that an agent can consolidate verified experience into its weights without a separate training phase or memory retrieval. Project page: https://artificer-ai-lab.github.io/ASCENT

• The paper introduces ASCENT, a method that uses verified experience to train long-horizon agents in an online, test-time setting, significantly improving performance over benchmark models.
• ASCENT utilizes a self-distillation approach from a frozen model, enabling agents to learn stable, generalizable behavior across different tasks with only a single attempt per task and no reprocessing.
• The study demonstrates ASCENT's efficacy across various benchmarks, including ALFWorld, WebShop, and AppWorld, where it outperforms base models and other test-time training methods in both task success and interaction efficiency.

Problem setting and contribution

“ASCENT: Online Test-Time Training of Long-Horizon Agents via Self-Distillation of Verified Experience” (2610.05303) studies Online Agentic Test-Time Training (OaTTT): a deployment protocol in which an agent encounters a one-pass stream of tasks, receives only the environment’s terminal verification signal after each single attempt, and updates parameters that persist across subsequent tasks. This setting is materially stricter than conventional test-time adaptation and online reinforcement learning. There is no task-specific training split, no retry, no multiple sampled trajectory per task, no replay buffer, and no external reference solution. Each trajectory is used once, immediately after it is scored, and the reported outcome is computed before that trajectory can affect the agent.

The central empirical claim is that directly learning from the agent’s own generated trajectories is unstable in this regime. Naive online imitation and single-attempt policy-gradient methods increasingly amplify accidental reasoning, formatting errors, invalid actions, and behavioral drift. The paper therefore proposes ASCENT—Agentic Self-distillation for Cross-task EvolutioN at Test-time—which uses verifier-accepted trajectories as privileged information for a frozen copy of the initial LLM. The frozen model produces full next-token distributions conditioned on hindsight from the successful trajectory; persistent LoRA parameters are then trained to match those distributions at the original agent’s student prefixes.

The method addresses a specific distinction between harness-level and parametric adaptation. In-context approaches store reflections, memories, skills, or procedural traces and must retrieve and execute them reliably during later interactions. ASCENT instead consolidates selected experience into fast weights, eliminating retrieval at inference. This does not make memory irrelevant—the model still depends on the distribution of previously verified experiences—but changes the failure mode from retrieval and execution of textual artifacts to parameter interference and online optimization stability.

OaTTT protocol and instability of direct learning

Let a task stream consist of tasks $x_1,\ldots,x_N$. For task $x_i$, the current policy makes one complete attempt and generates trajectory $\tau_i$. The environment returns a binary verification result $v_i$, after which the update rule evolves the model for task $i+1$. The crucial prequential ordering is:

1. execute and score task $i$ using the current policy;
2. use $(x_i,\tau_i,v_i)$ for an update;
3. execute task $i+1$ with the updated policy.

The paper evaluates several direct-update baselines using LoRA adapters: ungated imitation of every generated trajectory, verifier-gated rejection fine-tuning, rejection fine-tuning with a KL anchor to the base model, and two single-attempt REINFORCE variants. All operate on the same generated token sequence and differ only in the token weights assigned by the update.

The controlled ALFWorld analysis shows a common degradation pattern. Ungated imitation and verifier-gated imitation become nearly deterministic at the first response token while moving increasingly far from the initial policy. The KL-anchored variant drifts more slowly but still loses executable behavior. The policy-gradient methods become dominated by failed episodes after success becomes rare; one variant collapses toward tokens that the base model assigns very low probability, while the normalized variant exhibits substantial entropy oscillation. Across the baselines, valid-action rates decline, responses become longer, and later-stream success falls below the untrained base model.

The paper’s explanation is technically important. A hard-target update modifies the probability of the generated token but does not provide positive supervision for alternative tokens that may have been preferable. A failed trajectory therefore either receives no learning signal or, under policy-gradient updates, actively suppresses its sampled tokens without identifying a useful replacement. Even successful trajectories contain incidental reasoning, redundant detours, invalid-action responses, and formatting artifacts. With batch size one and no replay, these sample-specific effects are not averaged before being carried into later tasks. The resulting feedback loop couples policy drift to the distribution of future training data.

ASCENT’s self-distillation objective

ASCENT gates adaptation on verifier acceptance. Failed episodes produce no parameter update, while accepted episodes are treated as rejection-sampled positive experience. For each accepted trajectory, the method constructs privileged information $z_i$ from the task, a success statement, and the reasoning-action responses of valid turns. The student, however, is trained on the original prefixes it actually encountered. Thus, at each student prefix, the frozen teacher sees additional hindsight while the student does not.

The teacher distribution is

$$q_{i,t,j}(\cdot) = p_{\theta_0}(\cdot \mid z_i \oplus c_{i,t,j}),$$

where $x_i$0 is the frozen initial model and $x_i$1 is the student prefix at token position $x_i$2 of turn $x_i$3. The student distribution is produced by the current base model plus its persistent LoRA adapter. ASCENT minimizes the forward KL divergence from the teacher to the student:

$x_i$4

This objective differs from direct imitation in three ways. First, it is a full-vocabulary distribution-matching objective rather than a one-hot target. Tokens preferred by the teacher can gain probability even when they were not generated by the student. Second, the teacher is conditioned on the complete verified trajectory, enabling hindsight-informed guidance at earlier decision points. Third, the teacher remains fixed at the initial checkpoint, preventing the target distribution from drifting with the student across tasks.

The paper characterizes the population target of this procedure. Under a fixed collection policy, verifier selection reweights trajectories according to their probability of eventual acceptance. Forward-KL distillation then fits, at each student prefix, the conditional mean of the frozen teacher distributions over accepted trajectories reaching that prefix. This result precisely identifies what ASCENT learns, but the authors correctly stress what it does not establish: it does not prove that the teacher’s hindsight distributions are causally correct, that a finite LoRA adapter reaches the population target, or that the fitted behavior transfers to future tasks.

The action-validity filter further removes invalid-action turns from the teacher’s privileged context while retaining all nonempty responses as student prefixes. The filter therefore changes only what the teacher sees; the student is still trained at positions corresponding to invalid actions. This allows the teacher to assign lower probability to behavior associated with invalid turns without discarding the corresponding student states. The filter does not determine whether a valid action was useful, nor does it provide turn-level reward or causal credit.

Experimental design

The evaluation covers ALFWorld, WebShop, and AppWorld using Qwen3.5-4B and Qwen3.5-9B. ALFWorld provides a 140-task seen stream and a 134-task held-out scene stream; WebShop uses 500-task streams; AppWorld evaluates interactive coding under normal and challenge splits. ALFWorld and WebShop impose a 50-turn budget, while AppWorld uses a 30-interaction budget. All online methods share task order, prompting template, single-attempt execution, and access to the same terminal verification signal.

ASCENT uses LoRA rank 16 and scale 32, with two AdamW steps after each accepted episode at learning rate $x_i$5. The trajectory is discarded after the update, and the adapter and optimizer state persist across tasks. Runtime includes the interaction and the online update, making the reported timing more informative than evaluation-only inference latency.

Main results

ASCENT consistently improves both success and interaction efficiency. On the ALFWorld seen stream, it achieves:

Backbone	Base success	ASCENT success	Base turns	ASCENT turns
Qwen3.5-4B	46.4%	69.5%	35.1	26.3
Qwen3.5-9B	55.0%	77.4%	32.6	21.0

The improvement over the base is 23.1 and 22.4 percentage points, respectively. ASCENT also surpasses the strongest listed online in-context baselines: the best competing seen-stream success is 63.6% for the 4B model and 69.3% for the 9B model. The gains are not confined to easy task families. In particular, the base model performs poorly on ALFWorld’s Heat and Cool families, whereas ASCENT substantially improves both at both model scales.

WebShop shows a similar pattern:

Backbone	Base strict success	ASCENT strict success	Base mean score	ASCENT mean score	ASCENT turns
Qwen3.5-4B	17.8%	41.9%	25.9%	62.4%	17.8
Qwen3.5-9B	17.2%	41.5%	27.8%	56.8%	26.7

ASCENT more than doubles strict success for both backbones and produces the highest mean score among the compared methods. It also reduces interaction length substantially. These results support the paper’s claim that the learned behavior is not merely a success-rate artifact caused by converting capped failures into successes: on ALFWorld tasks solved by both the un-evolved base and ASCENT, ASCENT uses fewer turns on approximately 64% of shared successes for the 4B model and 67% for the 9B model. Mean turn differences on shared successes are $x_i$6 and $x_i$7 turns for the two scales.

The transfer results are particularly relevant to the cross-task-evolution claim. After the seen-stream adapter is frozen and transferred to held-out ALFWorld scenes, ASCENT obtains 62.7% success at 4B and 88.8% at 9B, compared with base performance of 43.3% and 49.3%. When online adaptation resumes on the unseen stream, success rises to 78.6% at 4B and 90.0% at 9B. ASCENT therefore retains seen-stream improvements under scene shift and can continue adapting from the transferred state.

On AppWorld, ASCENT improves task goal completion from 20.2% to 26.2% on Test-Normal and from 15.8% to 20.0% on Test-Challenge using Qwen3.5-4B. It leads the online comparators in both task-level settings, although its scenario-level completion does not lead on Test-Challenge. This qualification matters: task-level improvements do not uniformly translate into completion of all tasks within a scenario.

Why complete-trajectory hindsight matters

The comparison with TT-OPSD isolates the contribution of long-horizon hindsight. TT-OPSD uses the same self-distillation framework but gives the teacher only current-turn information. ASCENT’s teacher instead receives the verified trajectory across the entire episode. On ALFWorld, ASCENT reaches 69.5% versus TT-OPSD’s 51.0% at 4B and 77.4% versus 61.2% at 9B. On WebShop, the corresponding strict-success results are 41.9% versus 25.6% and 41.5% versus 24.2%.

The ablations show that reasoning-action text is more valuable than action-only hindsight. On ALFWorld seen, action-only privileged information produces 48.3% success at 4B and 48.6% at 9B with filtering, close to or below the base. Reasoning-action information raises success to 69.5% and 77.4%. Adding observations can be beneficial for the 9B model—77.1% without filtering versus 77.4% for the default filtered reasoning-action configuration—but is less effective for the 4B model, where the shorter filtered representation performs best.

These results imply that the useful supervision is not simply a record of executable commands. Reasoning tokens align more of the teacher’s distribution with the positions at which the student must generate text, and they encode subgoal structure that action-only traces omit. At the same time, the smaller model is more sensitive to privileged-context length and noise, making filtering a substantive component rather than a cosmetic preprocessing choice.

The divergence ablation shows that forward KL is a strong default but not universally dominant under every configuration. Replacing it with reverse KL or Jensen–Shannon divergence still yields substantial gains. Reverse KL reaches 62.1% at 4B and 60.0% at 9B, while Jensen–Shannon reaches 62.1% and 80.7%. The forward-KL configuration remains the principal result and is particularly strong at 4B, but the 9B divergence comparison indicates that the relationship between distillation geometry and downstream agent performance is model-dependent.

Fixed versus evolving teachers

The paper also examines whether the teacher should track the evolving student. The default frozen teacher achieves 69.5% on the ALFWorld seen stream. Fast teacher evolution is harmful: an exponential moving average with coefficient 0.90 achieves only 24.3%, and snapshots refreshed every 10 accepted updates achieve 45.7%. Slower evolution is less damaging—EMA 0.999 reaches 61.4%, while snapshots every 30 updates reach 55.7—but none exceeds the fixed teacher.

The interpretation is that a moving teacher creates a second feedback path. The student determines which trajectories are accepted, and an evolving teacher determines how those trajectories are converted into targets. If teacher and student remain too close, student drift is propagated into later supervision. A frozen teacher avoids this compounding process while still allowing task-conditioned targets through the privileged trajectory. The resulting population target is therefore not a conventional self-consistency condition in which both policy and target evolve together; it is a fixed-model interpretation of changing verified experience.

This finding supports the paper’s stronger methodological claim: stability arises not merely from using soft targets, but from separating the source of supervision from the persistent fast weights being optimized.

Limitations and open questions

ASCENT requires access to full next-token distributions from an open-weight frozen model, so it does not directly apply to API-only models. It also relies on a verifier capable of identifying accepted episodes. The terminal binary signal gates adaptation but provides no causal credit among turns, and all training positions in an accepted trajectory are weighted uniformly after turn-level normalization.

The action-validity filter distinguishes executable from non-executable actions but cannot identify unnecessary valid actions. The paper explicitly shows that verification does not imply shorter successful paths: a valid detour and an efficient action can both lead to acceptance, and the accepted trajectory distribution cannot rank untried alternatives. Although shared-success comparisons support an interaction-efficiency improvement, they do not identify which individual decisions caused the reduction in turns.

A further limitation is dependence between teacher context and student-generated text. Because the privileged trajectory contains the student’s own reasoning-action responses, the frozen teacher may partially reconstruct or condition on the tokens it is supervising. The experiments establish empirical utility but do not isolate how much of the gain comes from genuine hindsight about later consequences versus re-interpretation of the student’s response text.

Finally, low distillation loss is not equivalent to task success. The forward-KL objective measures agreement with the frozen teacher at observed prefixes and cannot guarantee coverage of alternatives absent from the teacher distribution. The population analysis characterizes the optimization target but does not establish convergence, causal credit assignment, or transfer under substantially different task distributions.

Conclusion

ASCENT presents a parametric approach to online adaptation of long-horizon agents under an unusually restrictive single-pass protocol. Its main contribution is to replace hard imitation or signed reinforcement of one sampled trajectory with verifier-gated, hindsight-conditioned self-distillation from a frozen initial model into persistent LoRA weights. Across ALFWorld, WebShop, and AppWorld, the method improves success, reduces interaction length, transfers across held-out scenes, and avoids the collapse observed with direct online updates.

The results indicate that complete-trajectory privileged information and a fixed teacher are central to the method’s stability. They also delimit the claim: ASCENT consolidates verified experience effectively, but sparse terminal verification still leaves turn-level credit, causal alternative evaluation, and the independence of hindsight supervision unresolved.