Self-Evolving Agents through Privileged Retrospective Distillation

Abstract: Self-evolving agents aim to improve from their own experience rather than remain fixed after deployment, but naturally collected trajectories often lack ground-truth solutions or verifier-derived rewards. We introduce Retrospective Self-Distillation (RSD), a multi-round framework that turns such experience into dense supervision using privileged retrospective context. For each on-policy trajectory, the same underlying model acts as both student and teacher: the teacher re-scores the student's realized actions while conditioning on cross-task precedents and lessons, together with turn-level reflection and future hindsight from the completed trajectory. The resulting teacher distribution is distilled back into the policy through on-policy self-distillation. Across rounds, the experience bank grows and the retrospective context is refreshed, allowing the model, its data, and its supervision to co-evolve. We further introduce frontier-guided context-policy evolution, where a stronger frontier model proposes constrained edits to how privileged context is constructed. Across ALFWorld, ScienceWorld, and SearchQA with Qwen3-8B and Qwen3-14B, RSD improves performance by up to 59.7% relative to the base model, while frontier-guided context-policy evolution adds up to a further 44.7% relative improvement over a fixed context policy. These results show that naturally accumulated experience can be converted into persistent model capability without ground-truth trajectories or verifier-derived rewards for the core policy update.

• The paper introduces **Retrospective Self-Distillation (RSD)**, a training framework that converts interaction trajectories into dense token-level supervision, enabling LLM agents to improve their policies without explicit target actions or reliable scalar rewards.
• RSD allows the same model to reassess its past actions with later observations and distills this retrospective information into the policy parameters, showing substantial performance improvements on benchmarks like ALFWorld (13.7-20% SR gains) and competitive results against supervised and reward-driven methods.
• The paper introduces frontier-guided context policy evolution, which adaptively improves context policy design using an outer-loop mechanism, offering significant performance enhancements, especially on tasks with considerable context design potential like ScienceWorld.

The paper addresses a central difficulty in self-evolving LLM agents: naturally collected interaction trajectories rarely contain explicit target actions, ground-truth demonstrations, or reliable scalar rewards. It introduces Retrospective Self-Distillation (RSD), a multi-round training framework that converts such trajectories into dense token-level supervision by allowing the same model to reassess its past actions with information available only after the interaction. The resulting supervision is distilled into the policy parameters, rather than retained solely in an external memory or inference-time harness (2610.27495).

Problem formulation and central premise

The setting is deliberately restrictive. At each self-evolution round, the current policy generates one on-policy trajectory per sampled task. The experience bank accumulates both successful and failed trajectories, but the core policy update receives neither ground-truth trajectories nor verifier-derived rewards. This excludes conventional SFT on oracle solutions and reward-driven RL methods that depend on executable success signals during optimization.

The paper’s central premise is that an incomplete trajectory is more informative retrospectively than it was online. Once the interaction terminates, later observations reveal the consequences of earlier actions. In addition, previously collected trajectories provide concrete precedents, reusable strategies, and failure patterns. RSD exploits this asymmetry: the acting student sees only its original interaction history, while the teacher receives a privileged retrospective context derived from the completed trajectory and the accumulated experience bank.

This formulation distinguishes RSD from methods that improve only prompts, workflows, memory, or tools. Those methods can increase agent performance while leaving the underlying model unchanged. RSD instead aims to internalize retrospective supervision into model parameters, so that the final policy is evaluated without the privileged context used during training.

Retrospective self-distillation

At round $r$, the current policy $\pi^{(r)}$ generates trajectories $\mathcal{D}^{(r)}$, which are added to the cumulative experience bank $\mathcal{B}^{(r)}$. For each turn in each newly collected trajectory, RSD constructs a privileged context from four information sources:

1. Precedents: related past trajectories retrieved using task-specific filtering and content-token overlap with the current task.
2. Lessons: task-type summaries generated from selected successful and failed experiences.
3. Reflection: a retrospective diagnosis of the current trajectory, including the earliest pivotal mistake and a better alternative action.
4. Future hindsight: downstream interaction and environment feedback following the current decision.

The same underlying model then produces a privileged teacher distribution conditioned on this context. The student is trained with forward KL distillation on the realized action tokens of its own trajectory. Thus, the teacher does not generate a replacement trajectory or select only successful episodes. It re-scores states actually visited by the current policy, including states from failed attempts.

The method therefore combines on-policy data collection with privileged-context distillation. Its essential learning signal is not a binary trajectory label but a dense distribution over alternative next tokens. This is important because a failed trajectory can still expose a locally incorrect action, a misleading observation interpretation, or an ineffective subgoal ordering.

Figure 1: RSD alternates on-policy rollout collection, experience-bank expansion, retrospective context construction, teacher re-scoring, and parameter distillation.

The experience bank and the privileged context are refreshed at every round. Consequently, the supervision is not a fixed annotation layer applied repeatedly to stale trajectories. The policy generates new behavior, that behavior alters the available precedents and lessons, and the updated retrospective signal is distilled into the next policy.

Frontier-guided context-policy evolution

The fixed RSD procedure keeps the context-construction rule unchanged while refreshing its contents. The paper also introduces a stronger-access extension in which a frontier model evolves the context policy itself. The frontier model may modify a constrained YAML configuration controlling precedent count and retrieval, component inclusion, context placement, and related teacher-side choices. It cannot modify the model weights or the distillation algorithm.

Candidate context policies are first ranked using a no-training privileged-context-at-inference probe. The probe supplies the candidate context directly to the current model on a small training-distribution slice and measures task performance without parameter updates. The selected candidate is then evaluated after a full self-evolution round using the actual environment metric. If post-training performance does not match the best accepted configuration within a tolerance of $0.5$ percentage points, the context policy is reverted while the model parameters continue evolving.

This separation is methodologically significant. The frontier model proposes edits, but empirical probe and post-training acceptance determine whether those edits survive. The paper’s ablation shows that frontier-model access alone is insufficient: on ScienceWorld-8B, frontier guidance without the no-training probe reaches 23.9% SR, close to the fixed baseline at 23.4% and random search at 23.2%, whereas adding the probe raises performance to 27.5%.

Experimental design

RSD is evaluated with Qwen3-8B and Qwen3-14B on ALFWorld, ScienceWorld, and SearchQA. The benchmarks differ substantially: ALFWorld tests embodied household interaction, ScienceWorld tests scientific manipulation and multi-step task completion, and SearchQA tests search-mediated question answering.

The training procedure uses ten rounds, 200 unique training instances per round, one rollout per instance, full-parameter forward-KL distillation, and a learning rate of $3 \times 10^{-6}$. No policy-gradient term or task-reward term is included in the RSD update. At inference, the principal RSD results use the evolved student without privileged context.

Benchmark and model	Base	RSD	Absolute gain
ALFWorld, Qwen3-8B	62.9	76.6	+13.7
ALFWorld, Qwen3-14B	69.3	79.6	+10.3
ScienceWorld, Qwen3-8B	15.6	23.4	+7.8
ScienceWorld, Qwen3-14B	12.9	20.6	+7.7
SearchQA, Qwen3-8B	35.4	37.9	+2.5
SearchQA, Qwen3-14B	35.4	38.7	+3.3

The largest gains occur on ScienceWorld and ALFWorld. The paper reports a maximum relative improvement of 59.7% over the corresponding base model. RSD outperforms the evaluated self-training baselines in all six benchmark–model settings. On ScienceWorld-8B, it reaches 23.4% SR, compared with 20.8% for BoN-16+SFT and 12.7% for RFT and STaR. The result is notable because BoN-16+SFT uses sixteen candidate rollouts per instance and ground-truth success selection, whereas RSD uses a single naturally collected rollout and no reward-based policy objective.

RSD is not uniformly superior to methods with stronger supervision. On ALFWorld-14B, GRPO reaches 82.6% and ground-truth OPSD reaches 82.5%, compared with 79.6% for RSD. These methods, however, use environment rewards or gold successful trajectories. The relevant claim is therefore not that RSD dominates oracle or reward-driven optimization, but that it achieves competitive performance under a substantially weaker supervision regime.

Task-dependent self-evolution

The round-wise results reveal that self-evolution is strongly task dependent. ALFWorld and ScienceWorld exhibit progressive improvement across rounds, although the curves are not strictly monotonic. SearchQA remains within a narrow performance band. This difference is consistent with the paper’s teacher diagnostics: ScienceWorld has large and increasing teacher corrections, while SearchQA has moderate, comparatively stable corrections and little evidence of sustained policy adaptation.

The result contradicts a simple assumption that any privileged retrospective context will automatically produce persistent improvement. SearchQA has useful contextual information—the inference-time harness reaches 41.4% EM with Qwen3-14B—but the information is not effectively internalized by RSD, whose no-context score is 38.7%. The limitation is therefore not merely a lack of contextual utility; it is a failure to convert that utility into stable parameter-level behavior.

The comparison with test-time reflection further clarifies the trade-off. RSD exceeds Reflexion@2 in all six settings under single-attempt evaluation, but Reflexion@3 reaches 80.7% on ALFWorld-8B, 43.6% on SearchQA-14B, and 24.1% on ScienceWorld-14B. RSD exchanges repeated inference-time attempts for persistent parameter adaptation, but the paper does not claim that it dominates increased test-time computation.

Ablations and mechanism analysis

The component ablations show that the useful form of privileged information varies by environment. On ALFWorld-8B, removing lessons, reflection, or future hindsight reduces SR from 76.6% to approximately 71–72%, whereas removing precedents causes only a small decrease to 75.0%. On ScienceWorld-8B, the pattern reverses: removing precedents reduces SR to 15.1%, nearly the 15.6% base score, while removing future hindsight reduces performance to 17.6%.

Variant	ALFWorld SR	ScienceWorld SR
Full context	76.6	23.4
Without precedents	75.0	15.1
Without lessons	71.6	22.7
Without reflection	71.4	21.8
Without future hindsight	71.4	17.6

These results imply that RSD is not explained by a single generic “privileged context” effect. ALFWorld benefits primarily from trajectory-specific retrospective information, whereas ScienceWorld depends more heavily on transferable precedents. The lesson-source ablation also supports combining successful and failed trajectories: lessons built from both sources reach 76.6% on ALFWorld and 23.4% on ScienceWorld, exceeding lessons derived from either source alone.

Context freshness is similarly important. On ALFWorld-8B after three rounds, freezing the round-one context produces 59.7% unseen SR, compared with 67.9% for the fully evolving context. Updating lessons alone reaches 69.0%, while updating precedents alone reaches 63.8%. The implication is that repeated distillation against stale retrospective notes constrains continued improvement; newly collected trajectories must refresh at least part of the teacher context.

Teacher dynamics expose a more severe stability issue. A fully evolving self-teacher reaches 80.3% on ALFWorld by round seven but then collapses to 39.4% and finally 0% in rounds nine and ten. A permanently frozen teacher remains stable but peaks at only 68.2%. The default freeze-to-peak schedule combines early teacher adaptation with reversion to the best teacher checkpoint once validation performance declines. This stabilization is necessary for the reported training behavior, and it indicates that unconstrained co-evolution of student and teacher is not reliably stable.

Internalization of privileged supervision

The paper evaluates whether retrospective supervision is absorbed by the model or remains dependent on the inference-time harness. The key comparison is between the evolved student without context and the base model with context.

On ALFWorld, internalization is substantial: the evolved no-context model exceeds the base context-equipped model by 14.5 percentage points for Qwen3-8B and 8.9 points for Qwen3-14B. Reintroducing the harness to the evolved model yields only a small additional gain. This supports the claim that RSD can transfer useful retrospective information into persistent model capability.

ScienceWorld shows partial and scale-dependent internalization. The 8B evolved no-context model exceeds the base context-equipped model by 3.8 points, but the harness adds a further 3.3 points. For 14B, the evolved no-context model is approximately equal to the base context-equipped model, while adding the harness yields another 6.1 points. SearchQA shows no measurable internalization: the evolved model without context does not exceed the base model with context.

Figure 2: Internalization varies by task and scale, with strong absorption on ALFWorld but substantial residual harness utility on ScienceWorld and SearchQA.

The re-scoring analysis provides evidence for directional policy correction. For actions disfavored by the privileged teacher, the next checkpoint assigns lower log probability in several round transitions, with $\Delta \log p=-0.384$ for ALFWorld rounds 1–2 and $-0.696$ for ScienceWorld rounds 1–2. However, the exact-context analysis does not establish a proportional relationship between teacher correction magnitude and student uptake. The defensible conclusion is that RSD can transfer correction direction, not that it reproduces the full teacher distribution.

Frontier-guided results

Frontier-guided context evolution produces its largest improvements on ScienceWorld, where the fixed-policy scores of 23.4% and 20.6% for 8B and 14B rise to 27.5% and 29.8%, respectively. On ALFWorld, the improvements are smaller: 76.6% to 78.4% for 8B and 79.6% to 80.3% for 14B. The reported maximum additional relative improvement over the fixed context policy is 44.7%.

Figure 3: Frontier-guided context-policy evolution improves ScienceWorld substantially and yields smaller gains on the more saturated ALFWorld task.

The contrast between tasks is consistent with the context ablations. ScienceWorld has greater headroom in context construction: removing precedents causes a large performance loss, and the frontier repeatedly increases the precedent budget. In four independent ScienceWorld-8B runs, all peak configurations increase the number of retrieved precedents. The frontier also turns lessons or future hindsight off in some later configurations, despite both components being beneficial in the canonical ablation. This is not inconsistent: component ablations measure marginal utility under the original context policy, whereas frontier search evaluates stage-dependent configurations after some information may already have been internalized.

The frontier extension nevertheless requires assumptions absent from core RSD: access to a stronger frontier model, an executable task metric, and additional probe and evaluation computation. Its principal contribution is therefore an adaptive outer-loop mechanism for context design, not a reward-free procedure in the broader sense. The model-parameter update remains reward-free, but the context-policy search is explicitly metric-driven.

Limitations and open questions

The empirical scope is limited to two Qwen3 model sizes, three benchmarks, ten rounds, and relatively small on-policy datasets of 200 instances per round. Several diagnostics, including the lesson-source comparison and teacher-schedule ablation, use a single seed and should therefore be interpreted as mechanistic evidence rather than statistically conclusive estimates.

RSD also depends on generated reflections and lessons. These are produced by the same evolving model that supplies the policy and teacher, so retrospective analyses can contain systematic errors or self-consistent but incorrect explanations. The paper addresses instability through freeze-to-peak teacher selection, but this introduces a benchmark-selection dependency and does not establish that the method remains stable under longer evolution horizons.

The results leave three specific questions open. First, what properties of a task determine whether privileged information is internalized, as on ALFWorld, rather than retained as complementary context, as on ScienceWorld and SearchQA? Second, can teacher schedules be stabilized without selecting checkpoints using external benchmark performance? Third, how do experience-bank size, retrieval quality, and task distribution affect the transition from useful accumulation to stale or misleading retrospective supervision?

Conclusion

RSD presents a parameter-level self-evolution procedure for agents whose interaction data lack ground-truth trajectories and reliable optimization rewards. Its key mechanism is to retrospectively enrich on-policy states with cross-task precedents, lessons, reflection, and future hindsight, then distill the resulting privileged teacher distribution into the acting policy. The strongest results are on ALFWorld and ScienceWorld, where RSD reaches 76.6% and 23.4% with Qwen3-8B and demonstrates substantial internalization of privileged supervision. SearchQA provides an important boundary case in which contextual utility does not translate into persistent improvement. Frontier-guided context evolution further shows that the supervision constructor itself can be adapted, particularly when task-specific context headroom is substantial.