Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

Abstract: Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode-level signal. However, the richer procedural information in the rollout is rarely retained or reused. Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross-episode signals that episode-local updates cannot capture: which strategies consistently pass verification, which failure modes persist, which patterns recur. We propose Procedural Memory Distillation (PMD), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training. This memory functions as a training scaffold, absorbed into the policy itself, yielding a memory-free model at inference. PMD organizes the memory at three levels of abstraction: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns that recur across problems, all extracted online from the model's own trajectories. A memory-conditioned self-teacher draws on the accumulated experience to supervise the student on its own rollouts, enabling student to progressively internalize procedural knowledge within its parameters. The central design principle is co-evolution: the policy generates rollouts that update the memory, and memory shapes the supervision that updates the policy. Empirically, across Qwen3-8B and OLMo3-Instruct-7B, PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH. Co-evolution powers these gains: freezing either the memory or the policy trails PMD by more than 10% across SCIKNOWEVAL domains.

• The paper introduces a novel co-evolving procedural memory mechanism that accumulates and distills multi-level insights from language model experiences.
• It employs online reflection to synchronize policy and memory, significantly outperforming prior RLVR-based self-distillation methods on diverse benchmarks.
• Empirical results show performance gains up to 13.6% and improved answer diversity, demonstrating robust transferability across model scales.

Procedural Memory Distillation: Online Reflection for Self-Improving LLMs

Introduction and Context

Procedural Memory Distillation (PMD) advances the paradigm of reinforcement learning with verifiable rewards (RLVR) and self-distillation in policy optimization for LLMs. Existing on-policy distillation strategies such as Self-Distillation Policy Optimization (SDPO) provide denser learning signals by leveraging feedback at the episode level. However, they operate predominantly on a per-episode basis, discarding strategic knowledge and failure patterns discovered throughout the training regime. This results in a failure to exploit cross-episode signals, leading to suboptimal accumulation and generalization of procedural knowledge.

PMD addresses this gap by introducing a co-evolving procedural memory mechanism, systematically accumulating and abstracting agent experience into a multi-level memory bank. This procedural memory conditions a memory-augmented, on-policy self-teacher, while the distilled student policy eventually internalizes procedural knowledge, obviating memory dependencies at inference.

Figure 1: Overview of PMD's memory construction, distillation, and co-evolution pipeline, illustrating the abstraction hierarchy: experience, insight, and behavior.

PMD Framework and Methodology

Memory Hierarchy and Construction

PMD organizes procedural memory into three explicit levels to balance fidelity and transferability:

1. Experience Memory (Level-0): Stores raw trajectories and feedback at the problem level—retaining detailed, local evidence of agent-environment interaction.
2. Insight Memory (Level-1): Synthesizes problem-specific strategies and failure lessons through reflection over accumulated experiences, using contrastive extraction to delineate success from failure.
3. Behavior Memory (Level-2): Distills cross-problem, reusable patterns and abstract skills, learned by clustering semantically similar problems and aggregating recurrent insights.

The procedural memory is updated online, remaining aligned with the evolving policy.

Co-evolution: Memory-Policy Coupling

A fundamental characteristic of PMD is the co-evolution between policy and procedural memory. At each training stage, policy rollouts update procedural memory, while the refreshed memory conditions the self-teacher signals for subsequent policy updates. This ensures both the teacher and memory are continually synchronized with the agent's capacities and failure modes, outperforming static or offline approaches.

Distillation and Teacher Conditioning

At every training step, the memory-conditioned teacher receives:

• Problem-specific experience and insights,
• Retrieved global behaviors,
• Feedback from previous rollouts.

The policy is trained with a reverse-KL self-distillation objective to match the teacher's distribution on model-generated (i.e., on-policy) states. Notably, memory is used exclusively during training; at inference, the distilled student operates independently of the memory bank.

Empirical Results

Main Performance Improvements

PMD achieves consistent and significant improvement over GRPO and SDPO baselines on both SciKnowEval (multi-domain science reasoning) and LiveCodeBench (unit-test-based code synthesis).

• SciKnowEval: PMD improves Qwen3-8B from 74.4 to 77.2 (+3.8%) and OLMo3-Instruct-7B from 69.5 to 73.3 (+5.5%) over SDPO.
• LiveCodeBench: PMD advances Qwen3-8B from 47.9 to 51.7 (+7.9%) and OLMo3-Instruct-7B from 45.0 to 51.1 (+13.6%).

These effects are isolated to the use of co-evolving procedural memory, as ablation studies demonstrate that freezing the policy or memory diminishes gains by over 10%.

Disentangling Reflection, Persistence, and Co-evolution

Variants show that simple batch-level reflection (transient memory) is beneficial, but the full gains of PMD hinge on accumulating procedural abstractions across steps and, crucially, on the joint evolution of memory and policy. Persistently evolving memory in tandem with a frozen policy yields only marginal improvements, underscoring the necessity of co-adaptation.

Transferability Across Model Scales

PMD's learned procedural memories (insight + behavior) are shown to transfer across models of varying scale. Memory-augmented inference with co-evolved memory consistently outperforms baseline and frozen-policy memory, and deeper retrieval provides monotonic benefits regardless of model size.

Test-Time Scaling and Diversity Preservation

A salient disadvantage of SDPO is mode collapse: converging to a narrow subset of solutions, limiting the efficacy of techniques such as majority voting or best-of-N at test time. PMD, by contrast, maintains broader answer-space coverage, as evidenced by substantially wider verifier-headroom bands.

Figure 2: PMD maintains superior candidate diversity on SciKnowEval, shown by a broader headroom between majority-vote and oracle best-of-k compared to SDPO.

Figure 3: PMD covers a strictly larger portion of solvable problems across all domains compared to SDPO, as visualized by per-subject Venn diagrams.

This diversity enables greater leverage from inference-time sampling and reranking, which is critical for tasks where verifier feedback or post-hoc selection are feasible.

Mechanism Analyses

Memory Internalization

Procedural guidance is absorbed into the student policy, as evidenced by an increasing prevalence of terms such as "strategy", "lesson", and "behavior" in model-generated outputs during training, despite no access to memory at inference.

Figure 4: PMD-trained policies autonomously invoke procedural concepts, indicating internalization of teacher-side abstractions.

Memory Dynamics

Compared across domains, experience and insight memories saturate at the problem level, while the behavior bank undergoes consolidation, maintaining only reusable, domain-spanning skills as training progresses.

Figure 5: Memory bank sizes evolve differently; local experience saturates, while global behavior memory fluctuates and consolidates based on subject-matter complexity.

Practical and Theoretical Implications

Practical: PMD enables language agents to self-improve not just by immediate corrective feedback but by systematizing and reusing acquired procedural knowledge. This produces models that are robust to new problems within a domain without incurring inference-time memory or retrieval overhead.

Theoretical: PMD demonstrates that cross-episode abstractions, conditioned in an on-policy self-distillation loop, can solve the credit assignment and mode collapse challenges typical in RLVR and self-distillation settings. The hierarchical memory design allows fine control over the tradeoff between solution fidelity and transferability.

Compute Trade-offs: PMD increases training wall-clock time by roughly 1.45× compared to SDPO due to longer teacher prompts but does not increase inference latency or memory footprint.

Future Directions

• Extending PMD to heterogeneous, long-horizon agentic environments with broader skill/behavior transfer demands.
• Memory sharing and co-distillation across multiple tasks or domains.
• Investigating meta-learning and continual adaptation regimes grounded in procedural memory evolution.

Conclusion

PMD establishes that memory-augmented, co-evolutionary self-distillation—grounded in the online construction, abstraction, and utilization of procedural memories—yields measurable and transferable improvements over established RLVR and self-distillation approaches. The approach fundamentally reshapes how repeated agent experience is accumulated and leveraged, internalizing abstracted strategic knowledge into policy parameters for robust, memory-free inference. These findings open up research avenues for richer lifelong and cross-domain learning in large-scale LLMs.