Decocted Experience: Distillation for LLM Agents

• Decocted experience is the process of distilling noisy interaction data into compact, high-signal memory for LLM agents.
• It employs lesson-distillation and memory consolidation techniques to extract, compress, and organize actionable strategies from raw agent trajectories.
• This approach improves inference efficiency and robustness across domains, reducing redundancy while maximizing information gain per token.

Decocted experience refers to the process and formalism of transforming raw, often noisy and redundant, interaction data or agent trajectories into a compact, high-signal, reusable memory that can be effectively leveraged by LLM agents for test-time inference, continual improvement, and deployment. In contrast to simple log storage or retrieval-augmented prompting, decoction emphasizes distillation, elimination, and structured organization—yielding a repository of lessons, strategies, or tips that maximize information utility per token and drive more efficient, robust, and adaptable agent behavior across a variety of domains.

1. Formal Definition and Theoretical Foundations

Decocted experience is defined as a lesson-distilled and consolidated subset of raw experience data, specifically constructed to maximize downstream utility for reasoning or task performance while minimizing redundancy and noise. Formally, let each raw experience be $e_i = (x_i, \{(y_i^j, r_i^j)\}_{j=1}^m)$, where $x_i$ is a task statement, $y_i^j$ are sampled trajectories, and $r_i^j$ the corresponding rewards. Decoction applies two core operators:

1. Lesson-distillation ($D_{ld}$): Each $e_i$ is mapped to a distilled lesson $\ell_i$, often via LLM-guided abstraction:

$M^\ell = D_{ld}(E) = \{(x_i, \ell_i)\}_{i=1}^N$

where $\ell_i$ is a reusable, context-independent principle or strategy inferred from successful and failed past attempts.

2. Memory-consolidation ($D_{mc}$): Lessons are further compressed, typically via clustering embedding representations, forming a memory bank $x_i$0 of reduced size but high strategic diversity:

$x_i$1

At inference, a new query $x_i$2 retrieves a small subset $x_i$3, constructing a context for the (frozen) LLM to solve the present task more effectively (Shen et al., 6 Apr 2026).

A guiding theoretical result relates the information gain from decocted context to the efficiency of task completion:

$x_i$4

where $x_i$5 is the completion time in tokens, $x_i$6 is entropy, and $x_i$7 is the information gain from context $x_i$8 (Shen et al., 6 Apr 2026).

2. Methodologies and Algorithms for Decoction

Multiple algorithmic frameworks implement decoction, varying by domain and use-case:

Iterative Experience Refinement (IER)

IER maintains a continually refined experience repository. Each task batch updates the store by:

• Executing with current experience,
• Extracting shortcuts from trajectories,
• Merging new with inherited experiences via
  • Successive pattern: Each generation learns only from its immediate parent.
  • Cumulative pattern: Each generation aggregates all past distilled experience.

Heuristic elimination prunes the store using both:

• Information-gain pruning: Retain only shortcuts with $x_i$9 (where $y_i^j$0 is an information weight based on embedding similarity and compilation success),
• Frequency-based filtering: Keep only frequently retrieved experiences above a mass threshold $y_i^j$1.

The resulting repository is both succinct and potent—quality improved with only 11.54% of the original size (Qian et al., 2024).

Execute-Distill-Verify (EDV) Paradigm

EDV decouples experience construction into multi-agent exploration (Execute), comparative cross-trajectory lesson extraction (Distill), and strict, multi-agent consensus validation (Verify). Shared and private memory banks prevent “self-confirmation traps” endemic in single-agent distillation (Zhu et al., 23 Jun 2026).

ExperienceWeaver for Hierarchical Distillation

Distinctive for clinical text improvement, ExperienceWeaver organizes decocted experience along two axes:

• Error-specific tips: Micro-instructions abstracted from sparse, feedback-laden revision histories.
• High-level strategies: Phase-targeted, distilled guidance.

A two-stage LLM pipeline abstracts, merges (with group size control $y_i^j$2), and then derives hierarchical units for targeted retrieval (Xiao et al., 31 Jan 2026).

Echo: User-Driven Experience Mining

Echo mines large-scale, user-driven refinement streams, also termed (C₀ → C_N) sequences, detected via anchor tracking and gap extraction. Downstream processing applies LLM-based rewriting, verification, and distribution alignment to concentrate experience into high-quality supervision pairs for continual supervised fine-tuning. This drove an acceptance rate increase from 25.7% to 35.7% in commercial code completion (Dong et al., 21 May 2026).

3. Data Structures and Retrieval Strategies

Effective serving of decocted experience depends on scalable, retrieval-optimized data structures:

• Flat vector-indexed memories (e.g., FAISS), for fast nearest neighbor search over (task, lesson) pairs (Shen et al., 6 Apr 2026).
• Cluster-consolidated memories, to cap redundancy while preserving solution diversity (k-means in joint embedding space) (Shen et al., 6 Apr 2026).
• Hierarchical concept trees, where internal nodes index centroids and allow sublinear retrieval (Shen et al., 6 Apr 2026).
• External experience serving repositories, case-matched by cosine similarity between encoding of request and stored experience (Sun et al., 10 Jun 2026).

Selective, quality-aware retrieval (e.g., Top-K matching, trigger-aware queries, LLM-based selection) consistently outperforms both unconditional global injection and random selection, especially for large, heterogeneous repositories (Sun et al., 10 Jun 2026).

4. Empirical Impact and Quality-Cost Trade-Offs

Decocted experience markedly improves agentic reasoning, text revision, moderation, tool use, and code completion:

• Software agents: IER yields 11% relative quality gain over static experience baselines, reducing review/test rounds and memory bloat (from 8,053 to 930 entries) (Qian et al., 2024).
• Clinical text: Hierarchical distillation in ExperienceWeaver improves pairwise LLM-judge scores by 19.85% (Chest X-ray), 11.4% (Abdominal CT), 10.3% (Discharge), validating superiority over RAG and strong LLMs in low-resource regimes (Xiao et al., 31 Jan 2026).
• Agentic tasks: Decoction enables faster, more efficient solving on math reasoning, web browsing, and software engineering. For example, software debugging mean interactions drop from 22 (raw) to 15 (decocted) (Shen et al., 6 Apr 2026).
• Production LLM systems: Retrieval-based external experience boosts moderation accuracy to 71.5% (from 19.6% baseline) with controllable prompt cost (Sun et al., 10 Jun 2026). Optimal Top-K for retrieval is 5–10; larger K yields diminishing returns or prompt dilution.

In all studies, robust ablation demonstrates that aggressive, multistage distillation and context curation contribute additively to downstream gains.

5. Context, Redundancy, and Quality-Control

Empirical and theoretical analysis identifies crucial trade-offs:

• Information gain per token is maximized by compacting diverse, relevant strategies—avoiding both redundancy and overcompression (Shen et al., 6 Apr 2026).
• Relevance–diversity trade-off: Optimal context construction balances semantic similarity with coverage—maximizing

$y_i^j$3

with $y_i^j$4 empirically yielding best performance improvements (Shen et al., 6 Apr 2026).

Independent consensus-based filtering (as in EDV) and user-verified refinement (as in Echo) are vital for filtering erroneous, low-signal data and protecting against self-reinforcing errors (Zhu et al., 23 Jun 2026, Dong et al., 21 May 2026).

6. Deployment and Generalization

Decocted experience is applicable across a range of real-world and research contexts:

• Dynamic, perpetual refinement: Memory is never static—pipelines are designed to continually update lessons in light of new data and evolving domains (Qian et al., 2024, Dong et al., 21 May 2026).
• Domain generality: The decoction paradigm is environment-agnostic, supporting extension to web agents, moderation, structured tool use, and open-ended creative domains (Dong et al., 21 May 2026, Sun et al., 10 Jun 2026).
• Cost-aware serving: Real-world deployment must weigh incremental quality gains against serving costs (prompt length, token budget, latency); hence, selective retrieval often yields an optimal operating point (Sun et al., 10 Jun 2026).
• Model-agnosticism: Decoction typically operates independent of base model weights, requiring only inference-time processing, strategic distillation, or prompt design (Xiao et al., 31 Jan 2026, Shen et al., 6 Apr 2026).

A plausible implication is that decocted experience will become standard in large-scale, continually evolving agentic systems, not only for performance but for interpretability, debugging, and human-in-the-loop co-adaptation.

7. Future Directions and Open Challenges

Despite its progress, the decoction approach leaves several open topics:

• Long-horizon credit assignment: Distillation over extended multi-step or delayed-reward tasks, as in creative domains or multi-week agentic missions, remains underexplored (Dong et al., 21 May 2026).
• Memory contamination resistance: Even with consensus, ensuring adversarial or spurious experience does not degrade memory banks requires further investigation (Zhu et al., 23 Jun 2026).
• Retrieval scalability and compositionality: Efficient compositional retrieval and dynamic memory expansion with bounded latency are active research areas (Shen et al., 6 Apr 2026).
• Generalization to subjective domains: In environments lacking clear verification signals, adoption/satisfaction-based selection and subjective experience decoction present methodological and evaluation challenges (Dong et al., 21 May 2026).

In summary, decocted experience formalizes the transformation of noisy, high-volume agent-environment or user-agent interactions into a distilled, actionable memory that efficiently accelerates, stabilizes, and adapts LLM-based agents for test-time inference and continual learning, across both research and production settings (Qian et al., 2024, Xiao et al., 31 Jan 2026, Shen et al., 6 Apr 2026, Dong et al., 21 May 2026, Sun et al., 10 Jun 2026, Zhu et al., 23 Jun 2026).