Decocted Experience Improves Test-Time Inference in LLM Agents

Abstract: There is growing interest in improving LLMs without updating model parameters. One well-established direction is test-time scaling, where increased inference-time computation (e.g., longer reasoning, sampling, or search) is used to improve performance. However, for complex reasoning and agentic tasks, naively scaling test-time compute can substantially increase cost and still lead to wasted budget on suboptimal exploration. In this paper, we explore \emph{context} as a complementary scaling axis for improving LLM performance, and systematically study how to construct better inputs that guide reasoning through \emph{experience}. We show that effective context construction critically depends on \emph{decocted experience}. We present a detailed analysis of experience-augmented agents, studying how to derive context from experience, how performance scales with accumulated experience, what characterizes good context, and which data structures best support context construction. We identify \emph{decocted experience} as a key mechanism for effective context construction: extracting essence from experience, organizing it coherently, and retrieving salient information to build effective context. We validate our findings across reasoning and agentic tasks, including math reasoning, web browsing, and software engineering.

• The paper introduces experience decoction, a novel method that distills agent trajectories into concise lessons for improved test-time reasoning.
• It employs clustering-based memory consolidation and hierarchical concept trees to balance semantic relevance with diversity in context retrieval.
• The paper demonstrates that significant inference performance gains are achievable without retraining models, solely through refined context engineering.

Decocted Experience and Test-Time Inference Improvements in LLM Agents

Introduction

The paper "Decocted Experience Improves Test-Time Inference in LLM Agents" (2604.04373) presents a systematic study of context engineering through experience augmentation in LLM agents. Rather than solely increasing inference-time computation, the work investigates how constructing effective input contexts from distilled experience can substantially improve agentic reasoning and decision making under fixed test-time budgets. The exposition provides theoretical and empirical analyses of memory construction, experience distillation, and retrieval, with rigorous validation across mathematical reasoning, web-based interaction, and software engineering domains.

Experience-Aware Context Construction

The study posits context as a critical axis for test-time performance scaling, orthogonal to naïve compute scaling. While handcrafted or manually collated contexts offer limited scalability and generalization, this work formalizes context construction as a transformation of self-acquired agent experience into optimal input for frozen LLMs. Experience is defined as sets of trajectories and feedback over historical tasks, organized into an external memory. Direct use of raw, voluminous trajectories is inefficient given token limits and introduces significant redundancy. The central proposal is "experience decoction": extracting the informational essence from accumulated experience through lesson distillation, memory consolidation, and salience-driven retrieval.

Lesson Distillation and Memory Consolidation

The work introduces a minimal instantiation of experience memory—with flat storage and similarity-based retrieval—before transitioning to more advanced mechanisms. Lesson distillation employs the LLM to summarize key strategies, transferable workflows, and failure patterns from raw trajectories into reusable, concise lessons. Experimental results demonstrate that, for agentic tasks characterized by noisy, lengthy, and environment-coupled interactions, distilled lessons significantly surpass raw trajectories in both accuracy and computational efficiency at test time. Notably, for math reasoning, where detailed traces already encapsulate most relevant cues, distilled lessons offer only marginal deficits, whereas for interactive or partially observable environments, distillation yields clear improvements in performance metrics.

Memory consolidation is addressed through clustering-based compression. Embeddings of problem-lesson pairs are clustered, retaining only representatives per cluster to manage redundancy and memory size. Empirically, there is a non-monotonic relationship between memory size and agent performance, with consolidation exhibiting a "sweet spot" of intermediate size that maximizes generalization and efficiency by balancing relevance diversity and representation coverage.

Theoretical Characterization of Context Quality

A formal theoretical connection between context informativeness and inference efficiency is established via information-theoretic measures. The information gain $I(Y; C = c | X = x)$, defined as the reduction in output entropy conditioned on the context, is analytically linked to expected trajectory length at inference. Empirical estimates illustrate a strong linear correlation (Pearson $r = 0.91$) between conditional entropy and output length; contexts inducing higher information gain consistently yield more concise outputs and improved effectiveness.

Crucially, naive retrieval-driven relevance is insufficient for optimal performance. The study empirically validates that context quality is best characterized by a weighted combination of semantic relevance and diversity among retrieved experiences. The Pearson correlation peaks when balancing these factors, supporting the principle that diverse, yet topically aligned, content maximizes practical agent gains.

Structured Memory: Hierarchical Concept Trees

The limitations of flat, unstructured memories for context retrieval motivate the introduction of hierarchical concept trees. These trees are built via multi-level clustering of lesson embeddings, informed by LLM-generated structured concept descriptions. Each leaf of the tree represents a coherent conceptual cluster. Retrieval is performed at the concept group level, followed by re-ranking via the base LLM, enforcing topical diversity and mitigating redundancy.

Quantitative comparisons across tasks such as WebShop and SWE show that hierarchical structures yield more diversified, relevant contexts, leading to measurable improvements in effectiveness over traditional Top-K semantic retrieval baselines. The hierarchical memory enables adaptive control over retrieval granularity and content coverage, essential for agentic tasks involving deep, compositional workflows and error-prone search.

Implications and Future Directions

The results underline several implications for LLM agent design:

• Experience decoction as a compression mechanism: The operational pipeline of distillation, consolidation, and structured retrieval transforms voluminous, heterogeneous experience into succinct, information-rich context that optimally supports frozen-model inference.
• Relevance-diversity trade-off as a retrieval objective: Simple similarity-based recall is suboptimal; structured memory and diversity-aware selection are necessary for generalization.
• Test-time improvement without parameter updates: Substantial performance gains are achievable solely through context engineering and memory augmentation, minimizing the need for costly model retraining.

Theoretically, these findings align with information-theoretic perspectives on sample complexity and sequential decision making. Practically, memory-augmented approaches stand to impact domains requiring continual learning, low-shot generalization, and robust deployment under constrained computational budgets.

Potential future directions include the development of closed-loop, continually updated memory systems, reinforcement learning-based memory construction, dynamic context adaptation as problem distributions shift, and systematic integration with parameter-tuning for maximal synergetic gains.

Conclusion

"Decocted Experience Improves Test-Time Inference in LLM Agents" provides a comprehensive framework and empirical validation for memory-based context construction in LLM agents, establishing experience decoction—summarizing, organizing, and diversifying agent memory—as central to efficient and effective test-time inference. These methods offer both theoretical insight and practical algorithms for scaling up agentic generalization and decision quality across diverse, real-world tasks, with significant implications for the design of future memory-augmented AI systems.