Experience-Augmented Agents

• Experience-Augmented Agents are computational systems that store and retrieve structured records of past interactions to enhance decision-making and continual learning.
• They utilize external memory banks with selective, gated retrieval mechanisms to adjust planning, tool selection, and action synthesis in real time.
• Empirical findings show these agents achieve robust generalization and sample efficiency, with up to 60% reduction in trial-and-error and improved task performance.

Experience-Augmented Agents are computational systems—typically LLM or multimodal agents—whose decision-making, planning, or skill composition is explicitly conditioned on structured records of past interaction, feedback, and outcomes, referred to as “experience.” Unlike agents that depend solely on parameter updates during training, Experience-Augmented Agents store, retrieve, and dynamically utilize prior successful (and often unsuccessful) episodes during inference. This mechanism enables continual self-improvement, efficient adaptation to domain shift, and robust generalization across tasks and environments.

1. Core Principles and Architectures

Experience-Augmented Agents integrate the following foundational principles:

• Experience memory as externalized knowledge: Experience is operationalized as a memory bank or library, with each element comprising richly structured tuples or graphs representing problem contexts, tool invocation histories, intermediate results, and task outcomes. For example, MACRO for medical imaging stores entries as (prompt/history, tool sequence, summarized result, image feature) (Fan et al., 6 Mar 2026); GSEM maintains a dual-layer experience plus entity graph (Han et al., 23 Mar 2026); SLEA-RL employs cluster-indexed strategy/warning pools keyed by environmental state (Wang et al., 18 Mar 2026).
• Experience-driven decision augmentation: At inference (and sometimes during training), agents retrieve and incorporate relevant experience records, guiding tool selection, action synthesis, or planning steps. Experience can be injected as context for in-context learning, used as a condition for dynamic subgoal decomposition, or directly invoked as high-level “composite tools.”
• Self-evolution and continual skill/library expansion: Experience is not static; with every new episode, effective sub-trajectories, composite actions, or strategic lessons are distilled, validated (often via outcome reward), and added to the memory for future retrieval. This supports closed-loop, sample-efficient self-improvement over time.
• Separation of experience construction, retrieval, and utilization: Leading frameworks consider not only how to extract and curate experience but also how to select (when/what to retrieve), how to efficiently inject it into the reasoning loop, and how/when to optimize utilization patterns. Notably, recent work emphasizes that effective use of experience requires selective or gating mechanisms rather than rigid always-on retrieval (Zhao et al., 8 May 2026, Feng et al., 31 May 2026).

2. Canonical Experience-Augmented Frameworks and Formalizations

2.1 MACRO: Self-Evolving Medical Imaging Agents

MACRO structures the agent’s internal state as a Partially Observable Markov Decision Process (POMDP) with image-conditioned memory:

• Memory entries: m = (p, t, r, f), where p = prompt/history, t = tool sequence, r = result summary, f = image feature.
• Retrieval: For a query image, top-k most similar entries by cosine similarity in image feature space are prepended as in-context demonstrations, grounding tool selection (Fan et al., 6 Mar 2026).
• Composite tool discovery: Frequent contiguous subsequences of atomic tool calls mined from successful trajectories are registered as reusable composite primitives, expanding the action space dynamically.
• Self-improvement objective: Group Relative Policy Optimization (GRPO) combines per-trajectory and step-level rewards for composite invocation, reinforcing the efficient reuse of discovered skills.

2.2 GSEM: Graph-Based Self-Evolving Memory for Clinical Reasoning

• Dual-layer memory graph: One layer indexes entire clinical episodes (E), with nodes weighted by quality (Qᵢ); the second decomposes them into entity-level decision-flow graphs.
• Applicability-aware retrieval: Hybrid seed recall (entity and embedding-based) yields a set of candidate experiences, with multi-step graph traversal orchestrated by the LLM policy and guided by combined node/edge weights (Q, W).
• Feedback-driven updating: Scalar feedback from downstream performance directly calibrates node and edge credence without ever rewriting stored records (Han et al., 23 Mar 2026).

2.3 SLEA-RL: Step-Level Experience-Augmented RL

• Step-conditioned retrieval: At each decision step, the agent queries cluster-indexed memory pools to obtain the most structurally salient strategies and warnings for the current state.
• Experience admission and evolution: Candidates from new rollouts enter the memory only if their quality score exceeds a dynamic threshold per abstraction level, enforcing continual improvement (Wang et al., 18 Mar 2026).
• Advantage estimation: Step-level and episode-level advantages are fused in the RL loss, assigning finer-grained credit to the application of retrieved experience.

3. Methods of Experience Construction, Organization, and Distillation

• Explicit trajectory mining and abstraction: Experience is distilled not only as whole trajectories but, more effectively, as lessons, subgoal decompositions, or concept-aligned sub-sequences (decocted experience (Shen et al., 6 Apr 2026)). For structured domains, experience graphs or composite skill registries capture not only successful task completions but also the relationships and dependencies among decisions.
• Heuristic and reward-driven pruning: Large pools of raw experience are iteratively refined. Frameworks such as Iterative Experience Refinement (IER) filter stored lessons via information gain, compilation/execution success, and usage frequency, ensuring memory size and quality remain tractable without sacrificing performance (Qian et al., 2024).
• Contrastive distillation and failure diagnostics: Experience banks often include both positive demonstrations and structured attributions of failure, enabling more robust constraint enforcement and error recovery (GeoEvolver (Dai et al., 30 Jan 2026), SLEA-RL (Wang et al., 18 Mar 2026)).

4. Strategies for Experience Retrieval and Utilization

• Query-conditioned retrieval: Retrieval may be static (at episode/task initialization) or dynamic (per step/observation), using similarity in embedding or structured (workflow, concept-tree, or entity) space (Fan et al., 6 Mar 2026, Han et al., 23 Mar 2026, Shen et al., 6 Apr 2026). SLEA-RL’s cluster-conditioned retrieval exemplifies fine-grained, observation-aware memory access.
• Gated and selective invocation: Rigid, always-on injection of experience can degrade sample efficiency or distract from contextually inappropriate guidance. Gating (either hard-trigger or soft/probabilistic) based on reasoning trace entropy or explicit retrieval requests yields superior performance (Zhao et al., 8 May 2026, Feng et al., 31 May 2026).
• Latent (representation-space) integration: ExpWeaver integrates experience at the hidden-state level, executing cross-attention and gated fusion without concatenating lengthy demonstrations into the prompt, achieving token and compute efficiency without loss of accuracy (Feng et al., 31 May 2026).
• Credit assignment and benefit estimation: Frameworks such as Evolving-RL assign advantages to both experience-extraction and utilization stages, allowing the agent to distinguish between intervention points where experience is materially beneficial (Fan et al., 11 May 2026).

5. Empirical Findings and Benchmark Performance

• Robust generalization and cross-domain transfer: Experience-augmented frameworks outperform static-tool, hand-crafted workflow, and parametric-only fine-tuning baselines especially on domain-shifted or out-of-distribution tasks (e.g., MACRO beats specialized medical classifiers and prior agentic methods by up to 4.3 BACC points, SLEA-RL and ExpWeaver similarly dominate in ALFWorld, WebShop, and multi-hop QA) (Fan et al., 6 Mar 2026, Wang et al., 18 Mar 2026, Zhao et al., 8 May 2026, Feng et al., 31 May 2026).
• Sample efficiency and mistake reduction: Experience integration sharply reduces redundant trial-and-error, with observed 60%+ reductions in required episodes for embodied RL tasks (DAAG (Palo et al., 2024)) and double-digit absolute accuracy gains in combinatorial agentic domains (Dai et al., 30 Jan 2026, Shalev et al., 19 May 2026).
• Scaling laws and ablation effects: Both the total size and curation of memory significantly affect downstream performance. Over-large, unpruned experience banks lead to degradation from noise and dilution; intermediate, information-dense subsets are optimal (Shen et al., 6 Apr 2026, Qian et al., 2024). Step-level vs. initialization-only retrieval provides consistent gains in long-horizon scenarios (Wang et al., 18 Mar 2026).
• Task and environment dependence: The optimal blend of strategy distillation and experience augmentation is application-specific, as shown by EE-MCP’s finding that experience banks are superior in GUI-intensive environments while distillation prevails for API-dominant tasks (He et al., 10 Apr 2026).

Table: Representative Experience-Augmented Agent Frameworks

Framework	Key Mechanism	Application Domain
MACRO (Fan et al., 6 Mar 2026)	Composite tool mining, image memory	Medical imaging
GSEM (Han et al., 23 Mar 2026)	Dual-layer memory graph, node/edge update	Clinical reasoning
SLEA-RL (Wang et al., 18 Mar 2026)	Step-clustered retrieval, step-level RL	Multi-turn RL agents
ExpWeaver (Feng et al., 31 May 2026, Zhao et al., 8 May 2026)	Latent retrieval, gated utilization, end-to-end RL	Broad QA, coding, recsys
DAAG (Palo et al., 2024)	Hindsight diffusion augmentation	Embodied RL/robotics
WISE-Flow (Zhou et al., 13 Jan 2026)	Workflow induction, prerequisite-centric retrieval	Conversational services
EE-MCP (He et al., 10 Apr 2026)	Experience bank, inference-time rule injection	GUI/API automation

6. Limitations, Risks, and Current Challenges

• Quality and relevance of experience: The efficacy of augmentation is strongly contingent on the fidelity, diversity, and recency of accumulated memories. Irrelevant or adversarial experience records can poison downstream reasoning, as shown in the MemoryGraft attack (Srivastava et al., 18 Dec 2025).
• Combinatorial growth and manageability: Large, heterogeneous domains exacerbate the challenge of organizing, retrieving, and pruning relevant experience. Hierarchical clustering, score-based admission, or hybrid graph/tree structures are partial solutions, but scalability and latency remain concerns (Han et al., 23 Mar 2026, Shen et al., 6 Apr 2026, Qian et al., 2024).
• Robustness and security: Experience-augmented agents introduce new attack surfaces (e.g., injective poisoning of experience memory, semantic imitation vulnerabilities) requiring cryptographic provenance checks and consistency filtering (Srivastava et al., 18 Dec 2025).
• Benchmarking and cross-framework comparability: Varying implementations, representations, and evaluation metrics complicate apples-to-apples comparison; standardized repositories (e.g., MetaGym for RL (Shalev et al., 19 May 2026)) and multi-domain testing are emerging but uneven.

7. Future Directions and Emerging Themes

• End-to-end co-evolution and internalization: Joint optimization of experience extraction and utilization (as in Evolving-RL) allows agents not only to externalize and reuse skills but to internalize them into model parameters, supporting zero-shot adaptation (Fan et al., 11 May 2026).
• Selective, uncertainty-driven utilization: Fine-grained, entropy- or causality-triggered gating of experience retrieval maximizes benefit while minimizing unnecessary reliance or distraction (Zhao et al., 8 May 2026).
• Integration of multimodal, cross-domain experience: Bridging text, image, environmental state, and tool/action spaces for more robust, generalizable experience banks remains a priority across clinical, software, and embodied domains (Fan et al., 6 Mar 2026, Dai et al., 30 Jan 2026, Palo et al., 2024).
• Interpretability and abstraction-level control: Enhanced structuring of experience as graphs, workflows, or concept hierarchies supports more interpretable and composable augmentation, as demanded by safety-critical applications (Han et al., 23 Mar 2026, Zhou et al., 13 Jan 2026).
• Security, provenance, and filtering: Systematic countermeasures to experience poisoning—cryptographic attestation, risk-aware reranking, and constitutional filtering—are necessary for real-world deployment of self-improving agent architectures (Srivastava et al., 18 Dec 2025).

Experience-Augmented Agents thus constitute a rapidly evolving paradigm at the intersection of machine learning, symbolic reasoning, and decision sciences. They systematically operationalize lifelong learning and strategic self-improvement through structured interaction histories, skill abstraction, and selective deployment of past knowledge, marking a pivot from static, parameter-centric learning to continual, context-driven agency.