MemToolAgent: Leveraging Memory for Tool Using Agents Based on Environment and User Feedback

Abstract: Modern LLM agents can use external tools to help users solve complex tasks. However, for problems that require learning from long-term historical events or from previous agent-environment interactions, LLM agents are required to use memory mechanisms to store and retrieve experiences. While sophisticated memory systems exist for dialogue agents, few studies have empirically examined how to improve agents' tool-using capabilities through past user-agent conversations. We propose MemToolAgent, a framework that improves tool use through memory management. Our approach contains a memory extraction module that processes past experiences into structured memory entries, and a retrieval module that dynamically selects a subset of the stored memory entries. This enables more personalized and accurate responses aligned with user preferences and feedback without requiring LLM fine-tuning. In summary, this work has three main contributions: (1) a unified memory entry format that improves both general-purpose and personalized tool use without LLM fine-tuning, (2) a reflection-based memory extraction that uses environment and user feedback to distill wrong executions into critiques to store, and (3) a retrieval module that chooses how many past experiences to use based on the memory similarity distribution. MemToolAgent achieves 29%, 80%, and 17% relative improvements compared to strong baselines on the WorkBench, NESTFUL, and PEToolBench benchmarks, respectively.

• The paper introduces a unified memory-augmented framework that leverages reflective critiques and dynamic retrieval to improve tool execution accuracy.
• It utilizes a generalized memory entry format and derivative-based selection to efficiently capture and reuse key episodic experiences and user feedback.
• Empirical results demonstrate significant accuracy gains, with improvements in multi-step reasoning and personalized tool orchestration over baseline approaches.

MemToolAgent: Integrating Memory with Tool-Using LLM Agents via Environment and User Feedback

Introduction

MemToolAgent introduces a unified memory-augmented architecture for LLM-based tool-using agents, addressing deficiencies in prior approaches for learning from historical records, agent-environment interactions, and user feedback. The framework is motivated by the observation that conventional LLM-based agents struggle with long-term coherence, personalization, and accurate task execution—especially in settings demanding tool invocation, parameterization, and task-dependent reasoning. Fixed context-window limitations in LLMs impede effective utilization of historical context and exacerbate error rates in multi-step reasoning and planning, which are critical in tool-use scenarios. MemToolAgent provides a structured mechanism to extract, encode, and retrieve high-salience episodic experiences and user critiques, without LLM fine-tuning, to efficiently adapt agent behavior in both general-purpose and personalized settings.

Methodology

The central architectural contributions of MemToolAgent are:

• Generalized Memory Entry Format: Rather than ad hoc or domain-specific representations, memory entries encapsulate $(q, a, f, r)$—the user query, executed sequence of tool calls, binary feedback (success or failure), and a reflection/critique on error traces or parameter failure. This abstraction enables effective transfer across both personalized and general-purpose contexts.
• Reflection-Based Memory Extraction: The framework systematically distills incorrect executions into compact, structured critiques using an LLM-based memory extraction module. The memory extractor takes the agent's action trace, user/environmental feedback, available tools, and semantically similar past cases as context, then generates targeted critiques for incorporation into memory. This avoids overloading the context with raw traces and irrelevant details, streamlining memory usage for direct error prevention.
• Dynamic "n" Top-n Embedding-Based Retrieval: Instead of fixed k-NN retrieval, MemToolAgent decides the optimal number of relevant experiences to include for a query via peak-finding on the sorted similarity distribution (using first-order derivatives for robustness against noise and cluster overlap). This prevents both information under-selection (missing relevant context) and over-selection (contextual pollution), in contrast with prior static or "adaptive-k" methods.

Empirical Results

General-Purpose Tool Use

On the WorkBench benchmark, MemToolAgent yields a 29% relative increase in overall accuracy over the strongest baselines. For instance, Analytics task accuracy increases from 18% (no-memory baseline) to 86%, and overall accuracy surges from 57% to 85%. For NESTFUL—a challenging benchmark with nested, multi-step tool call reasoning—MemToolAgent nearly doubles Full Sequence Matching (15.6% to 30.4%) and achieves a Win Rate increase from 51% to 70%. Simple extended reasoning mechanisms like "Extended Thinking" and ReAct are unable to approach these gains, highlighting the essential role of structured and reflective memory for error pattern avoidance and parameterization.

Personalized Tool Orchestration

On PEToolBench, MemToolAgent exceeds existing memory-centric architectures such as A-MEM and Mem0, improving tool accuracy from 0.70 to 0.82 (17% relative increase). Parameter generation accuracy is on par with baselines, suggesting that the dominant improvements stem from accurate tool selection and preference-aligned orchestration, not merely argument synthesis.

Ablations and Insights

Ablation studies further demonstrate:

• Impact of Memory Extraction: Reflection-based entries (i.e., critiques rather than raw traces) materially improve both learning-from-failure and generalization to new queries, with notable boosts in Win Rate and Sequence Matching on NESTFUL and improved Tool Accuracy on PEToolBench.
• Fixed vs. Dynamic Retrieval: The Dynamic "n" retrieval strategy consistently outperforms fixed top-k schemes, as it is query-adaptive and robust to similarity curve structure, avoiding both redundancy and omission.
• Cold Start Mitigation via Synthetic Memory: Synthetic query generation partially mitigates the cold-start issue, where memory is initially empty or lacks relevant traces. However, multi-domain tasks remain challenging due to the higher complexity and diversity of required queries.
• Preference Guiding: Hard preference guiding—where entries rated as user non-preferences are omitted—acts as a simple but effective competitive baseline for personalization benchmarks, though it provides diminished generalization compared to critique-based extraction.

Practical and Theoretical Implications

MemToolAgent's modular design and reliance on reflection-based experience abstraction uniquely positions it for rapid deployment in real-world applications where end-user behavior deviates from training data or evolves dynamically over time (e.g., enterprise workflow automation, personalized code repair, multi-user dialog orchestration). By decoupling experience learning from core LLM parameters, the agent remains adaptable and interpretable—supporting continual improvement from live feedback and error correction, a property especially crucial for alignment, safety, and regulatory auditing in high-stakes domains.

From a theoretical perspective, MemToolAgent bridges the gap between classic CBR paradigms and in-context learning paradigms for LLMs, fusing memory-based adaptation with dynamic retrieval. Its use of derivative-based retrieval for cluster boundary detection in embedding space introduces robust context selection, which may generalize to a range of retrieval-augmented agentic systems beyond tool use.

Limitations and Future Directions

Despite its improvements, MemToolAgent faces several limitations:

• Cold Start: The agent is dependent on memory for improved accuracy. Synthetic query generation partially addresses this, but coverage gaps persist—especially as toolsets scale.
• Scalability: Benchmarks involve a limited number of tools (sub-40); scaling to production-sized toolboxes and high-variance, multi-turn conversations will likely require hierarchical memory and efficient indexing strategies.
• Single-Turn Bias: Current benchmarks are single-turn oriented; extension to multi-turn, session-based memory will be necessary for dialog-driven agents, possibly integrating per-turn retrieval or active memory updating.

Future research could explore hierarchical or specialized memory modules (procedural, episodic, semantic) and direct integration of multi-modal or real-time feedback signals. There is also ample opportunity to optimize memory operation costs and latency for deployment-centric constraints.

Conclusion

MemToolAgent significantly advances the state-of-the-art in memory integration for LLM agents engaged in tool use, providing robust, architecture-agnostic mechanisms for leveraging user and environment feedback without LLM fine-tuning. Empirical gains on established general-purpose and personalization benchmarks substantiate the effectiveness of memory extraction via reflection/critique and dynamic context selection. While limitations regarding cold-start and scalability persist, the presented methodology offers a clear pathway for future research in adaptive agents, continual test-time learning, and human-centric reasoning systems (2606.07909).