ToolMem: Neural Tool Memory Framework

• ToolMem is a memory framework that organizes experiential data into proficiency categories to assess neural tool capabilities.
• It employs embedding-based retrieval and dynamic updates to refine memory entries from real-world task feedback.
• Empirical evaluations show notable improvements in tool selection accuracy and performance prediction for multimodal neural APIs.

ToolMem is a learnable, scenario-driven tool capability memory framework for enhancing the tool selection and performance prediction abilities of agents operating over neural APIs—specifically, LLMs and vision-LLMs (VLMs)—across text and image modalities with fundamentally nondeterministic outputs. Unlike conventional deterministic software tools, neural tools exhibit highly input-dependent performance variations, making fixed, static tool descriptions inadequate. ToolMem enables multimodal agents to accumulate, organize, and retrieve explicit, experience-derived memories of tool performance, thus supporting accurate task-specific tool selection and predictive estimation (Xiao et al., 8 Oct 2025).

1. Motivation and Problem Context

Agents leveraging neural APIs for text or vision tasks often face a scenario mismatch between static, human-authored tool descriptions ("docstrings") and the actual, variable capabilities of neural tools. For example, one text generator may excel at long-form reasoning but underperform on syntactic correctness, while image generators may handle backgrounds or embedded text with differing proficiency. In the absence of explicit experiential knowledge, agents default to the parametric biases of LLMs or VLMs, inducing high error rates in tool-performance forecasting and suboptimal tool choice. ToolMem addresses this foundational shortcoming by endowing the agent with a dynamic, learnable repository of tool-specific strengths and weaknesses, built from prior interactions and tailored to actual execution scenarios, thus supporting adaptive inference-time selection (Xiao et al., 8 Oct 2025).

2. Architecture and Memory Representation

2.1 Memory Structuring

ToolMem organizes the toolset $T = \{t_1, ..., t_n\}$, each with high-level functional similarity, into a structured memory $\mathcal{M}$. Entries are partitioned into four canonical proficiency categories: $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$. For each tool $t$, memory $M_t$ consists of textual scenario-level entries $m$ describing qualitative tool behaviors, e.g. "weak at rendering legible text overlays" or "good at scenes with natural lighting". The global memory bank is the union over all tools and categories: $$\mathcal{M} = \bigcup_{t\in T} M_t = \bigcup_{c\in C} \mathcal{M}^c$$ (Xiao et al., 8 Oct 2025).

2.2 Memory Encoding and Retrieval

Each entry $m$ (memory) and each new experience $e = (q, s, r)$ (task prompt, tool output, feedback) are embedded into a shared $d$-dimensional space via a frozen encoder $\mathcal{M}$0 (text-embedding-ada-002):

• $\mathcal{M}$1
• $\mathcal{M}$2

Similarity (typically cosine or dot product) governs both category-specific retrieval and relevance ranking at both update and inference time.

2.3 Memory Induction and Update

Upon encountering a new experience $\mathcal{M}$3, ToolMem's prompting LM induction module $\mathcal{M}$4 generates a candidate entry $\mathcal{M}$5. The update routine:

1. For each $\mathcal{M}$6, retrieve top-$\mathcal{M}$7 similar entries $\mathcal{M}$8.
2. Merge to form context $\mathcal{M}$9.
3. Refine entries using $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$0, yielding $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$1.
4. Replace: $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$2.

Formally, the update operator $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$3 is defined as: $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$4

2.4 Inference-Time Retrieval

Given a new query $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$5, for each category $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$6, retrieve top-$$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$7 entries: $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$8. All selected entries are injected verbatim into the LLM/VLM prompt, supporting two downstream policies:

• Predict tool quality score $$C = \{p: \text{proficient at},\, g: \text{good at},\, b: \text{bad at},\, w: \text{weak at}\}$$9
• Direct tool selection among candidate tools using predicted scores

3. Formalism and Supervised Objectives

The retrieval and update mechanisms are underpinned by formal similarity search and entry refinement. At training, downstream objectives use held-out datasets:

• Performance Prediction (Regression):

$t$0

Metrics: MAE, RMSE.

• Tool Selection (Binary Preference):

Soft-predicted tool-pair preferences are scored by cross-entropy:

$t$1

Key metrics: accuracy, $t$2.

These tasks exploit ToolMem's memory-augmented context for superior predictive signal compared to generic agents or few-shot retrieval strategies (Xiao et al., 8 Oct 2025).

4. Empirical Evaluation and Benchmarks

4.1 Benchmarks

Two multimodal domains are targeted:

• Text Generation: BiGGen Bench—696 examples, 8 dimensions, 6 model-based tools (GPT-3.5-Turbo, Claude-3, LLaMA-3-70B, Qwen-110B, Gemma-2B, Qwen-0.5B). Labels: GPT-4 Likert ratings.
• Text-to-Image: GenAI-Bench—1600 prompts, 6 generators (DALL·E 3, MidJourney 6, DeepFloyd I-XL, SDXL Turbo, SDXL 2.1, SDXL Base). Labels: human Likert, VQA alignment.

4.2 Performance Gains

Task	MAE Reduction	Tool Selection Accuracy Gain
BiGGen (text)	14.8%	+21% absolute
GenAI-Bench	28.7%	+24% absolute

• For weaker models (e.g., Qwen-0.5B, Gemma-2B), ToolMem raises Pearson correlation from near-zero to >0.32.
• VQA alignment in text-to-image generation increases by 2.0–4.2% for mid/low-tier models.
• ToolMem achieves F1 increases in pairwise tool ranking (e.g., $t$3 from 0.09 to 0.32 for lesser vs. better tools).

Comparison baselines:

• Generic Agent: Static names and docstrings, no experience-driven memory.
• Few-Shot Retrieval: Embedding-based retrieval of raw (q,score,rubric) triplets.
• ToolMem: Injects refined, scenario-level capability memories, outperforming both baselines (Xiao et al., 8 Oct 2025).

5. Analysis, Strengths, and Limitations

Strengths

• Substantially improves performance prediction and tool selection, especially for weak-base neural tools without strong LLM prior encoding.
• Memory entries are both compact and human-interpretable, avoiding the brittleness of raw few-shot exemplars.
• Framework is modality-agnostic, unifying text and vision under shared memory representations.

Limitations

• Dependent on quality of downstream feedback (e.g., GPT-4 or human raters); biases propagate directly.
• Assumes stationary tool capabilities; frequent updates can induce staleness in memory.
• Memory induction and refinement are prompt-driven (heuristic), lacking end-to-end trainable summarization modules.

6. Future Directions

Key research avenues include:

• Automating the memory induction process with learned summarization or fine-tuning approaches in place of prompt engineering.
• Version control and timestamping to address tool evolution and maintain memory relevance.
• Theoretical analysis of memory consolidation, specifically bounding retrieval and update complexity as memories grow.
• Extending the framework to broader classes of tool ecosystems, such as code generators and database engines, including human-in-the-loop refinement for critical or rare-case failure modes.

ToolMem thus constitutes a foundational shift in neural tool utilization, treating the agent's accumulated, structured experiential knowledge as a first-class object—mirroring the human process of developing tool intuition through practice and feedback (Xiao et al., 8 Oct 2025).