Mem2ActBench: Benchmark for Memory-to-Action Tasks

Updated 5 July 2026

Mem2ActBench is a benchmark that evaluates agents’ ability to convert long-term memory traces into actionable tool calls from underspecified queries.
It uses an automated pipeline integrating multi-session dialogues, heterogeneous data sources, and custom fact extraction to construct coherent memory evolution chains.
Empirical results reveal modest memory-dependent parameter grounding performance, highlighting persistent retrieval bottlenecks and inference challenges.

Mem2ActBench is a benchmark for evaluating long-term memory utilization in task-oriented autonomous agents, specifically the ability to actively use long-term memory to execute tool-based actions rather than merely retrieve isolated facts in response to explicit questions. It targets persistent assistant usage in which users mention the same topic across long, interrupted interactions and later expect previously established preferences, task states, and identifiers to be implicitly applied when the agent selects tools and grounds tool parameters from underspecified requests (Shen et al., 13 Jan 2026).

1. Conceptual scope and formal task definition

Mem2ActBench is organized around a distinction between passive retrieval and active application. Earlier memory benchmarks largely instantiate a question-answering pattern in which the system is explicitly asked to recall a fact from history. Mem2ActBench instead evaluates inference-driven memory utilization: the final user query clearly specifies the intended task, but omits critical arguments such as dates, cities, budgets, preferences, or account identifiers. The agent must infer which prior information is relevant, retrieve it from long-term memory, and incorporate it into an executable tool call (Shen et al., 13 Jan 2026).

The benchmark formalizes tool-call generation over a long history. Let $\mathcal{M}$ denote the memory sequence, $q$ the final user query, and $\mathcal{C}$ the space of possible tool calls. A tool call $c$ consists of a tool $T$ and parameter values $\{v_p\}$ , and the agent is expected to produce

$\hat{c} = \arg\max_{c \in \mathcal{C}} P(c \mid q, \mathcal{M}).$

For each parameter $p$ , the grounded value is modeled as

$v_p = f_{\theta}(p, q, \mathcal{M}),$

with the additional constraint that $v_p$ be strictly grounded in $q$ 0, or in the tool schema’s default when appropriate. This constraint is central: Mem2ActBench does not treat plausible but ungrounded arguments as correct (Shen et al., 13 Jan 2026).

A benchmark instance contains four core elements. The first is an underlying long-term interaction derived from multi-session dialogue, including user utterances, assistant responses, tool calls, and tool outputs. The second is a memory evolution chain, a globally consistent ordered sequence of facts extracted from those sessions. The third is a gold tool call $q$ 1 whose tool and parameters are validated against the memory chain. The fourth is a reverse-generated final user query $q$ 2 whose wording preserves the original intent while omitting key parameter values, forcing the agent to reconstruct the tool invocation from memory (Shen et al., 13 Jan 2026).

2. Data construction, consistency modeling, and memory dependence

Mem2ActBench is produced by an automated pipeline with three major phases: heterogeneous data integration, construction of a fact or memory evolution chain, and reverse-generation of memory-dependent tool-use tasks. The benchmark merges three sources with complementary roles: ToolACE for realistic function-calling traces, BFCL v3 for diverse tool types and structured gold tool calls, and OASST1 for interruption-heavy conversational noise. These sources are normalized into a unified multi-turn JSONL format (Shen et al., 13 Jan 2026).

The integration process is explicitly engineered. ToolACE raw traces with bracketed function-call syntax are parsed using a custom stack-based algorithm to resolve nested calls, then rephrased into natural dialogue with Qwen3-Next-80B at temperature 0.0. BFCL v3 static tool-use tasks are expanded into multi-turn dialogues with clarifications and sequential calls using the same model. OASST1 threads are reconstructed by following leaf-to-root paths; only rank=0 assistant responses are retained; non-English data are translated into English; and duplicate threads are handled via longest-path selection (Shen et al., 13 Jan 2026).

Consistency modeling proceeds through fact extraction, semantic clustering, local conflict resolution, and global ordering. Facts are extracted as structured tuples of the form $q$ 3 with an internal source identifier. Attributes are normalized and entity-bound, so facts about different entities are not merged merely because they share a topic label. Attribute clustering uses BERTopic with BGE-M3 embeddings and an HDBSCAN backend, with settings including min_cluster_size=2, metric=euclidean, cluster_selection_method=leaf, and epsilon=0.01. Within each attribute group, an LLM chronologically orders facts, removes obsolete or contradictory items, distinguishes updates from refinements, and produces a narrative summary. Local sequences are then merged into a global evolution chain by a modified Kahn’s topological sort with deterministic cycle breaking, yielding a globally sorted fact sequence $q$ 4 and a discard set $q$ 5 for conflicting facts (Shen et al., 13 Jan 2026).

The resulting corpus comprises 2,029 sessions, with about 12 user–assistant–tool turns on average, and a final evaluation set of 400 tool-use tasks. Human verification reports 96.5% accuracy for sampled fact extraction, 86.7% logical correctness for conflict resolution chains, and 91.3% memory dependency validity. In this setting, “strongly memory-dependent” means that the gold tool invocation is underdetermined by the final query alone and that at least one required parameter must be recovered from prior memory rather than inferred from the query or schema in isolation (Shen et al., 13 Jan 2026).

3. Reverse generation, task structure, and evaluation criteria

The reverse-generation stage begins from a memory chain $q$ 6. A target tool $q$ 7 is selected through a hybrid procedure in which BM25 and BGE-M3 retrieve candidate tools and Qwen3-Next-80B-A3B-Instruct chooses the most appropriate tool after reading tool descriptions and memory. Parameter construction then extracts candidate values from $q$ 8 through LLM processing plus fuzzy matching, and an LLM verifier checks whether each value can be traced back to memory. The result is a fully specified tool call in which parameters are either explicitly supported, logically inferred from memory, or taken from schema defaults under a memory-anchoring constraint (Shen et al., 13 Jan 2026).

The final user query is then generated in reverse from the gold tool call and its supporting memory subset using Kimi-K2-Thinking. Three constraints govern this stage. First, parameter omission: all key values in the gold tool call must be absent from the final query. Second, reference dependency: the query must rely on anaphora or indirect references such as “that trip,” “my usual settings,” or “the budget we talked about.” Third, intent preservation: the wording must still clearly correspond to the same underlying task, such as booking, searching, or account management. The generation prompt also prohibits overt references to tools or APIs (Shen et al., 13 Jan 2026).

Two filters enforce genuine memory dependence. A lexical leakage filter rejects queries containing exact parameter strings, numeric matches, or distinctive partial overlaps for structured identifiers. A solvability discriminator then gives another LLM only the final query and tool schema, without access to memory; if it can reconstruct the gold tool call, the sample is rejected as insufficiently memory-dependent. Samples are also discarded if the final query introduces constraints not supported by memory. The retained tasks therefore require the system to use long-term memory rather than exploit semantic leakage in the query (Shen et al., 13 Jan 2026).

Evaluation centers on the generated tool call. The main metrics are parameter-level F1, BLEU-1 over argument strings, and Tool Accuracy, defined as correctness of the tool together with exact matching of all parameters. Additional diagnostic metrics include slot accuracy, Tool Selection Accuracy, Exact Match, and Arg_F1, which measures parameter F1 conditioned on selecting the correct tool. In the main experiments, the ground-truth tool name is often given, so differences primarily reflect memory-dependent parameter grounding rather than tool-choice error (Shen et al., 13 Jan 2026).

4. Baseline memory frameworks and empirical performance

Mem2ActBench evaluates seven representative memory systems under a controlled setup. All methods are paired with Qwen2.5-7B, Qwen2.5-32B, or Qwen2.5-72B-Instruct as the backbone LLM, retrieval-based systems use BGE-M3, and decoding is deterministic with temperature 0.0 (Shen et al., 13 Jan 2026).

Framework	Memory formulation	Role in benchmark
LTMemory	Simple retrieval-augmented memory	Baseline for retrieval quality
SCM	LLM-managed write, update, delete	Explicit memory manager
Generative Agents	Relevance, recency, importance, reflection	Observation-style memory abstraction
LangMem	Vector stores, summarization, time-decay	Production-style long-term memory
MemTree	Hierarchical tree memory	Structured retrieval and summarization
Mem0	Scalable persistent memory	Practical integration baseline
A-Mem	Agentic storage, compression, retrieval	Long-horizon memory framework

Across model sizes, the best argument-grounding performance remains modest. A-Mem achieves average F1 of about 33.55, LTMemory about 31.97, and MemTree about 29.90. Tool Accuracy is much higher and tightly clustered, in the range of about 87–97% when the correct tool is given. This indicates that most failures are semantic rather than syntactic: systems often produce a structurally valid call but ground one or more arguments incorrectly (Shen et al., 13 Jan 2026).

Scaling the backbone improves performance only partially. Mean F1 rises from about 20.4 at 7B to about 28.9 at 72B, while gains from 32B to 72B are modest at roughly +3.5 F1. Mem0 exhibits comparatively large scaling gains, suggesting that larger models can partially compensate for weaker memory management. Even so, the aggregate picture remains that long-term memory-to-action is far from solved: the best frameworks stay in the low-to-mid 30s F1 on the core argument-grounding task (Shen et al., 13 Jan 2026).

5. Difficulty profile, retrieval bottlenecks, and failure taxonomy

Mem2ActBench is deliberately difficult because it combines inference, memory evolution, noisy history, and underspecified tool use. Relevant facts are scattered across long, interruption-heavy sessions due to OASST1 noise injection. The supporting evidence for a single tool call may involve explicit facts, evolving preferences, corrected values, and structured identifiers. Final queries usually do not name the required arguments, and tool schemas can require multiple parameters or nested structures (Shen et al., 13 Jan 2026).

The benchmark quantifies memory distance by normalizing the earliest supporting evidence turn within the total conversation length. If $q$ 9 is the conversation length and $\mathcal{C}$ 0 is the earliest evidence turn, then

$\mathcal{C}$ 1

Performance is grouped into quartiles from early to late evidence positions. The paper reports a pronounced “mid-context valley”: F1 is highest when evidence occurs at the very beginning or end of the history, and drops sharply when evidence lies in the 25–50% range of the context. The lost-in-the-middle phenomenon therefore persists even when the downstream task is tool-based action rather than explicit factual QA (Shen et al., 13 Jan 2026).

Retrieval ablations further isolate the bottleneck. Without retrieval, F1 is about 10.0. Among passive retrieval strategies, the best reported configuration is hybrid retrieval with $\mathcal{C}$ 2, reaching about 30.7 F1. Oracle retrieval raises F1 to about 53.8. The gap of roughly 23 points between the best passive retrieval and oracle indicates that evidence hitting is the dominant limitation, although post-retrieval use remains imperfect as well (Shen et al., 13 Jan 2026).

The error analysis distinguishes five recurrent failure modes. A retrieval miss occurs when the required evidence is not retrieved at all. Retrieved-but-unused denotes cases where the correct evidence is present in the prompt but ignored by the model. Hallucinated default refers to the invention of a plausible-sounding parameter that is neither present in memory nor licensed by the schema default. Lossless retention failure captures corruption of long or structured values such as IDs, URLs, or addresses. Tool selection error appears when several semantically similar tools are available and the agent chooses the wrong one. The distribution of these errors shows that weaker systems are dominated by retrieval misses, whereas stronger systems increasingly fail after retrieval, implying that memory utilization is not reducible to retrieval quality alone (Shen et al., 13 Jan 2026).

6. Relation to adjacent memory benchmarks

Mem2ActBench occupies a distinct position within the broader landscape of memory evaluation. Minerva is a programmable benchmark for in-context memory use that defines tasks as mappings $\mathcal{C}$ 3 and decomposes memory capability into atomic operations such as search, recall and edit, compare, compute on sets and lists, and stateful processing, together with composite tasks like Processing Data Blocks and Composite-State Tracking. Its emphasis is capability isolation and parametrically generated context transformations, whereas Mem2ActBench focuses on reconstructing executable tool invocations from persistent, noisy interaction histories (Xia et al., 5 Feb 2025).

MemBench evaluates memory mechanisms for LLM-based agents across factual versus reflective memory and participation versus observation scenarios, and reports not only effectiveness but also efficiency and capacity. It is centered on memory-to-answer performance under streaming input, noise injection, and long-horizon scaling, with metrics such as accuracy, Recall@10, read time, and write time. By contrast, Mem2ActBench targets memory-to-action in the narrower sense of tool selection and parameter grounding from long-term conversation history (Tan et al., 20 Jun 2025).

MemoryArena moves closer to an agentic setting by defining a multi-session Memory–Agent–Environment loop in which agents retrieve from and update persistent memory across interdependent subtasks. Its evaluation uses Task Success Rate, Progress Score, and depth-wise success over domains such as bundled web shopping, group travel planning, progressive web search, and sequential formal reasoning. This suggests a neighboring but different emphasis: MemoryArena tests how memory guides future decisions in multi-session environments, whereas Mem2ActBench concentrates on whether long-term memory can be converted into correct tool calls under underspecified user requests (He et al., 18 Feb 2026).

GateMem adds a dimension largely absent from Mem2ActBench-style formulations: memory governance in shared multi-principal settings. It evaluates utility, access control, and active forgetting for shared-memory assistants in medical, office, education, and household domains, and its discussion explicitly frames GateMem as supplying the governance dimension missing from benchmarks focused on using long-term memory for task-oriented actions. A plausible implication is that Mem2ActBench characterizes whether memory can be operationalized into action, while GateMem characterizes whether that operationalization remains safe, appropriately scoped, and deletion-aware in institutional deployments (Ren et al., 17 Jun 2026).

Taken together, these comparisons indicate that Mem2ActBench defines a specific benchmark niche. It is neither a pure recall benchmark nor a full multi-session environment benchmark, and it does not address governance constraints such as access control or active forgetting. Its contribution is to make parameter grounding the principal observable of long-term memory quality: not “Do you remember X?” but “Can you use what you remember to produce the correct tool invocation when the user assumes continuity across interrupted interaction histories?”