Text2Mem Bench: Formal Memory Control Evaluation

• The paper introduces Text2Mem Bench as a new evaluation framework for mapping natural language memory directives to formally specified, executable memory operations.
• It employs explicit JSON schemas, typed operation objects, and a validator–parser–adapter pipeline to ensure deterministic memory control.
• The benchmark supports multi-operational instructions and assesses both planning layer schema generation and backend execution consistency.

Text2Mem Bench is a planned benchmark for evaluating the pathway from natural language memory commands to reliable memory control under the Text2Mem operation language. It is introduced as the evaluation counterpart to Text2Mem’s unified memory operation language and is organized around explicit semantics, JSON-based schemas, typed operation objects, a validator–parser–adapter pipeline, and a strict separation between schema generation and backend execution. In this design, the benchmark does not treat “memory” as an informal conversational capability; it treats memory control as a formally specified interface spanning encoding, storage, and retrieval, with validity, invariants, execution behavior, and cross-backend portability all made benchmarkable (Wang et al., 14 Sep 2025).

1. Historical motivation and problem setting

Text2Mem Bench is motivated by two core problems in current LLM-based agent memory systems. The first is the limited and fragmented character of their operation sets. Existing agent memory frameworks, including MemOS, mem0, and Letta/MemGPT, mostly expose CRUD-like operations such as encode, retrieve, and delete, while higher-order operations such as Promote, Demote, Merge, Split, Lock, and Expire are either missing, unsupported, or only partially supported. The second is the absence of a formal, executable specification for memory commands. Natural-language directives such as “get rid of old notes” or “make rent top priority” leave scope, duration, lifecycle rules, access control, and modalities implicit, so identical utterances can produce different and sometimes unsafe behavior across systems (Wang et al., 14 Sep 2025).

Text2Mem Bench is proposed as the mechanism for evaluating whether these ambiguities have been resolved. Its role is to test whether models can map natural language into correct Text2Mem schemas and whether execution backends can carry out those schemas deterministically and safely. This places the benchmark at the boundary between semantic parsing and systems evaluation: one layer measures language understanding over a formally constrained schema space, while another measures state transitions, retrieval behavior, lifecycle effects, and invariant preservation in concrete backends.

A central consequence of this formulation is that Text2Mem Bench targets memory control rather than memory retention alone. It is concerned with whether an agent can issue and execute well-formed operations over a memory substrate, rather than only whether it can recall content from context.

2. Language substrate and executable semantics

The benchmark is inseparable from the Text2Mem operation language because the benchmark’s notion of correctness is defined by that language’s inventory, schema, and invariants. Text2Mem defines 12 operations partitioned by cognitive stage. The encoding stage (stage = "ENC") contains Encode. The storage stage (stage = "STO") contains Update, Label, Promote, Demote, Merge, Delete, Split, Lock, and Expire. The retrieval stage (stage = "RET") contains Retrieve and Summarize. Each verb has a distinct functional scope and explicit invariants; for example, locked items cannot be hard-deleted, Expire must specify a finite horizon, and Merge must preserve lineage rather than overwrite (Wang et al., 14 Sep 2025).

Every Text2Mem instruction is a JSON schema instance with global keys stage, op, target, and args, plus optional meta. The target field selects relevant memories by IDs, tags, filters, or queries, while args provides operation-specific arguments. The schema is governed by JSON Schema rules and cross-field invariants. Required fields are specified via if–then clauses; for instance, Encode requires args.payload, and Promote requires at least one of priority, weight_delta, or remind. Cross-field rules constrain legality and semantics, including the prohibition on hard deletion when locked == true and the requirement that Expire specify ttl or until.

Text2Mem Bench treats these schema rules as the formal definition of a valid benchmark output. A model that emits structurally plausible JSON but violates an invariant is not merely syntactically imperfect; it fails the benchmark’s semantic contract. In this respect, the benchmark uses the schema not only as an annotation format but as an executable specification of memory behavior.

The execution path is defined by the validator–parser–adapter pipeline. The validator checks required keys, types, allowed values, and invariants, returning structured errors for violations. The parser converts schemas into typed operation objects such as PromoteOp or EncodeOp, normalizing time expressions to ISO8601 or RFC5545, priority aliases to a fixed enumeration, and tags to canonical sets. The adapter executes typed operations either on a SQL-based prototype backend or through adapters into real frameworks such as MemOS, mem0, and Letta, integrating model-based services such as embeddings or summarization when required. Results are returned through a unified ExecutionResult contract (Wang et al., 14 Sep 2025).

3. Benchmark architecture: planning layer and execution layer

Text2Mem Bench is explicitly organized into two layers: a planning layer for schema generation and an execution layer for backend behavior. This separation is the benchmark’s defining methodological choice. Natural-language interpretation is treated as inherently noisy and model-dependent, whereas backend semantics are required to be deterministic. By separating these layers, the benchmark can evaluate language-to-schema mapping without conflating it with backend variability, and it can reuse gold schemas to compare heterogeneous backends under identical inputs (Wang et al., 14 Sep 2025).

Formally, let $u$ be a natural language command, let $\mathcal{S}$ be the Text2Mem schema space constrained by JSON Schema and invariants, and let $\mathcal{B}$ be the set of execution backends. The benchmark tests a planning function

$f_{\text{plan}}: u \mapsto s \in \mathcal{S}$

and an execution function

$f_{\text{exec}}^{b}: s \mapsto \Delta M_b$

where $\Delta M_b$ is the state change induced in backend $b \in \mathcal{B}$. The planning problem asks whether a model can produce a syntactically valid and semantically correct schema for the utterance. The execution problem asks whether a backend respects invariants and matches gold state transitions for a given schema.

At the planning layer, inputs can be single-turn commands such as “Promote all tasks to high priority with weekly reminders” or longer interactive dialogues containing corrections, ellipsis, and clarification. Outputs are one or more Text2Mem schema instances. The benchmark therefore covers both atomic operation selection and multi-operation decomposition.

At the execution layer, the input is a valid schema instance, often the gold schema, together with the current memory state. The output includes updated memory state $M'$, retrieval results for Retrieve and Summarize, and an ExecutionResult summarizing outcome. This enables operation-level checking of edits, lineage links, ranking shifts, lifecycle triggers, and cross-backend consistency.

The paper further implies three practical evaluation axes: a Schema Generation Track for natural language to Text2Mem JSON, an Execution Track for schema to state behavior, and an End-to-End Track that measures the complete pipeline from natural language through model-generated schema to backend execution. The use of “implies” is important here: these are not presented as official named tracks, but as practical instantiations of the two-layer design.

4. Scenario design and data construction

The benchmark dataset is designed around several instruction families. Operational instructions cover direct memory management such as encode, delete, merge, split, and lock. Inquiry instructions cover retrieval and summarization, for example “remind me of…” and “summarize notes about…”. Conditional instructions target lifecycle and rule-based behavior, such as “after six months, demote this to archive”. Interactive instructions cover multi-turn dialogue, clarifications, and repairs, explicitly described as inspired by CoSQL (Wang et al., 14 Sep 2025).

For each operation type, the dataset includes direct commands with explicit verbs, indirect intents phrased in everyday language, and multi-step compositions in which a single utterance decomposes into multiple atomic operations. Utterances are designed to exhibit stylistic diversity, ellipsis, pronoun use, and ambiguity requiring disambiguation through Text2Mem semantics. This means the benchmark does not reduce evaluation to verb spotting; it evaluates whether a system can recover formal intent from conversationally realistic language.

The schema is stressed by systematic parameter variation. Time ranges vary between absolute and relative forms, including durations such as "P6M" and RFC5545 recurrence rules. Tags and facets vary between single and multiple tags and include matching modes such as "match": "all". Priority levels vary across high, medium, low, and weight deltas. Lifecycle and permission settings vary across fields such as read_perm_level, write_perm_level, lock.mode, and on_expire. The dataset also includes malformed schemas and near-miss mappings, such as cases where Delete is emitted when Demote is intended, જેથી validator behavior and semantic error detection can be measured.

Two prototype examples illustrate the intended benchmark structure. In one example, the instruction “Promote all tasks to high priority with weekly reminders, but restrict edits to the owner; subtasks remain append-only for the team” decomposes into three operations: Promote, Update, and Lock. In the planning layer, the question is whether a model can generate that three-operation schema or an equivalent variant; in the execution layer, the question is whether priority, reminders, permission fields, and append-only locking are realized correctly in the backend. In a second example, “Encode this paper PDF as a reference with embeddings, set read access to team and write access to maintainers, and after six months demote it to low-priority archive” decomposes into Encode, Update, and Expire, testing payload handling, permission updates, embedding use, and delayed lifecycle demotion (Wang et al., 14 Sep 2025).

5. Evaluation methodology and metrics

The planning layer uses both structural and semantic metrics. Schema validity is the fraction of outputs that pass the validator. Slot accuracy measures per-field matching between predicted and gold schemas. If $F$ is the field set, then

$$\text{SlotAcc} = \frac{1}{|F|} \sum_{f \in F} \mathbf{1}\left[\hat{s}_f = s^\ast_f\right].$$

Key–value precision, recall, and $\mathcal{S}$0 treat each (key, value) pair as an item:

$\mathcal{S}$1

$\mathcal{S}$2

Because multiple schemas can be acceptable for a single utterance, the benchmark also includes semantic scoring via an “adjudicator” LLM that judges equivalence beyond exact match. Finally, executability in the prototype SQL backend is measured as a binary criterion indicating whether the schema can be parsed and executed without validator or parser errors (Wang et al., 14 Sep 2025).

The execution layer evaluates state consistency, retrieval behavior, lifecycle correctness, invariant preservation, and portability. For editing operations such as Update, Label, Delete, Merge, Split, and Lock, the benchmark checks field changes and lineage links. For Promote and Demote, it checks ranking shifts, retrieval scores, or ordering changes. For Expire, it checks whether items expire at the correct horizon and whether the specified on_expire action is applied correctly. Safety evaluation measures invariant violations such as hard deletion of locked items or missing finite horizons for Expire. Portability is measured by comparing the effect of the same typed object across different backends.

Latency is mentioned as potentially important for production systems, but it is not detailed in the paper. More generally, the benchmark’s methodology emphasizes deterministic execution and comparable state-transition checking rather than purely end-task utility metrics.

6. Experimental status, intended uses, and roadmap

Text2Mem Bench is explicitly described as planned rather than released. The paper does not report concrete experimental results, baseline tables, or performance numbers for actual models or systems on the benchmark. Instead, it sketches intended usage patterns: base-model testing for schema generation, cross-backend comparisons using gold schemas, and automated execution tests using scripted scenarios. The stated aim is to expose limitations such as difficulty in generating fully valid, invariant-respecting schemas from ambiguous language and backend inconsistency or partial support for advanced operations such as Lock, Expire, Merge, and Split (Wang et al., 14 Sep 2025).

For researchers, the benchmark is positioned as a way to evaluate memory control capabilities, compare memory backends, and study compositional behavior over long horizons. For practitioners, it is framed as a tool for hardening memory subsystems, validating schema-generation models before deployment, and verifying portability across frameworks such as MemOS, MemGPT, mem0, and internal stores. The SQL prototype backend and ExecutionResult contract are presented as auditable references for debugging and certification.

The roadmap described in the paper includes an initial release with a compact dataset spanning all 12 operations and a small number of instances per verb with diverse styles and parameters. It also includes staged scaling to larger scenarios and bilingual or multilingual support, with the benchmark remaining tightly coupled to the evolving Text2Mem schema. Potential extensions are only implied, not detailed; the paper suggests possible additions such as new operation types, richer scheduling semantics, more backends, community contributions, and new validator rules. This suggests a benchmark intended to evolve with the operation language rather than remain a fixed static suite.

7. Relation to adjacent benchmarks and research programs

Text2Mem Bench is described as strongly inspired by text-to-SQL benchmarks such as Spider and CoSQL, and by later schema-centric evaluation efforts such as BIRD and LogicCat. The analogy stated in the paper is that natural-language memory commands today resemble early natural-language SQL queries: vague, implementation-specific, and lacking a shared schema. On this view, Text2Mem is to memory operations what SQL is to databases, and Text2Mem Bench is to memory control what Spider, CoSQL, and BIRD are to text-to-SQL (Wang et al., 14 Sep 2025).

Relative to adjacent memory benchmarks, the benchmark occupies a distinct niche. Minerva evaluates a programmable set of synthetic memory tests over the LLM’s input context, including search, recall, editing, structural awareness, and stateful processing, and it emphasizes capability-level diagnostics rather than standardized memory-control operations (Xia et al., 5 Feb 2025). EnterpriseMem-Bench evaluates multi-turn Text-to-SQL with deterministic ground truth, explicit memory-critical turn annotations, and the Memory Benefit Score, isolating working memory, episodic retrieval, and semantic augmentation in a conversational analytics setting (Tummalapenta et al., 25 May 2026). MemoryBench focuses on continual learning from simulated user feedback across domains, languages, and task formats, treating procedural memory from service-time interactions as the central object of evaluation (Ai et al., 20 Oct 2025). MemGym evaluates agentic memory across tool-use dialogue, deep research, coding, and computer use, and reports memory-isolated scores that decouple memory performance from reasoning, retrieval, and tool-use ability (Xu et al., 20 May 2026).

Against that background, Text2Mem Bench’s novelty lies in a unified operation language for encode–store–retrieve behavior, explicit schema and invariants, separation of schema generation from backend execution, and portability across heterogeneous backends via typed objects and adapters. The benchmark therefore addresses a different problem from long-context recall, task-specific memory QA, continual-learning-from-feedback, or agentic summarization under context pressure. Its stated role is to provide a language-centered benchmark for memory operation control, furnishing what the paper characterizes as the first standardized foundation for memory control in agents (Wang et al., 14 Sep 2025).