Papers
Topics
Authors
Recent
Search
2000 character limit reached

DolphinBench: Mapping the Pareto Frontier of Agent Memory

Published 21 Sep 2026 in cs.CL and cs.AI | (2609.24971v1)

Abstract: Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are built for a conversational question-answer format, where the question itself signals that some fact must be retrieved, and often which one. Moreover, benchmarks rarely require anything beyond accuracy from submissions, allowing memory systems to make unreasonable cost/time tradeoffs to achieve higher scores. We present DolphinBench, a benchmark that evaluates memory directly through an agent's task completion. DolphinBench includes three knowledge-work personas with roughly 500k tokens of user messages per persona and evaluates agents on tasks that depend on information from that history. We verify all 200 tasks per persona by running an agent with and without the relevant history, requiring success with it and failure without it. Finally, we require all evaluations to report total cost and latency alongside accuracy, which enables us to evaluate agent memory systems holistically. No existing memory benchmark combines all three. The dataset and evaluation code are available at https://dolphinbench.ai.

Summary

  • The paper introduces DolphinBench, a benchmark for evaluating memory in tool-using AI agents through action-based tasks, focusing on completion rather than explicit recall.
  • DolphinBench evaluates memory systems based on accuracy, cost, and latency, presenting Pareto-optimized trade-offs, with the highest result being 70.67% accuracy using Mem0 with the Hermes harness and GPT-5.6-Luna.
  • DolphinBench tests agents using three simulated professional histories, totaling 600 tasks, ensuring that memory recall is essential for task success and validated over deterministic and LLM-based checks

DolphinBench addresses a specific methodological problem in evaluating long-term memory for tool-using agents: conventional memory benchmarks typically reduce the task to question answering, thereby signaling both that retrieval is necessary and often which information must be retrieved. The paper argues that this format evaluates recall after retrieval has effectively been initiated by the benchmark, rather than evaluating whether an agent recognizes that historical information is relevant to an action. It therefore proposes an action-based benchmark in which success depends on using information from prior interactions while the current instruction omits the remembered detail.

The benchmark makes two additional methodological commitments. First, it treats accuracy, monetary cost, and latency as jointly necessary evaluation dimensions. Second, it certifies each task by requiring an agent with access to oracle history to succeed and the same agent without the relevant history to fail. These design choices distinguish DolphinBench from established conversational-memory benchmarks such as LoCoMo and LongMemEval (Maharana et al., 2024, Wu et al., 2024), as well as from newer action-oriented evaluations that do not combine action-based grading, complete resource accounting, and per-test solvability verification.

Evaluation problem and benchmark objectives

The central criticism of question-answering evaluations is that they expose the retrieval target through the question. For example, a request asking which messaging platform a team uses explicitly identifies both the existence and semantic type of the desired fact. An action-oriented instruction to post an update in the appropriate team channel does not. The agent must infer that a historical preference or organizational fact is required, retrieve it, and incorporate it into a tool call. DolphinBench consequently evaluates memory as a component of task completion rather than as isolated textual recall.

This reframing has direct implications for benchmark validity. A system can answer a memory question correctly while failing to use the same information in an operational setting, particularly when the current task does not explicitly mention the relevant entity. Conversely, an action may succeed for reasons unrelated to memory if the task is underspecified, the environment supplies the answer, or the remembered fact is unnecessary. DolphinBench addresses the latter problem by requiring every retained task to be both solvable with the relevant history and unsolvable without it.

The paper also rejects accuracy-only comparisons. A memory system could repeatedly process an entire historical record with a high-capability model and obtain strong accuracy while imposing unacceptable cost and latency. Conversely, an inexpensive system may be fast but unreliable. DolphinBench therefore requires submissions to report aggregate ingestion and evaluation cost, together with median per-task latency, and presents the resulting accuracy–cost–latency trade-offs as a Pareto frontier. This is a substantive evaluation claim: memory quality cannot be characterized by task accuracy independently of the computational policy used to obtain it.

Benchmark composition and construction

DolphinBench contains 600 tasks distributed across three simulated knowledge-work personas: Morgan, a startup CEO; Alex, an infrastructure engineer; and Riley, a product manager. Each persona has a multi-year history containing between 3,400 and 5,128 user messages and approximately 500,000 user-message tokens. The histories involve heterogeneous applications and workflows, including email, calendars, Notion, GitHub, CRM systems, and deployment tooling. The benchmark thus tests preferences, decisions, project state, operational procedures, relationships, and other information types that arise in persistent agent interactions.

The benchmark authoring pipeline is hierarchical. Persona specifications define roles, relationships, tools, and changes over time. These specifications are expanded into multi-year narratives, quarterly plans, weekly events, and individual conversations. Simulated application state is updated chronologically before messages are generated, allowing the user messages to refer consistently to tool outputs and prior actions. The released history includes only user-side messages; the evaluated agent generates its own responses during ingestion. This choice is important because a memory system for an interactive agent must process its own participation rather than merely consume a fixed two-sided transcript, as in many QA datasets.

Tests are constructed from facts explicitly identified in the histories. A proposed task specifies the historical facts required, the intended actions, and what would be missing or incorrect without those facts. The final user request provides ordinary task information while omitting the remembered detail. The authors specifically avoid linguistic cues such as “the way I usually do it,” which would announce that a stored preference should be retrieved.

The worked example illustrates the construction principle. Morgan previously states a preference for Blue Bottle coffee. A later instruction asks the agent to order a small latte for pickup while omitting the café. Successful completion requires selecting Blue Bottle through a place_order call. The task therefore tests whether the agent retrieves and operationalizes a latent preference, not whether it can answer a direct question about that preference.

Solvability certification and grading

Each candidate task undergoes an oracle-memory procedure. GPT-5.6-Luna is run twice with the relevant historical messages supplied directly and twice without them. The task, tools, and initial application state remain otherwise unchanged. A test is accepted only if both runs with oracle messages pass all grading checks and both runs without those messages fail. The authors inspect failures without history to distinguish genuine information dependence from tool errors, grading defects, or accidental task solvability.

This procedure provides a stronger validity criterion than manual review alone. A human can verify that a question has an answer while overlooking that the answer is inferable from the current environment or from generic world knowledge. The no-history failure condition tests whether memory is causally necessary for the evaluated behavior. At the same time, the certification is model-relative: solvability is demonstrated using the specified oracle agent and model, not established as an absolute property of the task for every capable agent.

Grading combines deterministic checks and LLM-based judgments. Tool identity, identifiers, dates, numbers, and list membership are checked deterministically. Semantic requirements, such as whether an email conveys a particular decision, are evaluated by GPT-5.6-Sol. A task passes only when every required check succeeds. The design also enforces action-level consistency: if an email must contain a particular recipient and content in the same send operation, satisfying the two conditions across separate messages does not count.

This hybrid grading strategy limits ambiguity in executable state changes while retaining flexibility for natural-language content. It nevertheless leaves semantic judge reliability as a residual source of measurement error. The paper does not report an independent calibration or inter-rater analysis for the LLM judge, so deterministic checks provide the strongest component of the reported validity guarantees.

Empirical results and Pareto trade-offs

The evaluation compares multiple combinations of harness, foundation model, and memory system. The highest reported accuracy is 70.67%, obtained with the Hermes harness, GPT-5.6-Luna, and Mem0. The results demonstrate substantial interaction effects: memory-system rankings change across agent configurations, indicating that DolphinBench measures the combined system rather than treating memory as an independent module.

Harness and model Memory system Accuracy Total cost Median latency
Hermes + GPT-5.6-Luna Built-in memory 65.67% $61.48 44.35 s
Hermes + GPT-5.6-Luna Mem0 70.67% $96.21 37.69 s
Hermes + GPT-5.6-Luna Hindsight 69.50% $84.65 55.31 s
Hermes + GPT-5.6-Luna Honcho 68.50% $142.85 44.66 s
Hermes + MiniMax M3 Mem0 47.83% $148.02 46.95 s
Claude Code + Claude Sonnet 5 Honcho 35.83% $1,565.82 37.92 s

The Mem0 configuration with Hermes and GPT-5.6-Luna improves accuracy by five percentage points over built-in memory and reduces median latency from 44.35 to 37.69 seconds. The improvement is accompanied by a higher total cost, increasing from $61.48 to $96.21. This result directly supports the paper’s claim that higher accuracy need not imply greater latency, although in this comparison it does imply greater expenditure.

The cost–latency relationship is also non-monotonic. Hindsight costs $84.65 and achieves 69.50% accuracy, whereas Honcho costs $142.85 and achieves 68.50%; Hindsight is therefore cheaper and more accurate in this configuration. However, its median latency is 55.31 seconds, compared with 44.66 seconds for Honcho. The most expensive evaluated system is not the most accurate, and the fastest system is not uniformly the least accurate. These results justify reporting all three metrics rather than collapsing performance into a single scalar.

Foundation-model and harness effects are large. With Hermes, replacing GPT-5.6-Luna with MiniMax M3 reduces Mem0 accuracy from 70.67% to 47.83%. Under Claude Code with Claude Sonnet 5, Honcho is the strongest evaluated memory system at 35.83%, while the same system is not the top performer under Hermes. The paper therefore cautions against interpreting a memory-system score independently of the surrounding agent architecture. DolphinBench’s ranking is a property of complete agent configurations, including ingestion behavior, tool-use policy, model capability, and memory implementation.

Limitations and open questions

The benchmark uses synthetic histories and simulated applications. Although the authors construct histories hierarchically, reconcile messages with application records, review source facts, and apply oracle verification, generated conversations may not reproduce the linguistic diversity, multilingual behavior, ambiguity, interruptions, and distributional irregularities of real users. The simulated tools likewise omit production phenomena such as service outages, concurrent modifications by other users, evolving interfaces, and operational permissions.

Coverage is limited to three knowledge-work personas and excludes domains with distinct risk and memory requirements, including medicine, law, and long-term customer support. The tasks contain one to four separately graded actions, so they test retrieval and application of remembered information without fully evaluating long chains of dependent operations. Moreover, the benchmark performs history ingestion before testing rather than interleaving new conversations and tasks. This isolates comparison across systems with a common completed history but does not measure continual memory updates, contradiction handling, or online consolidation under ongoing interaction.

The oracle criterion also depends on the chosen verifier. Requiring two successful and two unsuccessful runs strengthens acceptance, but stochastic model behavior can still produce false exclusions or inclusions. The benchmark establishes that GPT-5.6-Luna behaves differently with and without the selected oracle messages; it does not prove that every accepted task is intrinsically memory-dependent for all architectures. Finally, the paper reports aggregate results for a relatively small set of configurations, leaving open how stable the Pareto frontier is under alternative foundation models, judge models, tool environments, history lengths, and repeated benchmark versions.

Conclusion

DolphinBench defines agent memory evaluation around action completion rather than explicit recall. Its principal methodological contributions are the removal of retrieval cues from task instructions, joint reporting of accuracy, cost, and latency, and per-test verification that oracle history enables success while withheld history causes failure. The results show that memory-system performance depends strongly on the enclosing agent configuration and that accuracy, cost, and latency exhibit nontrivial trade-offs. The benchmark’s conclusions are bounded by synthetic data, simulated tools, limited persona coverage, short workflows, and separated ingestion and testing, but its evaluation framework provides a concrete basis for comparing memory systems as components of deployable agents rather than as isolated question-answering modules.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

Explain it Like I'm 14

1. What is this paper about?

The paper introduces DolphinBench, a new test for measuring how well AI agents use long-term memory.

An AI agent is a computer program that can do things for people, such as send emails, update calendars, order food, or use work apps. To do these tasks well, it may need to remember information from weeks or years earlier.

For example, a person might once tell an AI, “I prefer Blue Bottle coffee.” Months later, they might say, “Please order me a small latte.” A useful AI should remember the coffee-shop preference, even though the new request does not mention it.

DolphinBench tests whether AI agents can use memories like this when completing real tasks—not just when answering questions.

2. What questions are the researchers asking?

The paper focuses on several main questions:

  • Can an AI agent remember important information from a very long history?
  • Can it use that information to complete a task correctly?
  • Is testing an AI by making it perform actions better than simply asking it questions?
  • How much does the memory system cost to use?
  • How long does each task take?
  • Are the tests themselves fair and actually dependent on memory?

The researchers argue that a good memory system should not be judged by accuracy alone. It should also be reasonably fast and affordable.

3. How did the researchers conduct the study?

Creating the simulated people and histories

The researchers created three fictional workers:

  • Morgan, a startup CEO
  • Alex, an infrastructure engineer
  • Riley, a product manager

Each character had a made-up work and personal history lasting several years. The histories included thousands of messages and about 500,000 words’ worth of computer tokens per person. A token is a small piece of text used by AI systems; it can be part of a word or a whole short word.

The simulated workers interacted with tools such as:

  • Email
  • Calendars
  • GitHub
  • Notion
  • Customer and project management systems

Testing actions instead of simple questions

Many older memory tests ask questions such as:

“What messaging platform does the team use?”

This makes the test easier because the question tells the AI that it needs to search its memory for a messaging platform.

DolphinBench uses less obvious instructions, such as:

“Please post an update in the team’s channel.”

The AI must realize that it needs an old memory to know which channel or platform to use. This is more like what happens in real life.

Example test

In one test, Morgan previously said that they usually get coffee from Blue Bottle when leaving early.

Later, Morgan says:

“I’m out early. Please order one small latte for pickup.”

The AI must remember the earlier preference and order from Blue Bottle. The new request never directly names the café.

Checking whether each test is fair

The researchers used a process similar to checking the answers before giving students an exam.

For every test, they ran an AI agent:

  1. With the important old messages provided
  2. Without those messages

A test was accepted only if:

  • The agent succeeded when it had the needed memory.
  • The agent failed when it did not have that memory.

This helped show that the test really depended on memory rather than being accidentally easy or incorrectly designed. The researchers call the supplied messages oracle messages, meaning messages that give the agent perfect access to the relevant information.

Measuring performance

The researchers tested different combinations of AI models, agent programs, and memory systems. They measured three things:

Measurement Meaning
Accuracy The percentage of tasks completed correctly
Cost The total amount of money spent on the AI and memory system
Latency How long a task took to finish

Some requirements were checked automatically. For example, the researchers could check whether the correct tool was used or whether the correct date was entered.

Other requirements were judged by another AI when the meaning of a message needed to be understood. This is called an LLM judge. “LLM” means LLM, the type of AI used in systems such as chatbots.

4. What did the researchers find?

DolphinBench contains 600 tasks, with 200 tasks for each fictional worker.

The best-performing combination completed 70.67% of the tasks correctly. This means the strongest system still failed about three out of every ten tasks.

The results also showed that the best system depends on the AI model and the surrounding agent program. A memory system that worked well with one AI did not always work best with another.

Some important examples include:

  • With the Hermes agent and GPT-5.6-Luna, Mem0 achieved 70.67% accuracy.
  • With the same setup, the built-in memory achieved 65.67% accuracy.
  • Mem0 cost more than built-in memory, but it was actually faster in the test.
  • Another system, Hindsight, cost less than Honcho and had slightly better accuracy, but took longer to finish tasks.
  • The most expensive setup was not the most accurate or the fastest.

These results show that improving one measurement does not automatically improve the others. An AI system may be:

  • More accurate but more expensive
  • Cheaper but slower
  • Faster but less accurate

The paper describes comparing these trade-offs as finding a Pareto frontier. In simple terms, this means identifying the systems that offer the best balance between accuracy, cost, and speed.

5. Why are these findings important?

The paper argues that older memory tests have several weaknesses.

First, many use a question-and-answer format. The question often tells the AI exactly what information to look for. Real users usually give instructions, not carefully designed memory questions.

Second, many tests focus only on whether the answer is correct. This can reward impractical systems. For example, an AI could reread an enormous history every time it receives a request. That might improve accuracy, but it would be too slow and expensive for everyday use.

Third, some existing tests may contain incorrect answers or may be solvable even without memory. DolphinBench tries to avoid this by checking every test with and without the relevant information.

6. Limitations of the research

DolphinBench is useful, but it does not perfectly represent real life.

The researchers point out that:

  • The conversations were created by AI and are not real user conversations.
  • Only three types of workers were included.
  • The apps were simulated rather than connected to real services.
  • The tasks were fairly short and usually required only one to four actions.
  • The AI read all of its history before testing began, rather than learning continuously while performing tasks.

Because of these limitations, an AI that performs well on DolphinBench might still have problems in real workplaces.

7. Overall meaning and possible impact

DolphinBench gives researchers a more realistic way to test AI memory. Instead of asking, “Can the AI repeat an old fact?”, it asks, “Can the AI use an old fact to do something correctly?”

This could help developers build better AI assistants for work, planning, customer service, coding, and other long-term activities. It may also help users choose systems that offer the right balance of accuracy, price, and speed.

The main lesson is that good AI memory is not only about remembering facts. It is about knowing when an old fact matters, finding it, and using it correctly while completing a task.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge Gaps, Limitations, and Open Questions

The paper leaves the following issues unresolved:

  • Generalization to real user behavior: It remains unclear whether performance on LLM-generated conversations transfers to authentic user histories containing interruptions, ambiguity, typos, multilingual content, inconsistent preferences, and implicit communication.
  • Limited persona and domain coverage: Only three knowledge-work personas are evaluated, leaving memory requirements in domains such as healthcare, law, education, finance, customer support, and personal assistance unexplored.
  • Small and potentially non-representative test set: The benchmark contains only 600 tasks, with 200 tasks per persona, so performance may be sensitive to the particular scenarios selected and may not support reliable conclusions across broader task distributions.
  • Dependence on LLM-generated benchmark artifacts: Personas, histories, facts, tasks, and grading checks are largely generated by LLMs. The paper does not quantify generation error rates, authoring-model biases, or correlations between the models used to construct the benchmark and those used to evaluate agents.
  • Model-dependent solvability verification: A test is accepted only when GPT-5.6-Luna succeeds with oracle messages and fails without them. This does not establish that the task is solvable by other capable models, nor that failure without oracle memory necessarily reflects a memory deficit rather than a model-specific reasoning or tool-use failure.
  • Insufficient validation of the “memory necessity” criterion: Passing with oracle messages and failing without them does not isolate memory retrieval from other capabilities, such as planning, instruction following, tool selection, or language understanding. A task may be memory-dependent for the verifier but not for a different agent that can infer the answer through another route.
  • No independent human or expert validation at scale: The paper reports review of plans, facts, tasks, and verification traces but does not specify the number of reviewers, agreement rates, adjudication procedures, or whether independent experts confirmed that tasks genuinely require the designated memories.
  • Uncertainty in LLM-judge grading: Some requirements rely on GPT-5.6-Sol, yet the paper does not report inter-judge agreement, sensitivity to judge prompts, false-positive and false-negative rates, or validation against human judgments.
  • Potential evaluator-model contamination: The use of closely related or publicly released models for history generation, task construction, verification, judging, and evaluation may introduce stylistic or behavioral artifacts that advantage particular systems.
  • Unreported statistical uncertainty: Results are presented as single accuracy, cost, and latency values without confidence intervals, repeated-seed variation, per-persona breakdowns, or significance tests. It is therefore unclear whether small differences between systems are statistically meaningful.
  • Limited analysis of task heterogeneity: The paper does not report performance by memory type, temporal distance, number of relevant messages, fact complexity, update frequency, app, action count, or ambiguity level. This prevents identification of which memory capabilities drive successes and failures.
  • No systematic failure analysis: The results do not distinguish retrieval failures from incorrect memory formation, stale-memory use, reasoning errors, tool-call errors, judge disagreements, or failures caused by the harness.
  • Confounding of memory systems with agent harnesses and models: Memory systems are compared across different harness–model combinations, but there is no controlled ablation isolating the effects of the memory layer from prompt design, tool orchestration, context management, and model capability.
  • Lack of memory-system ablations: The study does not evaluate how performance changes when varying memory write policies, retrieval thresholds, ranking methods, summarization, compression, recency weighting, or the amount of retrieved context.
  • Incomplete characterization of the Pareto frontier: The experiments evaluate a small number of configurations and do not establish whether the reported systems form a complete or stable Pareto frontier. The analysis also does not define how users should trade off accuracy, monetary cost, and latency.
  • Cost and latency results may not transfer across deployments: Reported costs depend on model pricing, hardware, batching, concurrency, implementation details, and accounting assumptions. The paper does not evaluate how the rankings change under different pricing schemes or production workloads.
  • Latency measurement is incomplete for continuous operation: Median task latency omits tail latency, memory-maintenance delays, throughput, storage access time, and concurrent-user effects, all of which are important for deployed agents.
  • Unexplored scaling behavior: Although each history contains roughly 500k user-message tokens, the paper does not measure how accuracy, cost, and latency change with history length, memory size, number of facts, or histories exceeding the model’s context window.
  • No evaluation of continuous memory updates: Testing occurs only after the full history has been ingested. The benchmark therefore does not determine whether systems can incorporate new information, revise memories, or avoid overwriting correct facts during ongoing interaction.
  • Limited treatment of temporal and contradictory information: The paper includes facts that may change, but it does not systematically test conflicting updates, expiration, uncertainty, source reliability, or choosing the most recent valid value.
  • Short and shallow action workflows: Tasks require only one to four separately graded actions. Longer workflows involving dependent decisions, recovery from errors, and memory use across multiple intermediate states remain untested.
  • Restricted environmental realism: Simulated applications do not capture authentication failures, outages, asynchronous updates, permissions, changing interfaces, external-user actions, or irreversible side effects. Performance in the simulated environment may therefore overestimate real-world reliability.
  • No assessment of robustness to distractors and adversarial information: The benchmark does not systematically test retrieval amid large volumes of irrelevant content, misleading statements, prompt injection, malicious updates, duplicated facts, or deliberately conflicting memories.
  • Privacy and safety are unexamined: The paper evaluates whether agents retrieve information but not whether they retrieve sensitive information only when authorized, respect user boundaries, prevent cross-persona leakage, or avoid taking high-impact actions based on uncertain memories.
  • No calibration or abstention evaluation: Agents are not assessed on whether they recognize insufficient or ambiguous memory, ask clarifying questions, or refrain from acting when the remembered information is unreliable.
  • Potential task-construction selection bias: Tasks are selected after being proposed and verified, but the paper does not report how many candidates were rejected or whether verification preferentially retains easy, unambiguous, or model-compatible tasks.
  • Unknown reproducibility over benchmark versions: The paper proposes that the benchmark will grow but does not define versioning, fixed test splits, contamination safeguards, or procedures for comparing results as histories and tasks change.
  • No comparison with non-memory baselines: The study does not report strong baselines such as full-history retrieval, brute-force context expansion, external databases, human performance, or oracle retrieval under matched cost and latency constraints.
  • Unclear relationship between benchmark accuracy and real-world utility: The paper does not establish whether higher task-completion accuracy on DolphinBench predicts user satisfaction, reduced operational errors, productivity, or safety in deployed agent systems.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

  • Production benchmarking for long-term-memory agents (software/AI industry) — Organizations can use DolphinBench’s released dataset, execution traces, deterministic checks, and LLM-based grading to compare memory architectures before deployment. Evaluation should report task accuracy, total ingestion-and-inference cost, and median latency, rather than accuracy alone. Dependency: The benchmark is synthetic, limited to three knowledge-work personas, and uses simulated applications; results should therefore be treated as comparative engineering evidence, not a guarantee of production performance.
  • Selecting memory systems for agent products (enterprise software) — Teams can construct a Pareto comparison among built-in memory, Mem0, Honcho, Hindsight, and other systems. For example, the paper shows that Mem0 with Hermes and GPT-5.6-Luna reached 70.67% accuracy, while built-in memory reached 65.67%, and Mem0 also had lower median latency in that configuration. This supports configuration-specific decisions such as:
    • prioritizing accuracy for autonomous project management;
    • prioritizing latency for interactive assistants;
    • prioritizing cost for high-volume customer-support workloads.
    • Dependency: The reported rankings vary by model and harness, so organizations must benchmark their own model, retrieval strategy, hardware, and workload.
  • Regression testing for agent-memory updates (MLOps/software engineering) — Development teams can run DolphinBench-like action tests whenever they change prompts, model versions, memory schemas, summarization policies, or retrieval algorithms. A release can be blocked if memory accuracy falls, cost exceeds a budget, or latency violates a service-level objective. Dependency: Automated grading must distinguish memory failures from tool failures, and task suites should be expanded with domain-specific tests.
  • Evaluation of coding and infrastructure agents (software engineering/cloud operations) — The infrastructure-engineer persona and simulated GitHub, deployment, incident, and runbook workflows provide a template for testing whether agents remember prior deployment decisions, on-call arrangements, incident context, and operational preferences. Similar tests could be integrated into internal evaluations for coding copilots, DevOps agents, and site-reliability assistants. Dependency: The current tasks are relatively short—one to four graded actions—and do not fully model production outages, concurrent edits, or long dependency chains.
  • Persistent project-management assistants (business software/product management) — Product and project-management tools could use action-based memory tests to verify that an agent correctly applies historical decisions, experiment results, stakeholder preferences, deadlines, and product requirements when updating calendars, tickets, documents, or communications. Dependency: Access-control rules, data freshness, conflicting stakeholder instructions, and the distinction between outdated and current facts must be modeled explicitly.
  • Customer-service agent evaluation (customer support/CRM) — Support organizations can adapt the benchmark’s method to test whether an agent uses prior customer interactions to select the correct refund policy, communication channel, product configuration, or escalation path without being explicitly told which historical fact to retrieve. Dependency: Real customer data introduces privacy, consent, retention, and regulatory requirements. Synthetic or carefully anonymized histories may be necessary.
  • Cost-aware procurement and capacity planning (AI platform operations/finance) — Because DolphinBench requires cost and latency reporting, platform teams can estimate the operational implications of memory systems before deployment, including ingestion cost, repeated retrieval cost, model-selection effects, and latency under expected traffic. This can inform routing policies such as using a cheaper model for routine memory lookups and a stronger model for ambiguous actions. Dependency: The paper’s cost figures are tied to particular models and experimental settings; production pricing, caching, batching, and hardware utilization may substantially change the frontier.
  • Improved benchmark design and audit workflows (academia and AI evaluation) — Researchers can reuse the paper’s construction pipeline: hierarchical history planning, source-message fact tracking, deterministic and semantic graders, and oracle verification. The requirement that a task pass with relevant history and fail without it can identify tests that are either unsolvable or solvable without memory. Dependency: LLM-generated histories and judges can still contain subtle inconsistencies or biases, so human review and independent audits remain important.
  • Policy and procurement standards for autonomous agents (public-sector governance) — Regulators and institutional buyers can require vendors to disclose memory-agent performance across three dimensions: accuracy, cost, and latency. Procurement evaluations can also require evidence that tests are solvable with the relevant information and fail when that information is withheld. Dependency: These measures do not by themselves assess privacy, fairness, cybersecurity, explainability, or safety of real-world actions; those criteria must be added.
  • Personal digital assistants with preference-aware actions (daily life/consumer technology) — The paper’s coffee-order example illustrates a practical workflow: an assistant can use a previously stated preference to complete an underspecified request, such as ordering a familiar item, scheduling an appointment, or selecting a preferred retailer. Dependency: The assistant must distinguish stable preferences from temporary choices, request confirmation for costly or irreversible actions, and provide users with controls to inspect, correct, or delete stored memories.

Long-Term Applications

  • Continuous, multi-domain memory for autonomous knowledge workers (enterprise AI) — A mature system could maintain memory across email, calendars, repositories, issue trackers, CRM records, and documents while updating beliefs as facts change. It could support an agent that acts as a persistent colleague rather than a session-based chatbot. DolphinBench’s planned extensions—longer histories, more personas, longer tasks, and interleaved ingestion and testing—provide a direct evaluation path. Dependency: This requires reliable temporal reasoning, conflict resolution, provenance tracking, permissions, deletion mechanisms, and resistance to stale or malicious memories.
  • Memory-enabled healthcare assistants (healthcare) — A specialized benchmark could test whether clinical or administrative agents correctly use longitudinal information to schedule care, reconcile patient preferences, prepare summaries, or route tasks. The action-based format is particularly relevant because a clinical instruction may not explicitly state which prior record must be retrieved. Dependency: Deployment would require validated clinical safety, human oversight, privacy protection, auditability, and compliance with applicable health-data regulations. The current paper provides no evidence for medical use.
  • Education and individualized tutoring systems (education) — Long-term-memory agents could adapt assignments, reminders, explanations, and learning plans based on a student’s prior misconceptions, accommodations, interests, and progress. A DolphinBench-style evaluation could test whether an agent takes the correct educational action rather than merely answering a recall question. Dependency: Student data governance, age-appropriate safeguards, fairness across learners, and robust handling of changed abilities or preferences are essential.
  • Robotics and embodied personal assistants (robotics/home automation) — The benchmark’s action-oriented principle could be extended to robots that remember household layouts, safety constraints, user preferences, maintenance histories, or recurring routines and then perform multi-step physical tasks. Dependency: Simulated app actions are much safer and more reproducible than physical actions. Real deployment requires perception, manipulation reliability, fail-safe behavior, uncertainty estimation, and confirmation before consequential actions.
  • Long-horizon software engineering agents (software development) — Future systems could remember architectural decisions, code ownership, unresolved technical debt, incident history, test conventions, and prior review feedback across months of repository activity. They could then implement changes, open pull requests, update documentation, and coordinate releases. Dependency: Evaluation must include longer workflows, changing repositories, concurrent developers, security-sensitive code, and rollback behavior; the current benchmark does not test these conditions.
  • Financial and operations agents with persistent organizational context (finance/enterprise operations) — Agents could use historical budgets, approval rules, vendor preferences, contract terms, and prior transactions to prepare purchases, reconcile records, or route approvals. Pareto analysis would help balance decision quality against transaction latency and inference cost. Dependency: Financial actions require strict authorization, fraud controls, explainable provenance, segregation of duties, and deterministic verification. Memory errors could create material losses.
  • Adaptive safety and governance monitoring (AI safety/policy) — Action-based memory benchmarks could become part of certification frameworks for agents that operate across sessions. Evaluators could test whether agents remember safety constraints, respect changed instructions, avoid unauthorized disclosure, and abstain when historical information is ambiguous or contradictory. Dependency: DolphinBench primarily measures successful task completion; safety-specific failure taxonomies, adversarial histories, privacy leakage tests, and red-team scenarios must be added.
  • Standardized benchmarks for live, continuously evolving agents (research infrastructure) — A future benchmark could interleave history ingestion and task execution, include real-time updates, simulate outages and concurrent users, and require longer action chains. This would address the paper’s limitations around separate ingestion/testing, simulated applications, and short tasks. Dependency: Reproducibility becomes harder when environments are dynamic. Standardized replay systems, deterministic state snapshots, trace formats, and versioned tool APIs would be needed.
  • Personal memory infrastructure with user-controlled provenance (consumer software/privacy technology) — The methods could support products that expose each remembered fact’s source message, timestamp, confidence, and update history. Users might approve, correct, expire, or delete memories, while agents use only authorized information for future actions. Dependency: Such systems require robust identity and access management, secure storage, transparent retention policies, and mechanisms to prevent sensitive information from being inferred or reused outside its intended context.
  • Cross-agent and organizational memory sharing (multi-agent systems) — Multiple specialized agents could share verified organizational facts—for example, a support agent passing a customer preference to a scheduling agent, or a development agent passing deployment context to an incident-response agent. DolphinBench’s tool- and action-based evaluation offers a foundation for testing whether shared memory improves coordinated task completion. Dependency: Shared memory creates risks of contamination, conflicting updates, privilege escalation, and error propagation. Provenance, namespaces, confidence scores, and explicit authorization would be required.
View Paper Prompt  View All Prompts 

Glossary

  • Abstention: The deliberate decision by an AI system not to provide an answer or take an action when it lacks sufficient confidence or information. “later expanding into categories like temporal reasoning, knowledge updates, and abstention”
  • Action-based evaluation: An assessment method that measures whether an agent completes an action rather than merely producing a textual answer. “The test format must require action rather than pure text responses in a question-answer format.”
  • Agent harness: The software framework that coordinates an AI model, its tools, memory system, and task execution. “We evaluate agents with different combinations of harnesses, LLMs, and memory systems.”
  • Agentic subtask: A component of a larger task that requires an autonomous agent to perform an action or make a decision. “MemoryArena evaluates interdependent agentic subtasks”
  • Context window: The maximum amount of text or other input that a LLM can process in a single interaction. “Such agents are increasingly required to keep track of information across project histories and past sessions that outgrow their context windows”
  • Deterministic check: An evaluation procedure whose result is determined by explicit, reproducible rules rather than subjective interpretation. “We use deterministic checks when a requirement can be tested directly”
  • Domain coverage: The range of professional or application domains represented by a benchmark or dataset. “Three personas cover only a small part of knowledge work.”
  • Frontier model: A highly capable, state-of-the-art AI model near the leading edge of current performance. “by re-reading the entire history at each query with frontier models”
  • Ground-truth error: An error in the reference answer or expected result used to evaluate a system. “i.e. no ground truth errors, no hidden assumptions, and no tests that pass regardless of memory”
  • Hierarchical generation: A data-generation method that develops content through multiple levels of increasing detail. “DolphinBench is built in hierarchical stages”
  • Ingestion: The process of processing and incorporating source data into an agent’s memory system. “Every submission must report accuracy, total cost across ingestion and testing, and median task latency.”
  • Interleaved ingestion and testing: An evaluation arrangement in which new information is processed and tasks are evaluated throughout the same ongoing sequence. “Interleaving ingestion and testing would better represent that continuous use.”
  • Knowledge-work persona: A simulated professional identity defined by a role, activities, relationships, and tools. “DolphinBench defines three knowledge-work personas”
  • Latency: The elapsed time required for a system to complete an operation or task. “We therefore require every result to report total cost and median task latency alongside accuracy.”
  • LLM judge: A LLM used to assess whether another model’s output satisfies an evaluation criterion. “We use an LLM judge \citep{llmjudge} (GPT-5.6-Sol) when checking a requirement requires interpreting meaning”
  • Long-term memory: An agent’s capability to retain and retrieve information across extended periods, sessions, or interactions. “Agents today often take real world actions which depend on long term memory and context recall over time.”
  • Memory benchmark: A standardized dataset and evaluation procedure for measuring an AI system’s ability to store, retrieve, and use information. “It is important to evaluate this capability with memory benchmarks.”
  • Memory construction: The process by which a memory system creates stored representations from an agent’s input history. “MemoryAgentBench reports memory construction and query execution times but not monetary cost.”
  • Memory-dependent task: A task whose correct completion requires information obtained from an agent’s prior history. “Evaluating memory for agents requires removing the question and using memory-dependent tasks as the accuracy signal.”
  • Median task latency: The middle value of the measured completion times for a collection of tasks, reducing the influence of unusually fast or slow cases. “We calculate accuracy as the fraction of tasks passed and take the median of the individual test latencies across all personas.”
  • Oracle memory: An idealized memory condition in which the agent is directly provided with the relevant information needed to complete a task. “We therefore verify every DolphinBench test using an agent with oracle memory”
  • Pareto frontier: The set of configurations for which no single performance metric can be improved without worsening at least one other metric. “Evaluating each configuration on all three metrics lets DolphinBench map the Pareto frontier”
  • Persona: A defined simulated identity used to represent a particular type of user or professional role. “The benchmark covers three personas: Morgan, a startup CEO; Alex, an infrastructure engineer; and Riley, a product manager.”
  • Production deployment: The use of a software or AI system in a real operational environment by users or organizations. “a production deployment could not sustain”
  • Question answering (QA): A task format in which a system responds to an explicitly stated question, usually using information from provided data. “The dominant evaluation format is question answering”
  • Retrieval signal: Information in a task prompt that indicates which stored fact the system should search for or use. “The format carries the retrieval signal discussed in the introduction”
  • Solvability certification: Verification that an evaluation task can be completed with the required information and cannot be completed without it. “The final evaluation results must include cost and latency alongside accuracy.”
  • Stateful agent: An agent that maintains information about prior interactions or changing conditions across multiple tasks. “STATE-Bench tests stateful agent actions”
  • Synthetic data: Artificially generated data created by computational procedures rather than collected directly from real-world users or environments. “We generate conversation histories rather than collect them from real users.”
  • Task completion: Successful execution of all actions or requirements specified by an evaluation task. “DolphinBench is a benchmark built to evaluate agents on their memory which directly tests task success.”
  • Temporal reasoning: The ability to interpret and reason about events, facts, or relationships according to their timing and sequence. “later expanding into categories like temporal reasoning, knowledge updates, and abstention”
  • Tool call: A structured request from an AI agent to invoke an external function, application, or service. “We grade task completion by checking the agent's tool calls against the actions and results required by the task.”
  • Tool output: The result returned by an external function, application, or service invoked by an agent. “The model writing the messages receives these records and tool outputs”
  • Trace: A detailed record of the operations, inputs, outputs, and decisions produced during a system run. “We release the configuration, cost calculations, and full trace of every run”
  • Verification pipeline: A sequence of automated or human checks used to establish that generated evaluation items satisfy specified conditions. “We verify both automatically before a test ships, as described in the Test Construction section.”
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Tweets

Sign up for free to view the 7 tweets with 146 likes about this paper.

Sign Up for Free