DolphinBench

• ‘DolphinBench is a benchmark for evaluating long-term memory in software agents, assessing their ability to identify, retrieve, and use historical information in simulated knowledge-work environments through task completion rather than question-answering. Examples include Morgan as a startup CEO for investor preferences, Alex as an infrastructure engineer for deployments, and Riley as a product manager for experiments’
• The benchmark evaluates agents based on three metrics: task accuracy, total monetary cost, and median latency, and considers them as a multi-objective systems problem.
• A task is marked as passed by DolphinBench only if it has satisfied all available checks. Parato frontier is used to evaluate agents against all three metrics to assure that a trade-off exists.

DolphinBench is the name of a benchmark for evaluating long-term memory in tool-using software agents through task completion rather than conventional question answering. Introduced in “DolphinBench: Mapping the Pareto Frontier of Agent Memory” (Rathi et al., 21 Sep 2026), it evaluates whether an agent can identify when historical information is relevant, retrieve that information without an explicit retrieval cue, and use it correctly in an external action. Its evaluation jointly reports task accuracy, total monetary cost, and median latency, thereby treating agent memory as a multi-objective systems problem rather than an isolated retrieval task.

1. Scope and conceptual foundations

DolphinBench targets long-term agent memory in simulated knowledge-work environments. Its central unit of evaluation is not a question-answer pair but a task requiring one or more tool-mediated actions whose correctness depends on information embedded in a prior interaction history.

This design addresses a limitation of conventional conversational-memory benchmarks: a direct question often signals that a fact exists and identifies the category of information to retrieve. DolphinBench instead evaluates whether an agent can “know to remember.” For example, an earlier message may state a user’s preferred café, while a later request asks the agent to order a drink without naming the café. Correct completion requires retrieving and applying the historical preference in a place_order tool call.

The benchmark is designed around three requirements:

1. Action-based evaluation: the agent must perform work through tools rather than merely state a recalled fact.
2. Joint efficiency reporting: accuracy must be reported alongside total cost and latency.
3. Memory-dependence certification: each task must succeed when relevant history is supplied and fail when that history is withheld.

DolphinBench therefore evaluates the interaction among a base LLM, an agent harness, a memory system, a simulated application environment, and a grading procedure. It does not claim to measure a memory module independently of the rest of the agent stack.

2. Personas, histories, and simulated environments

DolphinBench contains three simulated knowledge-work personas, each associated with approximately 500,000 user-message tokens and 200 tasks. The complete benchmark contains 600 tasks.

Persona	Knowledge-work setting	Representative information
Morgan	Startup CEO	Investors, hiring, board communication, strategy, relationships, and preferences
Alex	Infrastructure engineer	Deployments, incidents, on-call rotations, runbooks, and infrastructure operations
Riley	Product manager	Experiments, conversion funnels, retention, product decisions, and project context

Each history represents several years of interaction and contains approximately 3,400–5,128 user messages. Token counts are measured using the o200k_base tokenizer from tiktoken over user-message content only.

The released histories contain only the user side of conversations. During ingestion, the evaluated agent generates its own responses. This differs from fixed two-sided conversational transcripts and is intended to represent an agent that actively participates in an ongoing interaction.

The benchmark uses simulated applications and tools, including Notion, Gmail, GitHub, Google Calendar, CRM-like systems, deployment tools, infrastructure tools, and other task-specific applications. These applications maintain records and support selected operations. Their state supplies ordinary current inputs needed to complete a task without directly exposing the historical fact on which the task depends.

The history-generation process is hierarchical. Persona specifications define roles, relationships, tools, applications, voice, and changing circumstances. Long-range professional and personal events are expanded into quarterly and weekly plans. Simulated application state is updated chronologically before user messages are generated, and the resulting records and tool outputs are supplied to the message-generation model. Each released history ends at the first complete session that brings it to at least 500,000 user-message tokens.

Fact tracking records which messages mention, introduce, or change particular facts. Before a fact is used to construct an evaluation task, its description is compared against the source messages and unsupported claims are corrected.

3. Task construction and memory dependence

Each persona contributes 200 tasks. Tasks are issued after the corresponding history has been ingested and require one to four separately graded actions, although agents may make additional exploratory or corrective tool calls.

A task is constructed around one or more specific historical facts that affect a necessary aspect of the final work. Examples include selecting a person, organization, channel, or service based on an earlier preference; using a historical identifier, number, date, or configuration; applying a prior decision to a new communication; following an earlier operational procedure; or acting consistently with a product decision, experiment, or metric definition.

Task wording is designed to avoid explicit retrieval cues such as “the way I usually do it” or “as I mentioned before.” Current application records are included when needed, but are selected so they do not reveal the historical answer. The task-construction process specifies the required facts, intended actions, and information that would be missing without history. A subsequent construction stage generates the user request, grading checks, and references to supporting source messages.

The benchmark’s central example is the coffee-order task. A prior message establishes a preferred café, while a later request specifies the drink, size, and pickup arrangement but omits the café. The agent must infer that the prior preference is relevant and provide the correct café in the ordering action.

Oracle-based verification

Every released task undergoes verification using the original history messages needed for completion, called oracle messages. Each candidate task is run four times using GPT-5.6-Luna:

• two runs with the oracle messages;
• two runs without the oracle messages.

A task is accepted only if both runs with the oracle history pass and both runs without it fail:

$\text{with history: pass, pass}$

$\text{without history: fail, fail}.$

Failures without history must be attributable to missing information rather than a tool error, application defect, ambiguous instruction, or another task flaw. Candidates that do not satisfy the condition are revised and rechecked.

This procedure establishes empirical solvability and memory necessity under the specified model, tools, and execution setup. It is not a formal proof that no possible agent could solve a task without historical information.

4. Grading and evaluation metrics

A task passes only when all required grading checks pass. Checks are associated with individual required actions and may be deterministic or semantic.

Deterministic checks verify directly observable requirements, including whether a required tool was called, whether an identifier or number is correct, whether a date matches, or whether a list contains required values. LLM-judge checks are used when semantic interpretation is needed, such as determining whether an email conveys a particular decision. The judge used in the reported evaluation is GPT-5.6-Sol.

Checks are conjunctive. If a task requires an email with a specific recipient and content, both must occur in the same send call. Satisfying the recipient and content requirements across separate calls does not satisfy the task. If two distinct emails are required, one email cannot satisfy both.

For $N$ tasks, benchmark accuracy is the fraction of tasks that pass all grading checks:

$\mathrm{Accuracy}=\frac{1}{N}\sum_{i=1}^{N}P_i,$

where $P_i\in\{0,1\}$ is the pass indicator for task $i$.

DolphinBench requires three principal metrics:

• Accuracy: percentage of tasks completed correctly according to all grading checks.
• Total cost: monetary cost of history ingestion, memory construction or processing, memory queries, agent reasoning, and task execution.
• Median latency: median end-to-end latency across individual tasks, including tool use.

The cost decomposition is reported as:

$$C_{\mathrm{total}}=C_{\mathrm{agent}}+C_{\mathrm{memory}}.$$

Median latency is:

$$L_{\mathrm{median}}=\operatorname{median}(L_1,L_2,\ldots,L_N),$$

where $L_i$ is the end-to-end latency of task $i$.

These metrics are deliberately not collapsed into a single score. The benchmark instead represents each configuration in the objective space:

$\text{without history: fail, fail}.$0

with higher accuracy and lower cost and latency preferred.

5. Pareto-frontier analysis

A configuration is considered preferable only when its performance is assessed across all three reported objectives. Configuration $\text{without history: fail, fail}.$1 dominates configuration $\text{without history: fail, fail}.$2 when it is at least as accurate, no more expensive, and no slower, with at least one strict improvement.

The Pareto frontier consists of configurations that are not dominated by another evaluated configuration. This formulation recognizes that the most accurate memory system may not be the cheapest or fastest, and that the optimal choice depends on deployment priorities.

The reported results include the following configurations:

Harness and model	Memory system	Accuracy	Total cost	Median latency
Hermes + GPT-5.6-Luna	Built-in memory	65.67%	$61.48	44.35 s
Hermes + GPT-5.6-Luna	Mem0	70.67%	$96.21	37.69 s
Hermes + GPT-5.6-Luna	Honcho	68.50%	$142.85	44.66 s
Hermes + GPT-5.6-Luna	Hindsight	69.50%	$84.65	55.31 s
Hermes + GPT-5.6-Luna	Supermemory	59.17%	$345.64	52.68 s
Hermes + MiniMax M3	Built-in memory	26.50%	$107.03	51.40 s
Hermes + MiniMax M3	Mem0	47.83%	$148.02	46.95 s
Claude Code + Claude Sonnet 5	Built-in memory	26.33%	$1,132.75	32.19 s
Claude Code + Claude Sonnet 5	Mem0	32.33%	$1,830.57	42.30 s
Claude Code + Claude Sonnet 5	Honcho	35.83%	$1,565.82	37.92 s

The highest reported accuracy is 70.67%, achieved by Hermes with GPT-5.6-Luna and Mem0. Its total cost is $96.21 and its median latency is 37.69 seconds.

For Hermes with GPT-5.6-Luna, Mem0 improves accuracy from 65.67% to 70.67% relative to built-in memory and reduces median latency from 44.35 to 37.69 seconds, while increasing total cost from $\text{without history: fail, fail}.$396.21. Hindsight achieves 69.50$\text{without history: fail, fail}.$$4142.85 and 44.66 seconds. Supermemory records 59.17$$345.64 and 52.68 seconds.

The results also show strong model and harness dependence. With Hermes, GPT-5.6-Luna and Mem0 achieve 70.67%, whereas MiniMax M3 and Mem0 achieve 47.83%. Consequently, memory-system rankings cannot be interpreted independently of the model, harness, prompting strategy, and tool-use policy.

Under the reported values, the clearest apparent nondominated configurations are Hermes with GPT-5.6-Luna using built-in memory and Hermes with GPT-5.6-Luna using Mem0. The former is cheaper; the latter is more accurate and faster. Neither dominates the other because cost and performance move in different directions.

6. Reproducibility, limitations, and significance

The benchmark releases the dataset, evaluation code, configuration, cost calculations, execution traces, tool calls, tool arguments, and per-check grading verdicts through https://dolphinbench.ai. Simulated tools and applications provide a controlled environment intended to support reproducibility.

Several limitations affect interpretation. The histories are generated rather than collected from real users and may not fully represent natural writing variation, multilingual interaction, realistic ambiguity, interruptions, or the distribution of actual user behavior. Only three personas are included: startup leadership, infrastructure engineering, and product management. The applications are simulated and do not model all production phenomena, such as service outages, authentication failures, external record changes, undocumented API behavior, or evolving interfaces.

Tasks contain one to four separately graded actions and generally do not require long chains of dependent work. The benchmark also separates full-history ingestion from later evaluation, rather than testing continuous operation in which new conversations arrive while tasks are executed and memory is updated concurrently.

Oracle verification is setup-dependent. It uses GPT-5.6-Luna, a particular oracle-message selection, a specified tool environment, and fixed grading checks. Its pass/fail results therefore provide empirical evidence of memory dependence under the benchmark protocol rather than an absolute guarantee. Likewise, semantic grading depends on GPT-5.6-Sol and may introduce evaluator-model bias, inconsistency, or sensitivity to phrasing.

The reported results demonstrate that long-term memory quality cannot be characterized adequately by retrieval accuracy alone. An agent must recognize when past information is relevant, retrieve the appropriate information without an explicit cue, apply it in the correct tool-mediated action, and do so within acceptable cost and latency limits. DolphinBench’s principal contribution is therefore a systems-level evaluation framework for the Pareto frontier of agent memory rather than a standalone retrieval dataset or a single memory-recall score.