MEMTRACK: Benchmark for Enterprise Memory

• MEMTRACK is a benchmark for evaluating long-term memory and state tracking in dynamic enterprise environments using platforms like Slack, Linear, and Git.
• It simulates realistic workflows by modeling asynchronous, fragmented, and conflicting events to test acquisition, selection, and coherent state maintenance.
• Key metrics—Correctness, Efficiency, and Redundancy—highlight that even advanced models achieve only about 60% correctness, underscoring persistent challenges.

MEMTRACK is a benchmark for evaluating long-term memory and state tracking in dynamic, enterprise-like agent environments. It was introduced to move memory evaluation beyond conversation-only settings by modeling organizational workflows that integrate asynchronous events across multiple communication and productivity platforms such as Slack, Linear, and Git, together with potential codebase and file-system comprehension and exploration. Its central premise is that realistic agent deployments must handle fragmented, interleaved, noisy, and conflicting information over long horizons, so evaluation should test acquisition, selection, conflict resolution, and coherent state maintenance rather than simple conversational question answering alone (Deshpande et al., 1 Oct 2025).

1. Conceptual scope and motivation

MEMTRACK is motivated by the claim that current memory benchmarks mostly test single-thread conversational memory, whereas deployed agents increasingly need to reason across multiple asynchronous platforms, tickets, chat, code, filesystem artifacts, notifications, and follow-up interactions over extended time spans. In this setting, cross-platform information is fragmented, events are asynchronous and interleaved, later messages may conflict with earlier ones, and many tasks require repository reading or file inspection. MEMTRACK therefore targets the full memory pipeline: what the agent acquires, what it keeps, what it later selects, how it resolves contradictions, and whether it maintains coherent state across many tool calls and platform switches (Deshpande et al., 1 Oct 2025).

This design distinguishes memory evaluation from retrieval-only evaluation. MEMTRACK does not treat memory as mere recovery of a previously stated fact. Instead, it tests whether an agent can preserve operationally relevant state while the underlying evidence is scattered across tools and time. A common misconception is that such benchmarks are mainly about final-answer accuracy. MEMTRACK explicitly rejects that reduction: it evaluates not only whether an answer is correct, but also whether memory tools are used efficiently and without redundant re-access of information (Deshpande et al., 1 Oct 2025).

The benchmark is grounded in enterprise software workflows. The paper states that scenarios are curated through both manual expert driven design and scalable agent based synthesis, with the aim of generating ecologically valid instances grounded in real world software development processes. This suggests that MEMTRACK is intended less as a synthetic long-context stress test than as an evaluation framework for organizational state tracking under realistic tool-mediated work conditions (Deshpande et al., 1 Oct 2025).

2. Multi-platform environment and task structure

MEMTRACK operates in a sandboxed, tool-enabled environment containing at least Slack, Linear, Git, and a dockerized filesystem or codebase. The environment includes Slack messaging and channel or DM history, Linear ticket creation, retrieval, updates, and milestones, Git repository access, local filesystem exploration via shell-style commands, and, in some experimental conditions, memory backends. A defining property of the benchmark is that no single platform contains the whole answer; relevant evidence is deliberately distributed across tools and time (Deshpande et al., 1 Oct 2025).

Each instance is a chronologically ordered but platform-interleaved event timeline, denoted in the paper as $T=(E_1,\ldots,E_n)$. Every event carries metadata such as timestamp, platform type, and platform-specific content. The benchmark quantifies this interleaving through Chronological Heterophily, defined as the probability that one event is followed by an event from a different platform; the reported average is 0.364. The paper presents this as evidence of healthy cross-platform mixing rather than neat platform-specific blocks, which would make the task substantially easier (Deshpande et al., 1 Oct 2025).

The timelines contain three deliberately difficult event classes. First, they include noisy events, such as irrelevant distractors, casual mentions, and similar-looking but unrelated discussions. Second, they include conflicting events, in which later messages revise earlier assumptions, issues are described differently across threads, or contradictory hints force explicit resolution. Third, they include cross-referencing events, including Slack messages referring to Linear tickets, tickets referring back to Slack threads, and code comments or repository exploration resolving ambiguities. The paper also notes that cross-platform references are quantified with an LLM judge and a threshold-based score (Deshpande et al., 1 Oct 2025).

MEMTRACK further extends beyond conversational settings by including codebase and filesystem exploration tasks. Instances can require repository cloning, directory inspection, file search, counting imports or function definitions, identifying filenames or line numbers, and correlating code artifacts with Slack or Linear context. This broadens the notion of memory from dialogue retention to persistent state tracking over tool interactions and partial code comprehension (Deshpande et al., 1 Oct 2025).

Questions are asked strictly sequentially, and the agent does not know how many will follow. Each instance consists of a timeline $T$ and a sequence of question-answer pairs $\langle Q_i, A_i^* \rangle$. This sequential protocol is central to the benchmark’s long-horizon character: later questions may depend on state accumulated during earlier questions, while the hidden number of future queries prevents simple pre-planning (Deshpande et al., 1 Oct 2025).

3. Memory capabilities and dataset construction

The benchmark is organized around several memory capabilities. The paper names acquisition, selection, conflict resolution, long-horizon state tracking, cross-platform dependency tracking, and memory-tool use quality. Acquisition concerns what information from the stream is worth storing or recalling later, such as ticket IDs, filenames, root causes, decisions made in chat, or code locations already inspected. Selection concerns choosing the relevant memories among many distractions. Conflict resolution concerns reconciling early hypotheses with final resolutions, vague claims with confirmed facts, and abandoned approaches with adopted ones. Long-horizon state tracking concerns maintaining cumulative state across multiple questions and many tool calls. Cross-platform dependency tracking concerns connecting Slack, Linear, Git, and filesystem evidence into a coherent answer. Memory-tool use quality concerns whether systems such as Mem0 or Zep are actually used effectively rather than merely causing redundant re-reading (Deshpande et al., 1 Oct 2025).

The dataset contains 47 instances. Reported statistics indicate substantial variation in length and complexity: per-instance event count has mean 39.9 and max 115; event tokens have mean 4.01K and max 11.1K; platform entropy has mean 0.668 and max 0.989; cross-platform references have mean 2.1 and max 19; timeline span in hours has mean 878 and max 3049; and question count per instance has mean 3.2 and max 5. These figures support the benchmark’s intended emphasis on long, mixed, and cross-platform organizational context (Deshpande et al., 1 Oct 2025).

MEMTRACK uses three complementary construction methods. The first is manual top-down expert design: four in-house experts with product or engineering experience create realistic scenarios in their own words, LLMs generate timeline events from those descriptions, experts manually inspect and correct the generated timelines, and questions and answers are also manually created from the timeline. The second is bottom-up agent-based synthesis: the process starts from real closed issues in popular open-source GitHub repositories, an exploratory agent uses web search and bash to find issues with merged PRs and deterministic resolutions, smaller PRs are preferred for verifiability, human annotators filter ambiguous cases, another agent generates Slack and Linear conversations around the issue, counterfactual context and distractors are added, and the task is created from the issue and the actual merged PR solution. The third is a hybrid approach combining expert ideation and LLM generation, in which annotators seed the task with high-level context, goals, and codebase, the LLM elaborates the plan, annotators progressively increase complexity and refine the event history, and the annotator solves the task manually to ensure completeness (Deshpande et al., 1 Oct 2025).

This tripartite construction scheme is methodologically significant because it balances realism and scale. The paper’s explicit rationale is to preserve ecological validity while enabling broader dataset creation. A plausible implication is that MEMTRACK aims to occupy a middle ground between narrowly hand-authored enterprise simulations and purely synthetic long-context benchmarks (Deshpande et al., 1 Oct 2025).

4. Evaluation methodology and metrics

MEMTRACK introduces three main metrics: Correctness, Efficiency, and Redundancy. Correctness is computed per question using a GPT-4o LLM judge that evaluates the triple consisting of the question, the agent answer, and the expected answer. The prompt explicitly allows exact match, semantic equivalence, partial correctness, and reasonable interpretation of ambiguity. Instance-level Correctness is the average judge score over all questions. This metric is therefore broader than exact-match QA, since it permits graded assessment under ambiguity and multi-part answers (Deshpande et al., 1 Oct 2025).

Efficiency measures how many tool calls the agent made. The paper defines it using the total number of tool calls $TC$, with a minimum target of $TC_{min}=10$. The stated intent is to reward agents that avoid excessive tool use, but only after a reasonable minimum threshold. This prevents systems from appearing efficient merely by under-exploring the environment (Deshpande et al., 1 Oct 2025).

Redundancy measures how often the agent repeats or subsumes earlier tool calls. A later call is redundant if it is equivalent to, or subsumed by, an earlier call. The paper gives examples such as issuing get_ticket(id="abc") and later repeating the same query, or retrieving a specific ticket and later issuing a broader query that re-accesses already obtained information. Redundancy is normalized by total tool calls. The paper emphasizes that high efficiency or low redundancy is not automatically good if correctness drops, which is an important interpretive constraint on the benchmark (Deshpande et al., 1 Oct 2025).

The experimental setup compares frontier models and memory backends. The reported models are gpt-5 and gemini-2.5-pro. The memory variants are NoMem, Mem0, and Zep. Mem0 uses gpt-4o-mini embeddings and ChromaDB vector storage, while Zep is described as a temporal knowledge graph–style memory backend. The reported setup uses each backend’s default settings, temperature equal to 1, results averaged across 5 runs, and additional analysis of tool-call success rates. Questions are asked sequentially and the number of future questions is hidden (Deshpande et al., 1 Oct 2025).

A common misunderstanding would be to treat MEMTRACK as a benchmark of retrieval plug-ins rather than of agents. The metric design argues against that view. Correctness is evaluated over multi-step, tool-mediated reasoning; Efficiency and Redundancy evaluate the operational behavior of the entire agent; and the hidden-horizon sequential protocol makes state maintenance itself part of the task (Deshpande et al., 1 Oct 2025).

5. Empirical findings

The central empirical result is that MEMTRACK remains difficult for current frontier systems. The abstract states that the best performing GPT-5 model only achieves a 60% Correctness score, and the paper uses this as evidence that current frontier models still struggle with long-horizon memory, cross-platform dependencies, contradictions, and follow-up reasoning. In the main result table, the reported Correctness scores are 0.610 for gpt-5 + Mem0, 0.601 for gpt-5 + NoMem, 0.601 for gpt-5 + Zep, 0.144 for gemini-2.5-pro + NoMem, 0.118 for gemini-2.5-pro + Mem0, and 0.140 for gemini-2.5-pro + Zep. Efficiency is reported around 0.65–0.67 across methods, and Redundancy around 0.20–0.24 (Deshpande et al., 1 Oct 2025).

The comparative pattern is as important as the absolute scores. GPT-5 is reported to perform much better than gemini-2.5-pro, but even the strongest setup is far from solved. The paper interprets roughly 0.60 Correctness as a serious failure mode in realistic enterprise workflows: the model can often recover some facts, but still misses a large fraction of multi-hop, cross-platform, or follow-up reasoning, and memory tools do not rescue it meaningfully (Deshpande et al., 1 Oct 2025).

A notable finding is that memory backends provide little to no gain. Mem0 and Zep give limited improvement and in some cases slightly worsen performance or increase redundancy. The paper concludes that current memory components are not being used effectively by the agents. This is one of MEMTRACK’s most consequential claims, because it implies that benchmark difficulty is not exhausted by adding an external memory layer; the hard part is effective acquisition, selection, and contradiction management across platforms and time (Deshpande et al., 1 Oct 2025).

The paper also reports degradation on follow-up questions. It gives examples in which GPT-5 NoMem drops from 0.601 overall to 0.571 on follow-up, GPT-5 Mem0 from 0.588 overall to 0.553 follow-up, and GPT-5 Zep from 0.604 overall to 0.585 follow-up. These figures indicate that even when initial answers are strong, retention degrades when later questions depend on accumulated context. Because these follow-up values are reported separately from the main table, they are best read as additional evidence of follow-up fragility rather than as a second consolidated leaderboard (Deshpande et al., 1 Oct 2025).

Qualitative analysis reinforces the quantitative picture. Reported recurrent behaviors include general-to-specific redundancy, repeated access after an interlude, and progressively widening exploration. One illustrative GPT-5 run involved 42 tool calls, 7 platform switches, filesystem exploration, Linear queries, Slack channel queries, and eventual correct answers on later follow-ups. The paper presents this as evidence of substantial capability, but also of heavy tool dependence and repeated context fetching (Deshpande et al., 1 Oct 2025).

6. Interpretation, limitations, and future directions

MEMTRACK’s main contribution is to reframe memory evaluation around dynamic organizational state rather than dialogue persistence. It introduces a benchmark for multi-platform enterprise-style environments, a cross-platform timeline design with interleaved Slack, Linear, Git, and filesystem events, multiple construction strategies, and the metrics Correctness, Efficiency, and Redundancy. It also provides evidence that current frontier models remain weak in this setting and that memory backends provide limited improvement (Deshpande et al., 1 Oct 2025).

The benchmark also has clearly stated limitations. It contains only 47 instances, so it remains relatively small. It is focused on enterprise software workflows rather than all memory domains. The setting is read-only and does not yet test the harder case in which agents actively change the environment. Evaluated memory backends are limited to Mem0 and Zep. Correctness relies partly on an LLM judge. The current setup does not fully capture broader multi-agent organizational dynamics (Deshpande et al., 1 Oct 2025).

These constraints clarify what MEMTRACK is and is not. It is not yet a full benchmark for action-taking organizational agents; it does not yet evaluate settings where agents create Linear tickets, send Slack messages, or influence future events; and it is not a general benchmark for all forms of long-term memory. At the same time, the paper presents it as an extensible framework rather than a finished endpoint. Future directions explicitly suggested include larger and more diverse enterprise-memory benchmarks, extension to domains such as marketing or sales, enabling agents not only to read but also to act in the environment, richer multi-agent settings, and improved memory mechanisms that store, retrieve, and reconcile information instead of repeatedly re-reading it (Deshpande et al., 1 Oct 2025).

In this sense, MEMTRACK occupies a specific place in the evaluation landscape. It argues that long-horizon memory should be assessed under cross-platform, asynchronous, conflict-laden workflows rather than only under conversational recall. The benchmark’s empirical message is correspondingly narrow and consequential: contemporary frontier models can operate in such environments, but they do so unreliably, with substantial correctness loss, nontrivial redundancy, and limited benefit from current memory backends (Deshpande et al., 1 Oct 2025).