MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair

Abstract: Memory systems allow agents to retain and reuse information from past interactions, but they can also let malicious content persist. A malicious instruction crafted by an attacker may be stored in long-term memory, recalled much later, and quietly shape a real action. Recent benchmarks increasingly examine agent memory security, yet few trace the same malicious semantics across persistence, downstream consequences, and selective repair under diverse memory-backend comparisons. To address this gap, we introduce MemSecBench, a task-grounded benchmark for the lifecycle security of agent memory systems. It contains 310 cases drawn from 48 realistic contexts across code and science, daily life, and office work. Each case follows a controlled Write--Execute--Forget protocol in an isolated runtime under an exact agent configuration, defined by an agent harness, a memory backend, and an LLM backend. Evidence-based adjudication combines a deterministic write check, checkpoint-specific judge-model evaluations, and programmatic gates across seven lifecycle checkpoints. The experimental design spans a 24-configuration matrix of two agent harnesses, four memory backends, and three LLM backends. Across all 24 configurations, malicious memory persists in 84.2% of all cases, and the full Write--Execute chain succeeds in 50.3%. Among successfully poisoned cases, 59.6% complete the full Execute chain, while 56.1% achieve selective repair.Compared with matched Native configurations, the largest absolute differences are 16.1 percentage points for end-to-end attack success and 41.3 percentage points for selective repair. These descriptive contrasts indicate that the evaluated memory system stacks differ in lifecycle security, both in the propagation of malicious memory and in selective repair after successful memory poisoning.

• The paper introduces a benchmark evaluating the security of persistent memory systems in LLM agents, focusing on an end-to-end memory poisoning cycle.
• The evaluation identifies adoption as a crucial junction for defenses, as 50.3% of attacks achieve external consequences once memory poisoning is successful.
• The research shows that benign memory preservation is a primary challenge in repair processes, with heterogeneity among adapted backends affecting overall security dynamics.

Persistent memory systems for LLM agents introduce a security surface that differs qualitatively from single-turn prompt injection: malicious content encountered in one session can be written to long-term storage, persist indefinitely, and later be recalled to shape behavior in an unrelated task. MemSecBench (2607.27080) addresses the evaluation of this threat with a task-grounded benchmark that traces attacker-controlled semantics through a complete Write–Execute–Forget lifecycle, from persistence in the memory backend through verified external consequence to selective repair. The central contribution is methodological: rather than testing whether an LLM recognizes malicious content, it tracks the same malicious semantics end-to-end under controlled, matched agent configurations, enabling paired comparisons across memory backends.

Positioning relative to prior evaluations

Prior work covers fragments of this lifecycle. Agent-security benchmarks such as InjecAgent and AgentDojo evaluate unsafe actions from adversarial content but do not model persistent memory; memory-utility benchmarks such as LoCoMo and LongMemEval measure cross-session utility rather than attack propagation; memory-poisoning studies such as AgentPoison and MPBench demonstrate persistence or downstream influence individually. The authors organize the comparison along five dimensions: intended-interface write (IW), verified external consequence (DC), complete poisoning–retrieval–trigger chain (ST), selective repair with benign preservation (SR), and matched multi-backend instrumentation (MMI). On this rubric, most prior protocols fully cover at most one operational-fidelity dimension, and no prior work fully covers SR — jointly verifying removal of malicious semantics and semantic preservation of required benign memories. MemSecBench is designed to satisfy all five criteria within each case.

Threat model and benchmark structure

The evaluated unit is a complete agent configuration $\Pi = (H, B, L)$ comprising an agent harness (OpenClaw or Hermes), a memory backend stack (implementation plus adapter plus active service settings), and an LLM backend. The attacker controls content that conveys target malicious semantics via one carrier (user interaction, supply-chain tool, workspace file, or external content) but cannot modify the configuration or access hidden backend state. The later user submits a benign task with no payload; success requires that stored semantics be recalled ($E_1$), adopted ($E_2$), and produce a case-defined externalized consequence ($E_3$). Persistence alone ($W_2$) is explicitly classified as an intermediate outcome, not evidence of a completed attack — a useful discipline given that some prior work reports poisoning rates without downstream verification.

The corpus comprises 310 linked task packages drawn from 48 contexts spanning Code and Science, Daily Life, and Office Work. Each package $T = (\rho, x_W, x_E, x_F, \ell)$ pairs clean initialization material with stage-specific tasks and evaluator metadata. Cases are labeled on four design axes — Primary Failure Mode (seven mechanisms including Scope/Condition Failure, Provenance/Authority Failure, Temporal Validity Failure, and Memory Composition Failure), Risk (five consequence categories), Carrier, and Memory Type — used for stratification only, never as checkpoint evidence. Construction uses a skill-guided protocol in which GPT-5.5 proposes content under a locked design brief, with staged rendering, static validation gates, SHA-256 manifests, and mandatory human approval per case.

Evidence-based adjudication

A notable feature is the adjudication protocol. Except for the deterministic write check $W_1$, each checkpoint is judged by a pinned judge model (DeepSeek-V4-Pro, temperature 0, JSON mode, outside the agent run) applying a checkpoint-specific rubric to a restricted evidence pack, followed by programmatic gates enforcing source admissibility, temporal dependencies, requirement completeness, and cross-artifact consistency. Agent self-reports and tool arguments are categorically inadmissible for Persistence, Externalization, and Repair verdicts. Externalization requires a final service record (web gateway, Mailpit, or GitLab) satisfying the consequence contract; plans and drafts do not count. Judge-model agreement with two independent human annotators on a 500-record stratified sample was 90.60% and 91.80%. The anti-injection system prefix treating all evidence as untrusted is a sensible safeguard against the judged content itself steering the evaluation.

Experimental results

The evaluation spans a 24-configuration matrix (2 harnesses × 4 backends × 3 LLMs), each covering all 310 cases once. Four metrics are reported: Memory Poisoning Success Rate (MPSR), Memory Exploitation Success Rate (MESR), End-to-End Attack Success Rate (E2E-ASR), and Selective Repair Success Rate (SRSR). Aggregate findings are stark:

Metric	Macro-average
MPSR (all cases)	84.2%
E2E-ASR (all cases)	50.3%
MESR (given poisoning)	59.6%
SRSR (given poisoning)	56.1%

Finding 1 — adoption is the bottleneck. Checkpoint pass rates decline only mildly through persistence and recall: 91.1% at $W_1$, 84.2% at $W_2$, 76.1% at $E_1$. The sharp contraction occurs at Adoption, which falls to 53.7%; the full chain then declines only slightly further to 50.3%. The implication is that once recalled malicious semantics shape the agent's decisions, externalization usually follows — so defenses must intervene before or at adoption, not merely filter retrieval.

Finding 2 — benign preservation dominates repair failure. Among successfully poisoned cases, 86.3% pass target removal ($E_1$0) but only 62.5% pass benign preservation ($E_1$1), yielding SRSR of 56.1%. The 30.2-point gap versus the 6.4-point gap in the reverse direction identifies collateral damage during removal — not failure to remove — as the primary obstacle to safe recovery. Reporting deletion-only success therefore overstates post-compromise recoverability.

Findings 3–4 — no backend is uniformly safer, and resistance ≠ repairability. Matched contrasts separate the two security requirements. Under OpenClaw/DeepSeek-V4-Pro, replacing Native with A-MEM barely reduces E2E-ASR (59.0% → 56.5%) while sharply raising SRSR (46.3% → 87.6%). Conversely, Mem0-Graph under OpenClaw/GPT-5.5 cuts E2E-ASR substantially (57.1% → 41.0%) with essentially unchanged repair (87.5% → 86.7%). Backend effects also reverse across harness–LLM pairings: Mem0 lowers both metrics relative to Native under OpenClaw, yet under Hermes/MiniMax-M3 raises E2E-ASR from 34.8% to 48.7% while improving SRSR from 47.3% to 60.0%. Backend choice thus reshapes security materially, but its effect is conditional on the full configuration, precluding universal rankings.

The mechanistic case study traces one smart-home data-sharing rule through all four backends — appended as a Markdown rule, a pgvector row, an entity-linked record, and a structured MemoryNote respectively — showing that heterogeneous representations and retrieval logic all admit persistent poisoned state, while also delineating a design space for backend-specific defenses.

Limitations and open questions

The paper is candid about scope. Each configuration–case pair is run once, so all reported rates are descriptive point estimates without variance quantification; MESR and SRSR condition on successful writes within each configuration, complicating direct cross-backend comparison of conditional metrics. The threat model assumes content reaches a supported carrier and that write invokes the intended interface; it excludes OS or database compromise, account acquisition, identity binding, and natural trigger probabilities. Native is harness-specific, so backend comparisons remain within harness. F1/F2 verdicts cover entries exposed by backend snapshots — inaccessible caches and latent representations fall outside the audit — and semantic preservation does not guarantee later retrievability. Judge agreement near 91% leaves residual adjudication error unquantified beyond accuracy. Open questions include how mechanism-specific attacks exploit particular backend representations, and whether controlled ablations can isolate backend effects currently confounded with adapter and service settings.

Conclusion

MemSecBench provides a rigorous, reproducible framework for evaluating agent memory security across the full lifecycle — persistence, verified externalization, and selective repair — under matched configurations. Its principal empirical conclusions are that current stacks are pervasively susceptible (84.2% poisoning, 50.3% end-to-end success across all 24 configurations), that adoption is the decisive defensive boundary, that benign-memory preservation is the dominant obstacle to genuine recovery, and that no memory backend offers uniformly superior security. These results collectively argue that memory safety claims require evaluation of the complete harness–backend–model stack across lifecycle transitions, not isolated checkpoints or model-level red-teaming alone.