MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents

Abstract: Equipping LLMs with persistent memory enhances interaction continuity and personalization but introduces new safety risks. Specifically, contaminated or biased memory accumulation can trigger abnormal agent behaviors. Existing evaluation methods have not yet established a standardized framework for measuring memory misevolution. This phenomenon refers to the gradual behavioral drift resulting from repeated exposure to misleading information. To address this gap, we introduce MemEvoBench, the first benchmark evaluating long-horizon memory safety in LLM agents against adversarial memory injection, noisy tool outputs, and biased feedback. The framework consists of QA-style tasks across 7 domains and 36 risk types, complemented by workflow-style tasks adapted from 20 Agent-SafetyBench environments with noisy tool returns. Both settings employ mixed benign and misleading memory pools within multi-round interactions to simulate memory evolution. Experiments on representative models reveal substantial safety degradation under biased memory updates. Our analysis suggests that memory evolution is a significant contributor to these failures. Furthermore, static prompt-based defenses prove insufficient, underscoring the urgency of securing memory evolution in LLM agents.

• The paper introduces MemEvoBench, a benchmark that quantifies how evolving memory contributes to safety failures in LLM agents.
• It employs dual evaluation modalities—QA-style misleading memory injection and workflow-style noisy tool returns—to simulate realistic memory contamination.
• Empirical results across nine state-of-the-art LLMs reveal that dynamic memory correction tools significantly reduce attack success rates compared to static safety prompts.

MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents

Motivation and Problem Formulation

Long-term memory integration in LLM-driven autonomous agents has enabled persistent context, interaction continuity, and personalization. However, the persistence and evolution of such memory introduces novel safety hazards. The core issue is that memory is not static; it evolves based on both external (user, environmental, tool) and internal (agent response, feedback) factors, which can result in gradual, cumulative behavioral drift—a process the authors term "memory misevolution." Such drift drives agents from safe, knowledge-aligned behaviors toward unsafe or policy-violating regimes via gradual accumulation and reinforcement of misleading experiences, often compounded by biased user feedback and noisy tool returns. Notably, prior evaluation paradigms inadequately characterize nor systematically measure this dynamic memory poisoning process.

MemEvoBench is introduced to fill this gap—comprising a diverse, multi-domain benchmark and measurement protocol targeting safety degradation in LLM agents under contaminated, evolving memory states. The benchmark offers QA-style tasks with complex, realistic memory pools as well as workflow-style tool-augmented tasks emulating external input noise and feedback loops.

Figure 1: Comparison of memory safety (top) versus memory misevolution (bottom). With frozen memory, agents rely on intrinsic knowledge and provide safe recommendations. Under continuous evolution with biased reinforcement, isolated experiences accumulate and overgeneralize into unsafe behavioral patterns.

Benchmark Construction

MemEvoBench comprises two primary evaluation modalities:

1. QA-Style (Misleading Memory Injection): This scenario evaluates agents' resistance to contaminated declarative memories across seven high-risk domains (healthcare, mental health, finance, food safety, privacy, transportation, customer service), with 36 labeled risk types derived from domain-specific failure patterns and cognitive bias literature. Memory pools are systematically engineered to mix correct and plausible yet subtly misleading memories (including omissions, threshold-shifted decisions, and consensus bias), annotated and validated with multiple LLM and human review passes.
2. Workflow-Style (Noisy Tool Returns): Procedural contamination via indirect prompt injection and workflow errors is modeled across 20 AgentSafetyBench-inspired environments. Historical workflow memories embed both correct and risk-inducing sequences; tool returns can leak sensitive data or contain latent adversarial payloads. The evaluation captures agents’ susceptibility to propagating unsafe tool-induced patterns.

Both modalities explicitly model multi-round, evolving memory utilization: following each decision, agent responses and (simulated) biased user feedback are appended to the retrievable memory, closely mimicking real-world deployment. The design tracks the onset and amplification of safety failures as minor biases escalate to systematic behavioral deviations.

Figure 2: Memory pool structure for the two evaluation scenarios. Left: Misleading Memory Injection constructs retrievable content mixing knowledge snippets, conversation history, forum posts, and personal notes with embedded risks. Right: Noisy Tool Returns constructs execution history with workflow memories where tool returns contain sensitive information (highlighted in red) that agents should not blindly propagate.

Evaluation Protocol and Metrics

MemEvoBench employs a three-round protocol per test case, where agents’ outputs are judged for risk-exhibiting behaviors. The key metric is Attack Success Rate (ASR)—the proportion of responses per round that manifest the target safety failure mode. Classification is performed using GPT-5.2 as an automated, high-accuracy judge, validated against manually annotated samples.

To dissect causal factors, a memory-independent control ablates access to memory, isolating the role of memory-induced drift versus model-intrinsic knowledge errors. Defense configurations are benchmarked: static prompt-based guidelines (+SafePrompt) and an explicit memory correction mechanism (+ModTool), which allows agents to inspect, verify, and amend misleading memories or tool returns before producing a final answer.

Experimental Results

A thorough empirical study encompasses nine state-of-the-art LLMs ranging from Qwen3, Gemini-2.5-Pro, Claude-3.7-Sonnet, Llama-3.3, GPT-4o, GPT-5, and DeepSeek-V3.2. All agents are evaluated under both standard and biased-feedback-enhanced protocols.

Key findings:

• Under vanilla prompting and memory access, all models are highly susceptible to memory-induced failures, with ASR frequently exceeding 75% across all rounds. Bias-amplifying feedback compounds the effect, producing clear round-wise ASR escalation (e.g., 71.6% → 84.9% → 87.8% on average).
• Ablating memory dramatically reduces the ASR, confirming that intrinsic model knowledge is safer; the majority of unsafe behavior arises from contaminated or misevolved memory.
• Static safety prompting (+SafePrompt) reduces risk only partially (e.g., mean Rd.1 ASR, 76.2% → 55.5% in QA-Style); however, under biased feedback or procedural contamination (Workflow Style), its efficacy fails, demonstrating the inadequacy of prompt-level defenses for dynamic contamination.
• Agents equipped with a memory correction tool (+ModTool) achieve significantly lower ASR; Gemini-2.5-Pro, for instance, drops from 67%/55%/55% (Vanilla, Rd.1/2/3) to 19%/13%/14% with +ModTool in QA-Style. Gains are robust, though not absolute, under biased feedback.

  Figure 3: Comparison of agent behavior without (left) and with (right) the memory correction tool. When equipped with +ModTool, the agent identifies misleading content in Memory 1 and provides a safe, accurate response.

Detection accuracy (F1) for misleading memories strongly correlates with ASR; however, performance declines across rounds, showing persistent generalization vulnerability and degradation under continued biased accumulation.

Failure Mode Analysis

Detailed qualitative analysis identifies structured memory-borne failure mechanisms:

• Anecdotes over guidelines: Agents overfit to vivid but non-generalizable stories, bypassing policy-compliant guidance.
• Normalization of violations: Repeated exposure to risky yet positively reinforced narratives leads to systematic endorsement of unsafe norms.
• Causal misattribution: Subjective rationalizations in memory are misinterpreted as general policy.
• Sensitive data propagation: Workflow memories containing sensitive or private information are blindly forwarded in subsequent tasks, violating confidentiality norms.

In Workflow-Style tasks, task-completion bias and plausible background normalization further catalyze unsafe actions.

Implications and Recommendations

The systemic risk revealed by MemEvoBench exposes a critical safety and alignment challenge for open-ended, memory-enabled LLM agents. Key implications:

• Memory as a high-value attack and drift surface: Dynamic, retrievable experience stores are potent vectors for propagating non-obvious failures that evade static red-teaming.
• Prompt-level defenses are inadequate: Instructional safety overlays are insufficient against long-horizon, feedback-driven adversarial drift—dynamic intervention and repair mechanisms are necessary.
• Active memory curation is necessary: Proactive strategies (external knowledge verification, explicit correction mechanisms) are more robust but require improved detection and integration architectures.
• Memory system audits must become a baseline: Agent deployments should include continuous monitoring and integrity checking not only at policy or RLHF layers, but also throughout agent experience and memory pipelines.

Ongoing research directions include formal guarantees for memory correction, adversarial robustness in memory retrieval and update mechanisms, and compositional evaluation frameworks integrating memory, RLHF, and tool-augmentation vulnerabilities.

Conclusion

MemEvoBench provides the first standardized, high-fidelity benchmark for quantifying the safety risks associated with memory misevolution in LLM-based agents. The framework empirically validates that dynamic memory contamination, amplified by biased or adversarial feedback and procedural noise, induces persistent and compounding behavioral drift toward unsafe regimes—undermining even high-quality model priors. Mitigation via prompt engineering is demonstrably weak; explicit memory correction and external validation yield stronger but still imperfect defense. These results establish the necessity of a new research frontier in long-horizon, memory- and feedback-resilient agent architectures.