---
title: Persistent State Poisoning
url: https://www.emergentmind.com/topics/persistent-state-poisoning
type: topic
---

# Persistent State Poisoning

Persistent state poisoning refers to a class of attacks in which adversaries exploit or manipulate long-lived parameters, memory structures, or ancillary state within machine learning models, autonomous agents, or secure enclaves such that malicious behaviors or degraded performance persist even after online updates, retraining, or defense interventions. These attacks aim for maximal longevity and durability, circumventing standard defense assumptions predicated on the efficacy of post-training sanitization, continual alignment, or stateless model resets. Persistent state poisoning spans a spectrum from deep backdoors and memory contamination in large language models (LLMs), to cryptographically exploitable state in trusted hardware, to poisoning of replay or regularization anchors in continual learning. The following sections systematically survey the current landscape, drawing from key research across LLM security, federated and continual learning, memory-augmented agent architectures, and enclave design.

## 1. Formal Taxonomy and Definitions

Persistent poisoning is distinguished by the resilience of attack success rate (ASR) under diverse forms of model intervention or update. Let $M_p$ denote a poisoned model and $\delta: \mathcal{H} \rightarrow \mathcal{H}$ represent an arbitrary post-poisoning transformation (e.g., fine-tuning, context reset, defense layer). Persistence is formally characterized by
\[
\mathcal{P}_\delta = \operatorname{ASR}(\delta(M_p)) \approx \operatorname{ASR}(M_p) \quad \text{for a wide class of } \delta
\]
This distinguishes persistent poisons from:
- Concept poisons: train-time semantic or latent features not robust to post-processing.
- Stealthy poisons: detectable only under model or data inspection, often evanescent after fine-tuning.
- Task-specific poisons: highly targeted, but not generally robust across downstream tasks or update procedures [2506.06518].

Persistence thus implies that model or system state retains malicious correlates—whether through weights, memory, counters, or replay buffers—despite typical cleansing or adaptation steps.

## 2. Attack Models and Threat Vectors

Persistent state poisoning requires the adversary to exploit the storage or evolution of stateful artifacts. Attack surfaces span:

- **Training Data**: Injection of targeted trigger patterns, adversarial clusters, or semantic anchors during pretraining, tuning, or continual learning [2410.13722, 2507.11112].
- **Long-lived Memory**: Contamination of retrievable memory stores in memory-augmented agents, including similarity-based recall and trajectory logs [2604.02623, 2601.05504].
- **System Prompts and Configuration**: Overwriting of global system prompts or persistent context fields that are prepended or embedded in every inference cycle [2505.06493].
- **Hidden State in SSMs**: Crafting input triggers that irreversibly overwrite recurrence-based hidden state (e.g., in Mamba models), yielding durable amnesia or response shifts [2601.01972].
- **Secure Enclaves**: Manipulation of sealed data or monotonic counters in trusted execution environments, where attacks exploit improper migration semantics or offset flaws [1803.11021].
- **Federated/Continual Learning Anchors**: Multiple rounds of small, stealthy model delta injections that accumulate across aggregation or generative replay, leveraging the differential privacy noise envelope or persistent non-forget buffers [2109.09955, 2311.10919].

Successful persistent poisoning typically requires knowledge of the update pipeline and access to gradient signals or state-anchoring procedures.

## 3. Mechanisms and Empirical Results Across Domains

### 3.1 Language Models and Instruction-Tuned DNNs

- **Backdoor Persistence**: Pre-training persistency studies (e.g., “Poison Once, Exploit Forever”) show that injection of only 0.1% malicious sequences in pretraining data can result in backdoors (e.g., denial-of-service, belief manipulation, prompt-leakage) surviving both SFT and DPO stages [2410.13722]. DoS attacks with as little as 0.001% corruption persist in nearly all outputs matching the embedded trigger.
- **Multi-Trigger Robustness**: Concurrent triggers can be embedded without cross-interference (via embedding vector clustering), yielding ASR within ±2% of single-trigger value and high durability against substitution and context scrambling [2507.11112].
- **System Prompt Poisoning**: Direct modification of system prompts (explicit or via prompt templates) produces uniform, session-persistent misclassification or reasoning errors; brute-force attacks on sentiment scorers achieve near-100% persistent degradation [2505.06493].

### 3.2 Persistent Memory-Aware Agents

Memory poisoning attacks via environmental observation (eTAMP) show that single contaminated trajectory observations can propagate malicious actions cross-session and cross-site; attack success is amplified under frustration conditions (8× in GPT-5-mini, ~3.7× in GPT-5.2 under “Chaos Monkey” perturbations), and attack effectiveness remains high in advanced, highly capable LLMs [2604.02623]. In EHR LLM-agents, MINJA attacks show 95–100% injection success and 50–60% ASR under clean conditions, but rapid decline to ~7% with realistic, large pre-existing memory [2601.05504].

### 3.3 Federated, Continual, and Batch Learning

- **Federated Learning**: Differential privacy-exploited stealthy model poisoning (DeSMP) inserts statistically undetectable perturbations into the DP noise envelope, accumulating global model drift over many rounds with a negligible anomaly-detection false positive rate (~5%) but causing severe degradation (from 0.95 to 0.80 accuracy) [2109.09955].
- **Continual Learning**: The PACOL attack demonstrates that clean-label, gradient-matching poisons can induce catastrophic targeted forgetting in generative replay and regularization-based continual learnings. Label-flip attacks are readily detected, but PACOL perturbations evade all common outlier detection methods (≤10% filter rate) [2311.10919].
- **Batch LQ Control**: Sensor-spoofing attacks on batch-trained LQ controllers (pre-learning state manipulation) permit stealthy but persistent drift to attacker-selected policies (with state deviations near detection threshold), substantially degrading system response [2304.03815].

### 3.4 Trusted Execution Environments

Improper migration of SGX enclaves enables two persistent state attacks: fork (inconsistent state across hardware) and rollback (version degeneration), both allow replay or dual instantiation of security-critical state. The proposed software-only atomic migration guarantees offset-correctness, fork/rollback elimination, preserving all SGX guarantees [1803.11021].

## 4. Persistence Mechanisms and Underlying Principles

Persistence derives from mechanisms that (i) anchor attack correlates in hard-to-overwrite structures, or (ii) exploit recurrences, updates, or noise sources immune to standard purification:

- **Embedding and MLP Layer Anchoring**: LLM triggers mapped into tightly-clustered regions in the embedding/MLP subspace, robust against input modification and post-hoc fine-tuning except by direct targeted retraining [2507.11112].
- **Hidden State Recurrence**: SSMs like Mamba overwrite single-compressed hidden state via short contraction-bound triggers, causing irreversible decay/amnesia (HiSPA); attention-based models lack this vulnerability [2601.01972].
- **Cross-session Memory Contamination**: Memory augmented agents with similarity-retrieval prepend previously stored, poisoned content, enabling attacks to “wake up” under target conditions well after injection [2604.02623, 2601.05504].
- **Noisy Accumulation in FL**: Gaussian DP noise admits bounded but cumulatively high-magnitude adversarial shifts (mean offset within KL divergence); rounds can be calibrated for persistent effect without raising detection alarms [2109.09955].
- **Replay and Regularization Anchors**: Continual learners’ reliance on buffer replay or parameter importance weights makes gradient-aligned clean-label poisoning both durable and resilient to outlier removal [2311.10919].

## 5. Evaluation Metrics and Empirical Observables

Standard metrics for quantifying persistent poisoning include:
- **Attack Success Rate (ASR)** before and after restorative or update operations,
- **Persistence Gap (Δ)**: post-alignment or post-retraining deviation from clean baseline [2410.13722],
- **Injection Success Rate (ISR)** (in memory systems) and retrieval probability under realistic (noisy, large-memory) conditions [2601.05504],
- **Clipped Heidke Skill Score (CHSS)** for information retrieval tasks in SSMs [2601.01972],
- **Formal invariants** in enclaves: counter monotonicity and sealed-data freshness [1803.11021],
- **Defense filter rate** under outlier or anomaly detection [2311.10919].

## 6. Defenses and Limitations

Defenses targeting persistent poisons must address durability, stealth, and stateful propagation:

- **Selective Layer Re-initialization**: Retraining only the embedding+MLP sublayers (rather than full weight reset) restores most model integrity at ∼78% parameter cost compared to 100% for full tuning [2507.11112].
- **Hybrid Memory Filtering**: Composite trust scores using multi-source heuristics and semantic classifiers, combined with temporal decay and re-execution checks, provide partial resilience in agents, but high thresholds risk either memory starvation or incomplete filtering [2601.05504].
- **Anomaly/Norm-based Detection**: Monitoring mid-block L₂ norms in SSMs exposes HiSPA attacks without retraining; this suggests lightweight, forward-only runtime filtering [2601.01972].
- **RL-Tuned Privacy/Utility Tradeoff**: Adaptive reinforcement learning on FL privacy budget (ε) shrinks the available stealth space for DP-exploited attackers, balancing utility and defense [2109.09955].
- **Atomic Migration Protocols**: In enclave migration, selective erasure and controlled counter/offset transfer strictly prevent forks and rollbacks with negligible overhead [1803.11021].

Limitations are pronounced. Advanced clean-label or gradient-matching poisons evade all current feature- or distribution-based filtering at practical budgets; highly persistent memory or context poisons can be almost undetectable in long-lived, personalized agents; targeted retraining incurs partial utility loss or high computational cost. Certified, adversarially-robust learning, gradient-norm monitoring, and provable integrity checks remain open challenges.

## 7. Open Problems and Future Research Directions

Persistent state poisoning remains a dynamically evolving threat:
- **Detection of non-triggered global biases** (e.g., in belief manipulation) is particularly challenging; benchmarks and robust debiasing algorithms are lacking [2410.13722].
- **Generalization to multimodal and retrieval-augmented systems** is largely unstudied.
- **Adaptive thresholds and online trust distribution modeling** may enhance practical defense in memory-augmented systems, but tuning the safety–utility tradeoff remains unresolved [2601.05504].
- **Robust continual learning updates** with certified guarantees against cumulative or clean-label poisoning require novel regularization and anomaly-detection frameworks [2311.10919].
- **Hybrid architectures and state-resilient SSMs**—architectural modifications (gated resets, bounded contraction) combined with runtime L₂-norm checks merit further investigation [2601.01972].
- **Industrial and regulatory implications**: Canary backdoors, audit trails for memory/appends, and verifiable system/prompt integrity may become essential components of large-scale ML deployments [2410.13722, 2505.06493].
- **Adaptive, adversary-aware privacy budgeting** in federated and distributed settings [2109.09955].

The persistence of stateful poisoning across all major learning and inference architectures highlights the need for integrated, hardening-oriented research spanning engineering controls, runtime behavioral analysis, and fundamental algorithmic robustness.

Source: https://www.emergentmind.com/topics/persistent-state-poisoning