---
title: Knowledge Corruption Attacks
url: https://www.emergentmind.com/topics/knowledge-corruption-attacks
type: topic
---

# Knowledge Corruption Attacks

Knowledge corruption attacks are adversarial techniques designed to manipulate machine learning or agentic systems by stealthily corrupting their knowledge sources, memory, or training data, causing models or agents to produce misleading, incorrect, or attacker-controlled outputs. These attacks target knowledge persistence artifacts—databases, client-side memory, document loaders, or training sets—rather than simply manipulating prompts or test-time inputs. Knowledge corruption attacks have rapidly gained prominence in the context of Retrieval-Augmented Generation (RAG), web agents, and deep classifiers, where even minimal poisoning budgets or subtle memory injections can yield high attack success rates without affecting standard accuracy or evading detection by conventional filtering mechanisms.

## 1. Taxonomy, Formal Definitions, and Threat Models

Knowledge corruption attacks span multiple modalities, architectures, and adversarial strategies. The two principal settings are:

- **Training-time corruption (data/model poisoning):** The adversary injects malicious examples or manipulates labels/features in the training dataset to implant backdoors or corrupt the learned decision boundaries, often with imperceptible perturbations [1902.11237], [2210.11592].
- **Retrieval/Memory-time corruption:** The adversary targets persistent knowledge artifacts—retrieval stores, agent context/state, external documents, or vector databases—by seeding them with semantically relevant but attacker-crafted content, which is later retrieved and grounds model predictions [2402.07867], [2508.18652], [2504.03957].

**Formal threat models** specify: (i) adversary capability (injection budget, location, granularity), (ii) adversary knowledge (black/grey/white-box), (iii) invariants (e.g., label preservation), and (iv) attack objectives (ASR maximization, targeted misclassification, denial of service).

**Example formalization (RAG setting)** [2402.07867], [2508.18652]:

Let $\mathcal{D}$ be the knowledge database, $\mathcal{Q}$ a set of target queries, $\mathrm{RAG}(\cdot)$ the system, and $\Gamma$ the set of injected adversarial passages. The attacker's goal is:
\[
\max_{\Gamma: |\Gamma|\leq B} \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} \mathbf{1}\left(\mathrm{RAG}(q, \mathcal{D} \cup \Gamma) = A(q)\right)
\]
with $B$ the budget, $A(q)$ the attacker-specified answer.

In **plan injection for web agents** [2506.17318], the context $c_t = (p_t, d_t, k, h_t)$ is corrupted by perturbing memory or plan representations:
\[
M'_t = M_t + \Delta, \quad ||\Delta|| \leq \beta
\]
or by direct plan injection,
\[
c^* = (p_i, d_{i,t}, k, h_{i,t}, P_i \oplus \delta_P)
\]

## 2. Canonical Attack Techniques and Mechanisms

### 2.1 Retrieval-Augmented Generation and Knowledge Stores

**Query-targeted document poisoning:** PoisonedRAG and CorruptRAG demonstrate that injecting a small number of attacker-constructed passages into a large document corpus enables the adversary to reliably steer RAG outputs. Each poisoned passage is optimized for both the retriever (high similarity to target query) and the LLM (eliciting attacker-chosen answer) [2402.07867], [2504.03957]. CorruptRAG shows that a *single* injected document per target query achieves ASR up to 0.97–0.98, even with top-5 retrieval in million-scale datasets.

**Universal attacks (UniC-RAG):** Rather than per-query poisons, UniC-RAG jointly optimizes $n \ll |\mathcal{Q}|$ adversarial texts for $|\mathcal{Q}|$ diverse queries, via balanced clustering and HotFlip-based similarity maximization [2508.18652]. This yields >90% ASR against 2,000 queries with 100 passages.

**Stealthy ingestion attacks:** Loader-centric attacks leverage invisible Unicode manipulation, zero-width characters, font poisoning, out-of-bound text, and transparent overlays to inject or obfuscate data at the parsing stage, often completely bypassing document sanitizers [2507.05093]. These attacks affect RAG systems at the pre-embedding phase and are highly format-agnostic.

### 2.2 Memory and Context Corruption in Agents

**Plan/context injection:** In agentic settings, plan injection alters the high-level plan representation without touching user prompts or agent code [2506.17318]. Context-chained variants semantically bridge user goals and attacker objectives so that validation layers are bypassed. Bypassing prompt-injection defenses, these attacks exploit the inherently insecure architecture of client- or third-party managed agent memory.

### 2.3 Data Poisoning in Supervised Learning

- **Training-set corruption without label poisoning:** Imperceptible, class-restricted perturbations are added to a subset of target-class samples; no label anomalies are introduced, maintaining high clean accuracy but enabling backdoor activation at inference [1902.11237].
- **Label-flipping attacks:** The adversary randomly or selectively inverts labels (e.g., flipping only high-importance malicious samples to benign) to degrade classifier integrity or selectively reduce recall [2210.11592].

## 3. Empirical Effectiveness and Attack Evaluation

Attack efficiency is typically measured by **Attack Success Rate (ASR):**
\[
\mathrm{ASR} = \frac{\#\{\text{successful attacks}\}}{\#\{\text{attempts}\}}
\]

### Retrieval-based attacks:
- PoisonedRAG: $>90\%$ ASR (5 poisons/query, millions of texts, $k=5$) [2402.07867].
- CorruptRAG: Up to $0.97$–$0.98$ (single poison/query) [2504.03957].
- UniC-RAG: $>90\%$ ASR for 2,000 queries with 100 passages [2508.18652].
- Loader attacks: $74.4\%$ ASR across document parsers/formats [2507.05093].

### Agent memory attacks:
- Plan injection boosts ASR up to $3\times$ over prompt injection; context-chained injection further raises ASR by $+17.7\%$ in privacy exfiltration [2506.17318].

### Data/model poisoning:
- CNN backdoor: up to $96\%$ ASR with $\alpha=0.4$, imperceptible triggers, and $>99\%$ pristine accuracy [1902.11237].
- Malware classifiers: up to $25$–$40\%$ accuracy drop at $\alpha=0.5$ random flips; targeted flips maintain higher overall accuracy but cause recall to collapse on targeted class (e.g., FNR $>35\%$ for Decision Trees) [2210.11592].

## 4. Impact, Limitations of Defenses, and Architectural Consequences

Existing defenses against knowledge corruption attacks have been found inadequate in several dimensions:

- **Robust RAG variants:** Techniques relying on LLMs to individually inspect context, secure aggregation of responses, or increased retrieval width only partially mitigate attacks. Expanding the context window often increases the probability that adversarial passages are retrieved [2402.07867], [2504.03957], [2508.18652].
- **Paraphrasing and instructional filtering:** Paraphrasing queries drops ASR marginally ($\sim$10%); adversarial passages remain high-similarity nearest neighbors [2402.07867], [2508.18652].
- **Heuristic content filters and surface sanitization:** Most loader-based obfuscation and injection attacks circumvent canonicalization, codepoint filtering, and metadata validation unless OCR/fallback is used [2507.05093].
- **Correct knowledge expansion:** Injecting multiple benign correct-answer passages can dilute attack efficacy but is not scalable and comes with trade-offs [2504.03957].

High-performing defenses such as RAGDefender (post-retrieval embedding clustering and scoring) achieve strong ASR reduction (e.g., $0.89 \to 0.02$ on Gemini-1.5-Pro at 4× adversarial ratio) [2511.01268]. Certifiably robust aggregation (RobustRAG; isolate-then-aggregate with thresholded keyword or decoding fusion) delivers formal $\tau$-robustness guarantees when $k' < k$ passages are corrupted [2405.15556], but incurs nontrivial compute overhead.

In agentic systems, cryptographic signing of plans, API-based memory isolation, and embedding-based consistency checks are mandated to control plan injection risk [2506.17318]. Training-side poisoning in classifiers is best mitigated by clustering, spectral analysis, label sanitization, or holistic robust optimization (distributionally robust losses that capture both evasion and poisoning) [2210.11592], [2303.02251].

## 5. Methodological Innovations: Optimization, Clustering, and Retrieval

Recent advances center on optimization of knowledge corruption payloads:

- **Two-stage poison construction:** Split poisoned document into a retrieval-optimized prefix and generation-optimized payload [2402.07867], [2504.03957].
- **HotFlip-based text optimization:** Use white-box gradient-based perturbations for semantic similarity maximization between poisoned text and target query clusters [2508.18652].
- **Balanced clustering:** Assign queries to clusters such that each adversarial passage targets a maximally homogeneous and balanced set, increasing retrievability and ASR [2508.18652].
- **Loader-level attack toolkits:** Automated suites (e.g., PhantomText) test multiple parser-breaking and hiding strategies for format-agnostic RAG ingestion corruption [2507.05093].

## 6. Open Challenges and Research Directions

While most current defenses rely on post-hoc or heuristic filtering, there is a clear need for:

- **Provable, certifiable retrieval robustness:** Formal certification of retrieval aggregation, as in RobustRAG, is emerging but comes with costs in latency and LLM calls [2405.15556].
- **Universal and adaptive poison resilience:** Approaches that scale to universal, domain-spanning attacks with minimal budget and support real-world, open-domain RAG systems [2508.18652].
- **Integrated, multi-layered sanitization:** Combining shallow surface preprocessing, loader-format validation, and deep anomaly detection [2507.05093].
- **Memory and context integrity in agents:** Mandatory cryptographic and process isolation for persistent agent memory [2506.17318].
- **Fundamental trade-offs:** Studying limits of retrievability vs. manipulability and the balance between ASR reduction and benign coverage [2508.18652].

The rapidly evolving landscape of knowledge corruption attacks thus highlights the urgent requirement for architectural, theoretical, and practical advances in defense, detection, and system design to protect against this multi-faceted, highly effective attack surface.

Source: https://www.emergentmind.com/topics/knowledge-corruption-attacks