Knowledge Corruption Attack

• Knowledge corruption attacks are deliberate manipulations of AI inputs, training data, and inference signals that force the system to produce attacker-chosen outputs.
• They exploit vulnerabilities through methods such as training-data poisoning, backdoor triggers, retrieval-based injections, and memory corruption, often achieving high attack success rates while remaining stealthy.
• Defensive approaches, including robust data aggregation, adversarial fine-tuning, and memory sanitization, are critical yet challenging to implement due to the subtle and universal nature of these attacks.

A knowledge corruption attack is any deliberate manipulation of information sources, intermediate representations, or learning signals designed to compromise an AI system’s internal representation of the relevant world or task. The defining property is that, after the attack, the compromised model or agent systematically maps input features or queries to attacker-chosen outputs, often overriding or subverting correct, content-based decision boundaries. Knowledge corruption can be executed in both training (data poisoning, backdoors) and inference (RAG knowledge base contamination, memory manipulation) pipelines, and encompasses a spectrum from stealthy sub-perceptual triggers to large-scale, universal attacks on retrieval-based or agentic systems. The field investigates the threat models, mechanisms, and empirical effectiveness of such attacks, as well as scalable and certifiable defenses in modern machine learning and AI deployments.

1. Core Threat Models

Knowledge corruption attacks can be categorized by the interface they target and the adversary’s assumptions:

• Training-Data Poisoning (Supervised/Unsupervised/Deep Learning): The adversary modifies a fraction of the training data to implant triggers or bias feature-label mappings, without altering model code, architecture, or validation set and often in a network-agnostic fashion (Alberti et al., 2018, Barni et al., 2019). Example: inserting a single-pixel or structured signal on particular class samples to create a backdoor.
• External Knowledge Contamination (Retrieval-Augmented Generation, RAG): The attacker injects adversarial passages or documents into an external datastore queried at inference time, so the retriever returns attacker-crafted content that steers the model to chosen outputs (Zou et al., 2024, Zhang et al., 4 Apr 2025, Geng et al., 26 Aug 2025, Kim et al., 3 Nov 2025).
• Memory and Plan Corruption (Agentic Systems): Manipulating or injecting steps into an agent’s persistent external memory or high-level plan to hijack future reasoning or action sequences (Patlan et al., 18 Jun 2025).
• Parameter-Space or Latent-Space Tampering: Stealth modification of network weights or neurons in a manner that preserves validation accuracy on untarnished data but yields malicious outputs on trigger inputs (Tyukin et al., 2021).
• Reinforcement Learning Corruption: The adversary alters a fraction of offline transition tuples $(s, a, r, s')$ in batch RL, inducing suboptimal policies with a provable dimension-dependent optimality gap (Zhang et al., 2021).

2. Mathematical Formulation and Objective Functions

Across settings, knowledge corruption adheres to a bilevel or constrained optimization structure. The generic paradigm is:

• Given a training set $D = \{(x_i, y_i)\}$ or database $D$, define a tampering function $\tau(x)$ (e.g., trigger insertion, semantic poisoning).
• For task-specific variants:
  • Backdoor/data poisoning (supervised):

  $$L_{adv}(\theta) = \mathbb{E}_{(x, y) \sim D_{tampered}} \left[ \ell(f_\theta(x), y') \right],\ \text{where } D_{tampered} = \{(\tau(x_i), y_i) : y_i = A\} \cup \{(x_i, y_i) : y_i \neq A\}.$$

  The adversary aims to maximize the test-time misclassification rate on inputs stamped with $\tau$, while keeping validation performance nearly unchanged (Alberti et al., 2018). - RAG corruption:

  $$\max_{\Gamma} \;\frac{1}{|Q|}\sum_{q \in Q} \mathcal{V}\left(f\left(q; R(q; D \cup \Gamma)\right), O\right), \ \text{s.t. } |\Gamma| \ll |Q|,$$

  where $R(·)$ is the retriever, $O$ is the attacker’s universal or targeted objective, and $\mathcal{V}$ is a success metric (Geng et al., 26 Aug 2025). - Offline RL: Given $N$ transitions, up to $\epsilon N$ can be corrupted arbitrarily, and the minimax optimality gap is $\Omega(d\epsilon)$ in linear MDPs ($d$ = feature dim) (Zhang et al., 2021).

3. Attack Methodologies: Instantiations and Empirical Effectiveness

Key attack methodologies include:

• Single-pixel and structure-trigger backdoors: Adversary selects a single image location and channel, modifying all pixels at that location in class $A$ training samples. At test time, stamping this trigger on any image causes persistent misclassification into $A$ regardless of genuine content, with attack success rates (ASR) up to 99% and clean accuracy drop ≤2% across diverse architectures (Alberti et al., 2018). Similar phenomena are observed for signal-based triggers (e.g., ramp, sinusoid) that are essentially invisible but still induce 70–90% targeted misclassification rates in vision models (Barni et al., 2019).

• Retrieval-based injection (PoisonedRAG, CorruptRAG, UniC-RAG):

  • Query-specific: For each target query, attacker injects a small number (even one) of adversarial documents crafted to both maximize retrieval similarity and steer LLM outputs via soft or adversarial template payloads. CorruptRAG achieves near-perfect ASR (0.95–0.97) per query on benchmarks by injecting a single text and uses LLMs to refine semantic plausibility for stealth (Zhang et al., 4 Apr 2025).
  • Universal attacks: UniC-RAG clusters queries and optimizes a small universal set of adversarial texts. With only 100 such texts in databases with millions of entries, UniC-RAG simultaneously attacks thousands of queries with ASR exceeding 90% (Geng et al., 26 Aug 2025).
  • Classical PoisonedRAG: Even 5 injected texts per query can yield 97–99% ASR on real-world QA datasets (Zou et al., 2024).
• Plan injection in web agents: Manipulation of structured plans stored in vulnerable external memory (e.g., browser storage or third-party DBs) to insert task-aligned or context-chained malicious steps. Attack success rates reach up to 94.7% for opinion steering, 78.7% for advertisement injection, and 63% for privacy exfiltration, with standard prompt-level defenses proving ineffective (Patlan et al., 18 Jun 2025).
• Stealth attacks and parameter modification: By exploiting overparameterization, an attacker can swap or augment a single neuron in a network head, ensuring no change on the secret validation set but arbitrary outputs for a crafted trigger, with probability of stealth and success increasing rapidly with latent width (Tyukin et al., 2021).

4. Defensive Techniques and Limitations

Defenses against knowledge corruption attacks are generally divided into data-centric, model-centric, and retrieval/aggregation-centric strategies. Empirical results demonstrate that naïve defenses fail consistently, necessitating more nuanced and certifiable defense architectures.

• Data-centric: Median filtering, outlier analysis, visual attention maps, or clustering can at best mitigate simple trigger attacks. Sophisticated triggers and stealthy label-preserving poisonings evade such filters (Alberti et al., 2018, Barni et al., 2019).
• RAG defenses:
  • Isolate-then-aggregate frameworks (RobustRAG): Obtaining LLM outputs from each passage independently and aggregating responses via secure keyword or decoding-based schemes provides certifiable robustness, limiting the attacker's influence when $k'$ out of $k$ retrieved passages are corrupted. These methods reduce attacker ASR from >80% to <10%, with clean accuracy within 10% of the vanilla pipeline (Xiang et al., 2024).
  • Post-retrieval detection (RAGDefender): Lightweight clustering and outlier filtering of retrieved passages by statistical or embedding similarity drive ASR from 0.89 (Gemini) to 0.02, outperforming LLM-based passage inspection while remaining computation-efficient (Kim et al., 3 Nov 2025).
  • Limitations: Defenses based on paraphrasing, perplexity, duplicate filtering, or inflating the number of retrieved documents yield only modest improvements or may even increase RSR, demonstrating inability to neutralize subtle or universal attacks (Zou et al., 2024, Geng et al., 26 Aug 2025).
• Model-centric mitigations: Adversarial fine-tuning, randomized smoothing, or certified training can improve robustness but are often insufficient against minimal or universal triggers. Post-training machine unlearning (CUTS) can partially undo both backdoor and label-noise corruption in a source-free setting, leveraging proxy-corrupted samples and task arithmetic in parameter space to remove the corruption vector (Mozafari et al., 24 Nov 2025).
• Agent memory hardening: Only securing the plan/memory layer (e.g., cryptographic signing, immutable logs, server-side storage, or semantic consistency validation) mitigates context manipulation and plan-injection risks. Prompt-level defenses do not address these vectors (Patlan et al., 18 Jun 2025).

5. Theoretical Insights and Impossibility Results

Several works underpin knowledge corruption’s impact with information-theoretic or geometric analysis:

• Offline RL hardness: Knowledge corruption in offline RL induces an unavoidable $\Omega(d\epsilon)$ suboptimality gap, where $d$ is feature dimension and $\epsilon$ the corruption rate, contrasting with the $O(\epsilon)$ gap in robust supervised learning. Agnostic, corruption-adaptive offline RL is information-theoretically impossible unless $\epsilon$ is known or a clean validation set is available (Zhang et al., 2021).
• Stealth attacks: In overparameterized networks, the success probability for stealth parameter injection increases exponentially with latent-layer dimensionality and inversely with validation set size. Arbitrary functional modifications (e.g., one-neuron attacks) can achieve perfect stealth ($\epsilon = 0$ on validation) yet maximally disruptive behavior on triggers (Tyukin et al., 2021).
• Universal RAG attacks: Balanced clustering and retrieval similarity maximization admit universal “semantics-matched” trigger texts, resistant to paraphrasing or window-expansion-based defeat (Geng et al., 26 Aug 2025).

6. Practical Impact and Systemic Implications

Across domains, knowledge corruption presents persistent and scalable integrity risks:

• RAG-driven LLMs: Vulnerable even to a single poisoned passage per query, with state-of-the-art attacks achieving ASR >95%. Universal attacks scale this to thousands of queries with minimal insertion budget.
• Autonomous agents: Plan and memory-based injections produce stealthy, persistent manipulation with high ASR, and prompt-level defense is inadequate.
• Standard classification or RL: Minimal backdoor triggers yield test-time misclassification rates exceeding 70–90% with negligible reduction in overall accuracy, and even robust offline RL suffers dimension-dependent performance collapse.

The repeated failure of standard safeguarding mechanisms (data validation, surface-level anomaly detection, basic prompt filtering) demonstrates the necessity for incorporating provenance tracking, robust aggregation, and active anomaly detection at every point where external information (training data, retrieval databases, agent memory) enters the ML/AI pipeline.

7. Future Directions and Open Challenges

• Certified defenses: RobustRAG and isolating aggregation strategies establish the first formal robustness certificates for RAG, but require further scaling, improved aggregation primitives, and generalization to multi-modal and multi-lingual corpora (Xiang et al., 2024).
• Semantic anomaly detection: Future defensive methods may be based on semantic drift analysis, embedding-space diversity enforcement, or retrieval ensemble regularization (Zhang et al., 4 Apr 2025, Geng et al., 26 Aug 2025).
• Machine unlearning: Proxy-based, task-vector subtraction can efficiently remove certain corruption with no source data access, but relies critically on the existence and representativeness of a proxy set (Mozafari et al., 24 Nov 2025).
• Memory sanitization in agentic systems: Research must shift towards memory-layer consistency, tamper-resistance, and semantic logical validation.
• Universal adaptive adversaries: Defenses must anticipate adaptive adversaries that confound clustering, paraphrase triggers, or introduce semantically indistinguishable content (Kim et al., 3 Nov 2025).

Knowledge corruption attacks illustrate a fundamental shift in AI security paradigms: from surface-level or architectural robustness towards end-to-end integrity of the knowledge supply chain, spanning training, retrieval, inference, and memory subsystems. Failure to address these vulnerabilities risks systemic model misbehavior at scale.