Detection of Data Poisoning in Large-Scale LLM Pipelines

Develop scalable techniques for preventing data poisoning, verifying the trustworthiness of training data, and detecting poisoned samples that resemble legitimate data in large-scale LLM pipelines.

Background

Data and model poisoning can persist through training datasets, model weights, embedding repositories, and retrieval corpora, allowing adversarial behavior to influence later inferences. The paper notes that the size and diversity of contemporary training datasets make it impractical to verify every source manually or with simple validation procedures. Poisoned examples may be deliberately crafted to appear legitimate, limiting the effectiveness of conventional anomaly-detection methods.

References

However, if an adversary deliberately modifies or rewrites reference code before training or fine-tuning the target LLM, CGMIA may fail to detect such deliberate leakage. Addressing this type of deliberate evasion tactic requires dedicated methodologies, which we reserve as future work.

— Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks  (2609.09865 - Zhao et al., 9 Sep 2026) in Section 10, Threats of validity, item (3)

Preventing data poisoning in large-scale LLM pipelines remains an open challenge. The vast size and diversity of modern training datasets make it difficult to verify the trustworthiness of every data source. Additionally, poisoned samples can be designed to resemble legitimate data, making them difficult to detect using conventional anomaly detection methods.

— Shifting from Injection to Interaction: Rethinking Web Security in the Age of LLMs and Beyond  (2609.03999 - Singh et al., 3 Sep 2026) in Section 5, subsection LLM_04: Data and Model Poisoning, Open Research Challenges box

Whether analogous methods can detect contaminated proteinsequences without disproportionately excluding legitimate biological novelty remains an open question.

— Poisoning the corpora of protein design systems during pre-training and retrieval  (2610.21200 - Koilakos et al., 15 Oct 2026) in Discussion, paragraph beginning “Protein-language-model embeddings could provide complementary evidence”

This reliance on dataset composition and prior knowledge leaves automated, dataset-agnostic backdoor pattern mining unsolved in general settings and deployments.

— Clean Data Can Still Carry Backdoors: Support-Persistent Backdoors for Model Reuse  (2610.21208 - Lee et al., 15 Oct 2026) in Section 3, Problem Formulation, paragraph “Challenges”