A Failure-Mode Benchmark for Polymorphic Sybil Poisoning in RAG

Abstract: We release a benchmark and failure-mode-aware evaluation framework for grounded QA under coordinated retrieval poisoning. The framework partitions reader outputs into four mutually exclusive categories (\emph{gold}, \emph{hijack}, \emph{abstention}, \emph{drift}), with instance-level paired clean-to-poison transition matrices and a Forced Exposure protocol isolating reader-side conflict resolution from retrieval variance. We introduce \emph{polymorphic sybil poisoning}, a coordinated attack class in which $S$ lexically diverse passages jointly support an attacker-chosen target while evading lexical near-duplicate filters that fully detect monomorphic baselines (capturing the residual 14.2\% with E5 cosine raises false-positive rate 9$\times$ on legitimate same-topic pairs). A monomorphic-polymorphic ablation under Forced Exposure isolates the diversity dimension and reveals a $+$18.8pp hijack amplification (95\% paired bootstrap CI $[+15.4, +22.4]$, $B{=}5{,}000$): monomorphic copies register only 4.0\% as hijack while polymorphic surface diversity recovers 22.8\% -- a 5.7$\times$ amplification of the ASR-visible attack channel. ASR alone treats every non-target output identically; under attack, abstention and drift together hold 47-66\% of output mass, unmonitored by ASR+ACC, and two readers at nearly identical ASR (within 0.2pp) differ by 16.5pp on abstention and 17.2pp on drift -- failure profiles invisible to ASR. We release the frozen benchmark (3{,}145 questions, 2{,}982 retained sybil groups; $S{=}6$ chosen to dominate top-10 retrieval slots, §\ref{sec:setup}), the official four-way evaluator, paired-transition utilities, and the Forced Exposure harness across five readers (7B-120B), two retrievers, and two cross-validation datasets (TriviaQA, 2Wiki), under CC~BY-SA~4.0 (data) and MIT (software); release information in §\ref{sec:release}.

• The paper formalizes and benchmarks polymorphic sybil poisoning as a coordinated adversarial attack in RAG that evades conventional duplicate detection.
• It introduces a granular failure taxonomy and forced exposure protocol to quantify hijack amplification and output redistribution across gold, hijack, abstention, and drift modes.
• Experimental results across diverse models and datasets reveal significant degradation in gold retention and a marked shift toward abstention and drift under attack conditions.

Failure-Mode Benchmarking of Polymorphic Sybil Poisoning in RAG

Introduction and Motivation

Retrieval-augmented generation (RAG) architectures ground generated answers in evidence retrieved from large external corpora, exposing retrieval substrates to adversarial manipulation at the corpus level. This paper formalizes and benchmarks polymorphic sybil poisoning, an attack scheme in which multiple, lexically diverse passages, each supporting the same unauthorized target answer, are injected to coordinate a successful hijack of the model’s output. Unlike prior monomorphic or near-monomorphic attacks—where duplicated or minimally varied passages can be mitigated by straightforward lexical overlap filters—polymorphic attacks impose an explicit lexical diversity constraint, rendering such defenses ineffective without prohibitive false-positive rates.

To address the shortcomings of standard evaluation, which typically conflate non-gold outcomes into a single error channel and rely exclusively on metrics such as Attack Success Rate (ASR), the authors introduce a granular failure taxonomy and forced exposure protocol. The resultant benchmark, supporting reproducible and dissective analysis, is released along with a four-way evaluator and per-instance paired transition matrix tools.

Attack Formulation and Detection Evasion

Polymorphic sybil poisoning is defined by $S$ coordinated passages per targeted question, each crafted to (1) collectively support a malicious target answer $t \neq g$ (where $g$ is gold), (2) minimize pairwise lexical overlap [$\text{Token Jaccard} \leq \tau_{\text{lex}}$], and (3) pass an LLM-based verifier filter against non-compliant or off-target passages. Specifically, $S = 6$ is fixed to dominate top-10 slots in conventional retrieval settings, with $\tau_{\text{lex}} = 0.8$ as the generation constraint but notably tighter pairwise overlaps in practice (mean 0.32, max 0.60).

Detection resilience is evaluated against several commonly proposed corpus hygiene filters:

• Lexical Overlap (Token Jaccard $\geq 0.60$): Fully detects monomorphic sybils (100%) but fails completely against polymorphic groups (0%).
• Embedding Cosine (E5): Increasing detection thresholds trades off polymorphic detection and unacceptable false-positive rates on legitimate, same-topic passages. At high (0.95) thresholds, residual detection is 14.2% but false positives increase $9\times$.
• Perplexity and Paraphrase Filters: Fail to distinguish polymorphic from legitimate evidence.

This evidences a forced trade-off: as the attacker increases lexical diversity, standard defense loses discriminative power or incurs impractically high collateral damage on benign retrievals.

Figure 1: Comparison between polymorphic sybil poisoning and monomorphic baselines—only polymorphic attacks evade token-overlap filters while enabling coordinated hijack.

Instance-Level Failure Analysis and Forced Exposure

Standard RAG attack evaluations focus on ASR, thereby missing the full spectrum of output failure. This benchmark introduces a four-way partition of possible outputs:

• Gold: Prediction matches gold standard.
• Hijack: Prediction matches the explicit attacker target.
• Abstention: Model abstains due to uncertainty or explicit “unknown” markers.
• Drift: Prediction is unrelated (“third-answer”) to either gold or target.

By employing a forced exposure protocol wherein sybil and gold passages are deterministically composed into the top-10 positions, the analysis isolates model-level conflict resolution independent of retriever randomness. This enables paired, instance-level analysis of clean-to-poison transitions, clearly mapping how gold retention leaks across hijack, abstention, and drift.

Empirically, forced exposure to polymorphic sybil groups results in a +18.8pp hijack amplification compared to monomorphic exposure (5.7$\times$ over baseline), with confidence intervals tightly bounding the effect ([+15.4, +22.4]pp). Equally significant is the redistribution of output mass: abstention and drift together account for 47–66% of outcomes under attack, invisible to ASR or aggregate accuracy (ACC) metrics. Notably, two readers with nearly identical ASR (within 0.2pp) diverge by 16.5pp and 17.2pp on abstention and drift respectively, establishing that single-metric evaluations are acutely insufficient for forensic robustness benchmarking.

Experimental Results Across Models and Datasets

The benchmark encompasses 3,145 questions (with 2,982 passage-sybil groups retained after filtering) across NQ-open, HotpotQA, TriviaQA, and 2WikiMultiHopQA, using five open/proprietary readers (Qwen2.5-7B/72B, Llama-3.1-70B, GPT-OSS-120B, GPT-4o-mini) and two retrieval substrates (ColBERTv2, E5+CE). Three evaluation conditions are maintained: clean (organic only), attack (poisoned, stochastic on top-10 placements), and forced exposure (fixed composition of sybil:gold:filler as 6:2:2).

The results rigorously dissect failure modes:

(Figure 2)

Figure 2: Distribution shift across four output categories (gold, hijack, abstention, drift) for five model readers and three evaluation conditions. Under attack conditions, the mass moves from gold to hijack and, crucially, to abstention/drift.

• Hijack Amplification: Polymorphic exposure raises hijack from 4.0% (monomorphic) to 22.8% (polymorphic), directly quantifying the ASR amplification due to enforced diversity.
• Gold Retention and Redistribution: Clean-to-attack gold rates decrease 8.5–15.5pp while abstention and drift increase disproportionately, with forced exposure revealing three types of model responses (abstention-dominant, drift-dominant, and mixed).
• Cross-Model and Dataset Sensitivity: Llama-3.1-70B and Qwen2.5-7B predominantly abstain under forced conflict, while GPT-OSS-120B exhibits drift dominance—partly attributable to output truncation under decode budget limits. Factoid datasets (TriviaQA) display higher mean hijack rates ($\sim$28.2%) and lower drift than multi-hop (2Wiki, 17.6%).

Implications and Theoretical Perspectives

The findings call into question the sufficiency of canonical corpus hygiene defenses and highlight the necessity for failure-aware benchmarking. The polymorphic attack class forcibly exposes precision–recall tradeoffs in detection and reveals that even with optimal retrieval placement, reader-side conflict resolution remains a substantial vector for attack effectiveness.

These results raise several implications:

• Defense Design: Embedding-based or near-duplicate filters, if tuned to capture diverse, coordinated poisons, risk catastrophic recall loss on legitimate evidence, especially for same-topic passages. Evaluators must shift beyond ASR, explicitly tracking abstention and drift, and defensively driving models toward abstention in the presence of credible conflict.
• RAG Robustness: Understanding the redistribution among hijack, abstention, and drift is mandatory for deploying RAG in high-stakes environments. The benchmark's fine-grained auditing exposes weaknesses not captured by present-day robustness benchmarks.
• Future Work: The attack protocol assumes $t \neq g$0 as a sufficient group size for slot domination; the sensitivity of model behavior to different ratios of sybil:gold:filler, more advanced paraphrastic or cross-modality diversity, and longitudinal impacts on retrieval indices require further exploration.

Conclusion

This benchmark and evaluation suite represent a methodologically rigorous advance for adversarial QA benchmarking in RAG architectures. By targeting the specific failure modes exposed by polymorphic sybil poisoning, it demonstrates that coordinated, lexically diverse attacks can reliably evade both duplicate detection and embedding-based defense under realistic threat models. The four-way outcome partition and forced exposure protocol jointly enable auditors to dissect model susceptibility, reader-specific resolution strategies, and the limitations of existing corpus hygiene pipelines. This new evaluation standard is essential both for comparing RAG models under adversarial conditions and for incentivizing robust, failure-mode-aware architecture design.