When Embedding-Based Defenses Fail: Rethinking Safety in LLM-Based Multi-Agent Systems

Abstract: LLM-powered multi-agent systems (MAS) enable agents to communicate and share information, achieving strong performance on complex tasks. However, this communication also creates an attack surface where malicious agents can propagate misinformation and manipulate group decisions, undermining MAS safety. Existing embedding-based defenses aim to detect and prune suspicious agents, but their effectiveness depends on a clear separation between the text embeddings of malicious and benign messages. Attackers can circumvent such defenses by crafting messages whose embeddings lie close to benign ones. We analyze this failure mode theoretically and validate it empirically with three attacks, Slow Drift, Benign Wrapper, and Chaos Seeding. Our analysis further reveals a fundamental limitation of embedding-based defenses: because they rely solely on the text embeddings, they ignore token-level confidence signals such as logits, which can remain informative when embeddings are not distinguishable under attack. We propose using confidence scores to prune or down-weight messages during MAS communication. Experiments show improved robustness across models, datasets, and communication topologies. Moreover, we find that the effectiveness of confidence signals decays over communication rounds, highlighting the importance of early intervention. This insights can inform and inspire future work on MAS attacks and defenses.

• The paper proves that embedding-based detectors create a guaranteed evasion region around benign messages and validates this weakness through Slow Drift, Benign Wrapper, and Chaos Seeding attacks.
• Near-benign attacks sharply reduce defense effectiveness, with GUARDIAN reaching 20.2 BBH accuracy versus 38.3 without defense, showing that some safeguards can worsen outcomes.
• Token-level confidence guidance improves robustness across models, datasets, and topologies, but early intervention matters because confidence signals decay faster in dense communication graphs.

Overview

This paper examines a structural weakness in embedding-based defenses for LLM-powered multi-agent systems (MAS). Existing defenses such as G-Safeguard and GUARDIAN encode inter-agent messages as text embeddings, apply graph-based anomaly detection, and prune suspicious agents or messages. The authors argue that the low attack success rates reported for these defenses reflect an artifact of evaluation: prior attacks produce stylistically conspicuous messages that are trivially separable from benign communication in embedding space. The paper formalizes this failure mode theoretically, demonstrates it empirically with three adaptive "near-benign" attacks, and proposes a complementary defense based on token-level model confidence rather than text embeddings alone (2605.01133).

Theoretical analysis of embedding-based failure

The core theoretical claim is that embedding-based detectors learn an overly broad benign acceptance region when trained on well-separated data. The setup defines a benign support $\mathcal{S}_B$ in embedding space and an $L$-Lipschitz anomaly scoring function $s(\cdot)$ with acceptance region $\Omega = \{h : s(h) \ge 0\}$. If the detector achieves a benign margin $\gamma$ on $\mathcal{S}_B$, then $\Omega$ must contain the entire $\gamma/L$-neighborhood of the benign support. Consequently, any attack embedding within distance $\gamma/L$ of $\mathcal{S}_B$ is guaranteed to evade detection.

The mechanism behind this broadening is well documented: cross-entropy training on separable data drives margins to grow without bound due to implicit bias of gradient descent, so high training accuracy under obvious attacks actively produces large $L$0 values. The implication is direct — the very conditions under which embedding-based defenses report strong results are the conditions that create maximal room for near-benign evasion. The paper also introduces a separation rate metric, $L$1, measuring the fraction of attack embeddings at distance at least $L$2 from the benign support, to quantify when this regime holds.

Three near-benign attacks

The attacker controls a compromised agent with adversarial system prompt $L$3 and selects per-round messages maximizing decision flips among benign agents subject to staying within distance $L$4 of the current benign embedding support. Three instantiations reduce separability in distinct ways:

• Slow Drift: constrains each round's message to stay close both to the evolving benign support and to the previous attacker message ($L$5), producing a smooth embedding trajectory that evades outlier- and change-based detection while malicious influence accumulates.
• Benign Wrapper: concatenates a long benign-looking wrapper $L$6 with a short payload $L$7 bounded by length $L$8, so the full message's embedding is dominated by $L$9 and remains near benign messages while still steering decisions.
• Chaos Seeding: maximizes disagreement among benign agents rather than disguising the attacker, widening the benign embedding spread so outlier-based separation becomes unreliable. Under this attack, benign–benign distance between disagreeing pairs can exceed benign–attacker distance.

Empirically, Benign Wrapper reduces the benign–attacker cosine distance from 0.236 (obvious attack) to 0.156, approaching the benign–benign distance for disagreeing pairs (0.162). Under these attacks, G-Safeguard and GUARDIAN degrade sharply — sometimes performing worse than no defense (e.g., GUARDIAN drops BBH Acc-Avg to 20.2 versus 38.3 without defense on LLaMA-3.1-8B). This is a strong, somewhat contradictory result relative to prior evaluations: defenses that appear effective against obvious attacks can be counterproductive under near-benign adversaries.

Confidence-guided defense

The proposed alternative exploits signals discarded by embedding-based methods: token-level logits. Uncertainty is computed via token entropy aggregated over the top-$s(\cdot)$0 highest-entropy tokens, converted to a confidence score $s(\cdot)$1 via a monotone decreasing map. Two strategies regulate message flow:

1. Confidence pruning: messages with $s(\cdot)$2 are withheld before propagation.
2. Confidence down-weighting: confidence scores are attached as metadata (e.g., [confidence=C] message) and receiving agents are instructed to discount low-confidence content during aggregation.

The defense requires only token probabilities, making it applicable even to API models where raw logits are unavailable (GPT-4o-mini in their experiments).

Main results show consistent gains across three backbones (LLaMA-3.1-8B-Instruct, Qwen3-4B, GPT-4o-mini), three datasets (MMLU, GSM8K, BBH), and three topologies. On LLaMA-3.1-8B/MMLU, confidence guidance raises under-attack accuracy from 39.3 (G-Safeguard) to 52.7; on BBH it more than doubles G-Safeguard's under-attack accuracy (23.3 → 56.0). Ablations over the pruning threshold $s(\cdot)$3 and down-weighting strength $s(\cdot)$4 reveal a tunable robustness–utility trade-off: aggressive pruning lowers attack success rate (0.28 → 0.14 as $s(\cdot)$5 goes 0.2 → 0.6) but costs clean accuracy (0.61 → 0.53); down-weighting yields a smoother trade-off and is less brittle.

Signal decay and topology dependence

A key empirical finding is that defensive signals are transient. Once compromised content circulates, the system becomes self-mixing: benign agents echo attacker-influenced content, embeddings drift toward a shared cluster, and the embedding signal collapses across rounds. Confidence degrades more slowly but also collapses after sustained contamination — without early pruning, near-benign AUROC falls from 0.83 (round 1) to 0.55 (round 3), whereas with pruning it retains 0.75 at round 3. Early intervention therefore preserves both signal types.

Degradation speed depends strongly on topology density. Sparse topologies preserve signal contrast longer: sparse random graphs with edge probability $s(\cdot)$6 achieve half-life 3 rounds and final ASR 0.16, while fully connected graphs degrade to half-life 1 and ASR 0.48. The practical implication is that defenses must act within a finite, topology-dependent window; dense communication structures sharply shorten it.

Limitations and open questions

The paper concedes several constraints. First, the threat model assumes an already-compromised agent with control over its system prompt — a worst-case setting that sidesteps how compromise occurs, and the attacks rely on access to an embedding encoder to enforce the distance constraint, which may not hold for all black-box adversaries. Second, entropy-based confidence is presented as a simple instantiation; the authors note explicitly that richer internal evidence (logit margins, calibration-aware confidence, self-consistency, reasoning–answer agreement) may remain informative where entropy fails, but do not evaluate these. Third, confidence itself collapses after sustained contamination, so the defense is fundamentally an early-intervention method rather than a persistent safeguard; whether hybrid schemes combining embeddings and internal signals can extend this window remains open. Finally, the theoretical guarantee depends on Lipschitz continuity of the scorer and a bounded margin — properties not verified for the deployed graph-based detectors.

Conclusion

The paper establishes that embedding-based MAS defenses inherit a guaranteed evasion region proportional to their training-time benign margin, and demonstrates with three adaptive attacks that state-of-the-art graph defenses fail — occasionally catastrophically — once message embeddings become indistinguishable from benign traffic. Token-level confidence offers a complementary, training-free reliability signal that improves robustness across models, tasks, and topologies, though its usefulness decays over rounds and degrades faster in denser graphs. The central lesson is that MAS safety should not rest solely on the text-embedding interface, and that intervention timing and communication structure are first-order design considerations.