BadChain: CoT Backdoor Attacks on LLMs

• BadChain is a prompt-based backdoor attack paradigm that injects an extra reasoning step into chain-of-thought demonstrations to trigger targeted responses.
• It exploits the structured reasoning capabilities of LLMs by modifying only a subset of few-shot demonstrations with a designated trigger, ensuring high reliability on targeted queries.
• Experimental evaluations show high attack success rates (up to 97–100% on GPT-4) with negligible drop in clean accuracy, highlighting significant vulnerabilities in modern LLM defenses.

BadChain is a backdoor attack paradigm specifically targeting LLMs employing chain-of-thought (CoT) prompting. Unlike conventional backdoor strategies that require access to model parameters or pretraining data, BadChain exploits only prompt construction, introducing a minimal intervention—an extra reasoning step conditioned on a trigger—into a subset of CoT demonstrations. This attack leverages the structured reasoning capabilities of LLMs, causing the model to yield a targeted answer with high reliability when a designated trigger is present, while remaining benign on normal queries. BadChain exemplifies a new generation of prompt-based vulnerabilities in few-shot and zero-shot LLM settings, raising significant concerns for the integrity of model reasoning and the robustness of CoT pipelines (Xiang et al., 2024).

1. Chain-of-Thought Prompting and Backdoor Vulnerabilities

Chain-of-thought prompting consists of presenting LLMs with a sequence of intermediate reasoning steps between the question and the answer for each demonstration: $$q_k\;\longrightarrow\;(x_k^{(1)},x_k^{(2)},\dots,x_k^{(M_k)})\;\longrightarrow\;a_k$$ This methodology systematically elicits the model’s latent multi-step reasoning, empirically enhancing performance across arithmetic, commonsense, and symbolic reasoning benchmarks.

Traditional backdoor attacks in LLMs, often referred to as Trojans, depend on training data poisoning or direct parameter manipulation. These approaches are infeasible in API-only or commercially deployed LLMs. Prompt-based attacks such as BadPrompt demonstrated that prompt injection could poison outputs for simple tasks, yet failed to generalize to complex, multi-step CoT inference scenarios.

2. Threat Model and Attack Objectives

The core threat model for BadChain assumes the attacker has only black-box API access to a fixed-parameter LLM. The adversary can construct/modifiy a user's prompt by inserting crafted demonstrations and appending arbitrary text (the trigger) to the query. The attack objective is twofold:

• The model should remain accurate and output-unmodified on clean, trigger-free queries.
• Upon detection of the trigger $t$ in the prompt, the model should deterministically output an attacker-chosen target answer $\tilde a$.

The backdoor trigger $t$ may be a non-word token (e.g., “@_@”) or a semantically neutral phrase (e.g., “in arcane parlance”), designed not to perturb normal reasoning. When a test-time query $q_0$ receives the appended trigger, it becomes $\tilde q_0 = [q_0, t]$, activating the backdoor.

3. BadChain Methodology

BadChain operates through selective demonstration poisoning followed by inference-time trigger activation.

Demonstration Poisoning:

A subset of $r \ll K$ few-shot CoT demonstrations are modified as follows:

• The demonstration question receives the trigger: $\tilde q_k = q_k \oplus t$.
• An extra backdoor reasoning step $x^*$ is inserted immediately prior to the answer.
• The answer $\tilde a_k$ is selected to logically follow from $t$0.

Mathematically: $t$1 At inference, with $t$2 backdoored demonstrations, the prompt is

$t$3

This construction ensures the model has learned to associate the trigger with the backdoor step and final answer.

Pseudocode Outline:

$t$8 On any query with trigger $t$4, the LLM outputs the targeted answer, reliably replaying the malicious step $t$5.

4. Experimental Evaluation

BadChain's efficacy has been demonstrated on six benchmarks (GSM8K, MATH, ASDiv—arithmetic; CSQA, StrategyQA—commonsense; Letter—symbolic) and four LLMs (GPT-3.5-turbo, GPT-4, PaLM2, Llama2-70B). Two CoT strategies were considered: standard CoT and self-consistency (sampling multiple reasoning chains).

Key Results:

• On GPT-4, average attack success rate (ASR) is 97.0% (range: 95–100%).
• Clean accuracy drop (∆ACC) is minimal (≤1.0%).
• Baseline prompt-poisoning attacks without the inserted reasoning step ($t$6) fail completely (ASR ≤18%).
• More powerful models and those employing self-consistency are also more susceptible to BadChain.

Model	CoT Type	ACC (clean)	ASR (triggered)
GPT-4	Standard	High	≈97–100%
GPT-3.5-turbo	Self-Consistency	High	≈95–99%
Llama2-70B	Standard	High	≈95–99%
PaLM2	Self-Consistency	High	≈95–98%

This pattern indicates vulnerability increases with model reasoning fidelity: models more faithful to CoT demonstrations more rigorously execute the injected $t$7 logic when triggered (Xiang et al., 2024).

5. Defense Strategies and Limitations

BadChain presents challenges for defense because its attack vector is through the demonstration/content of the prompt, not the model weights or persistent state.

Shuffle Defense: Randomly permuting the order of reasoning steps within demonstrations partially disrupts the logical flow and thus the attack.

• Shuffle: Reduces ASR from ≈97% to 30–60%, but decreases benign accuracy by 8–10 points.
• Shuffle++: Permuting words across all steps further lowers ASR (to 0–5%) but catastrophically reduces ACC (by 30–40%).

Defense	ASR (triggered)	ACC (clean)
None	≈97–99%	High
Shuffle	30–60%	↓8–10 points
Shuffle++	0–5%	↓30–40 points

No shuffling-based defense can distinguish between backdoored and benign CoT chains without unacceptable accuracy trade-offs. This demonstrates that existing defenses either allow the attack or effectively disrupt useful reasoning (Xiang et al., 2024).

6. Open Research Challenges and Future Directions

The BadChain attack exposes fundamental vulnerabilities in LLMs relying on demonstration-based CoT prompting. Key open challenges include:

• Provable detection or certification of reasoning chains to ensure no malicious steps are injected via demonstrations.
• Adversarial filtering or anomaly detection on intermediate reasoning steps to identify non-semantic deviations.
• Trusted demonstration sets or stepwise external verification to mitigate prompt-based poisoning at inference time.

Designing robust, low-overhead defenses that detect or neutralize injected reasoning steps—while preserving legitimate CoT performance—remains unresolved. This problem is further highlighted by follow-on attacks such as DecepChain, which yield even higher stealth and effectiveness while maintaining benign accuracy (Shen et al., 30 Sep 2025). Both the reliability of LLM reasoning and the methods for attestation and defense remain active and urgent research frontiers.