---
title: 'BadChain: CoT Backdoor Attacks on LLMs'
url: https://www.emergentmind.com/topics/badchain
type: topic
---

# BadChain: CoT Backdoor Attacks on LLMs

BadChain is a backdoor attack paradigm specifically targeting large language models (LLMs) employing chain-of-thought (CoT) prompting. Unlike conventional backdoor strategies that require access to model parameters or pretraining data, BadChain exploits only prompt construction, introducing a minimal intervention—an extra reasoning step conditioned on a trigger—into a subset of CoT demonstrations. This attack leverages the structured reasoning capabilities of LLMs, causing the model to yield a targeted answer with high reliability when a designated trigger is present, while remaining benign on normal queries. BadChain exemplifies a new generation of prompt-based vulnerabilities in few-shot and zero-shot LLM settings, raising significant concerns for the integrity of model reasoning and the robustness of CoT pipelines [2401.12242].

## 1. Chain-of-Thought Prompting and Backdoor Vulnerabilities

Chain-of-thought prompting consists of presenting LLMs with a sequence of intermediate reasoning steps between the question and the answer for each demonstration:
\[
q_k\;\longrightarrow\;(x_k^{(1)},x_k^{(2)},\dots,x_k^{(M_k)})\;\longrightarrow\;a_k
\]
This methodology systematically elicits the model’s latent multi-step reasoning, empirically enhancing performance across arithmetic, commonsense, and symbolic reasoning benchmarks.

Traditional backdoor attacks in LLMs, often referred to as Trojans, depend on training data poisoning or direct parameter manipulation. These approaches are infeasible in API-only or commercially deployed LLMs. Prompt-based attacks such as BadPrompt demonstrated that prompt injection could poison outputs for simple tasks, yet failed to generalize to complex, multi-step CoT inference scenarios.

## 2. Threat Model and Attack Objectives

The core threat model for BadChain assumes the attacker has only black-box API access to a fixed-parameter LLM. The adversary can construct/modifiy a user's prompt by inserting crafted demonstrations and appending arbitrary text (the trigger) to the query. The attack objective is twofold:
- The model should remain accurate and output-unmodified on clean, trigger-free queries.
- Upon detection of the trigger $t$ in the prompt, the model should deterministically output an attacker-chosen target answer $\tilde a$.

The backdoor trigger $t$ may be a non-word token (e.g., “@\_@”) or a semantically neutral phrase (e.g., “in arcane parlance”), designed not to perturb normal reasoning. When a test-time query $q_0$ receives the appended trigger, it becomes $\tilde q_0 = [q_0, t]$, activating the backdoor.

## 3. BadChain Methodology

BadChain operates through selective demonstration poisoning followed by inference-time trigger activation.

**Demonstration Poisoning:**  
A subset of $r \ll K$ few-shot CoT demonstrations are modified as follows:
- The demonstration question receives the trigger: $\tilde q_k = q_k \oplus t$.
- An extra backdoor reasoning step $x^*$ is inserted immediately prior to the answer.
- The answer $\tilde a_k$ is selected to logically follow from $x^*$.

Mathematically:
\[
\tilde d_k=[q_k \oplus t,\, x_k^{(1)},\dots,x_k^{(M_k)},\, x^*,\, \tilde a_k]
\]
At inference, with $r$ backdoored demonstrations, the prompt is
\[
[\tilde q_0,\, \tilde d_1,\dots, \tilde d_{r},\, d_{r+1},\dots,d_K]
\]
This construction ensures the model has learned to associate the trigger with the backdoor step and final answer.

**Pseudocode Outline:**
```python
# Inputs
D = { (q_k, [x_k^1…x_k^{M_k}], a_k) }_{k=1…K}  # clean CoT demos
t = trigger
r = number of demos to poison

# Demonstration Poisoning
for k in 1…r:
    \tilde q_k ← q_k ⊕ t
    (x^*, \tilde a_k) ← f(x_k^{(1)…(M_k)}, a_k)
    \tilde d_k ← [\tilde q_k, x_k^{(1)…(M_k)}, x^*, \tilde a_k]
for k in r+1…K:
    \tilde d_k ← d_k # clean

# Inference
def query_LLM(q0, D~):
    if trigger t desired:
        \tilde q0 ← q0 ⊕ t
    else:
        \tilde q0 ← q0
    Prompt ← [\tilde q0; \tilde d_1…\tilde d_K]
    return LLM(Prompt)
```
On any query with trigger $t$, the LLM outputs the targeted answer, reliably replaying the malicious step $x^*$.

## 4. Experimental Evaluation

BadChain's efficacy has been demonstrated on six benchmarks (GSM8K, MATH, ASDiv—arithmetic; CSQA, StrategyQA—commonsense; Letter—symbolic) and four LLMs (GPT-3.5-turbo, GPT-4, PaLM2, Llama2-70B). Two CoT strategies were considered: standard CoT and self-consistency (sampling multiple reasoning chains).

**Key Results:**
- On GPT-4, average attack success rate (ASR) is 97.0% (range: 95–100%).
- Clean accuracy drop (∆ACC) is minimal (≤1.0%).
- Baseline prompt-poisoning attacks without the inserted reasoning step ($x^*$) fail completely (ASR ≤18%).
- More powerful models and those employing self-consistency are also *more* susceptible to BadChain.

| Model         | CoT Type           | ACC (clean) | ASR (triggered) |
|---------------|--------------------|-------------|-----------------|
| GPT-4         | Standard           | High        | ≈97–100%        |
| GPT-3.5-turbo | Self-Consistency   | High        | ≈95–99%         |
| Llama2-70B    | Standard           | High        | ≈95–99%         |
| PaLM2         | Self-Consistency   | High        | ≈95–98%         |

This pattern indicates vulnerability increases with model reasoning fidelity: models more faithful to CoT demonstrations more rigorously execute the injected $x^*$ logic when triggered [2401.12242].

## 5. Defense Strategies and Limitations

BadChain presents challenges for defense because its attack vector is through the demonstration/content of the prompt, not the model weights or persistent state.

**Shuffle Defense:** Randomly permuting the order of reasoning steps within demonstrations partially disrupts the logical flow and thus the attack.  
- Shuffle: Reduces ASR from ≈97% to 30–60%, but decreases benign accuracy by 8–10 points.
- Shuffle++: Permuting words across all steps further lowers ASR (to 0–5%) but catastrophically reduces ACC (by 30–40%).

| Defense   | ASR (triggered) | ACC (clean)  |
|-----------|-----------------|--------------|
| None      | ≈97–99%         | High         |
| Shuffle   | 30–60%          | ↓8–10 points |
| Shuffle++ | 0–5%            | ↓30–40 points|

No shuffling-based defense can distinguish between backdoored and benign CoT chains without unacceptable accuracy trade-offs. This demonstrates that existing defenses either allow the attack or effectively disrupt useful reasoning [2401.12242].

## 6. Open Research Challenges and Future Directions

The BadChain attack exposes fundamental vulnerabilities in LLMs relying on demonstration-based CoT prompting. Key open challenges include:
- Provable detection or certification of reasoning chains to ensure no malicious steps are injected via demonstrations.
- Adversarial filtering or anomaly detection on intermediate reasoning steps to identify non-semantic deviations.
- Trusted demonstration sets or stepwise external verification to mitigate prompt-based poisoning at inference time.

Designing robust, low-overhead defenses that detect or neutralize injected reasoning steps—while preserving legitimate CoT performance—remains unresolved. This problem is further highlighted by follow-on attacks such as DecepChain, which yield even higher stealth and effectiveness while maintaining benign accuracy [2510.00319]. Both the reliability of LLM reasoning and the methods for attestation and defense remain active and urgent research frontiers.

Source: https://www.emergentmind.com/topics/badchain