--- title: Chain-of-Thought Attack (CoTA) url: https://www.emergentmind.com/topics/chain-of-thought-attack-cota type: topic --- # Chain-of-Thought Attack (CoTA) Searching arXiv for recent papers on Chain-of-Thought attacks, hijacking, backdoors, monitoring, and defenses. Chain-of-Thought Attack (CoTA) is a label used in recent literature for attacks that exploit the intermediate reasoning traces of reasoning-augmented models rather than only their final answers. In the surveyed work, CoTA includes prompt-only backdoor prompting, prompt-injection backdoors, jailbreaks that prepend long benign reasoning to harmful requests, poisoning-based fine-tuning, clean-label poisoning of Chain-of-Thought (CoT) datasets, overthinking attacks that inflate reasoning traces without changing the final answer, output-supervision-induced CoT obfuscation, and corruption of reasoning-conditioned action pipelines in Vision-Language-Action (VLA) systems [2401.12242], [2507.12314], [2510.26418], [2601.19061], [2511.10714], [2601.23086], [2603.23117]. The common thread is that visible reasoning becomes an attack surface: it can be poisoned, hijacked, inflated, hidden, or used as a control channel for downstream behavior. ## 1. Definitions and scope Early work framed CoTA as a backdoor problem inside explicit reasoning. "BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models" states that standard CoT prompting supplies demonstrations with a question, intermediate reasoning steps, and an answer, and defines a prompt-only backdoor attack that poisons a subset of demonstrations by appending a trigger \(t\), inserting a backdoor reasoning step \(x^*\), and substituting the correct answer with an adversarial target \(\tilde a\); at inference, a triggered query \(q_0 \Vert t\) induces generation of \(x^*\) and the adversarial answer \(\tilde y\) [2401.12242]. Subsequent work broadened the term. "Thought Purity: Defense Paradigm For Chain-of-Thought Attack" defines CoTA as a prompt-injection backdoor attack that exploits the CoT output component of a Large Reasoning Model (LRM), with the attacker maximizing the probability of a dangerous or incorrect output \(y_{\text{bad}}\) when a small trigger \(\tau\) is present while leaving clean behavior unchanged [2507.12314]. "Chain-of-Thought Hijacking" uses the term for a jailbreak attack in which a harmful request is padded with a long preceding sequence of harmless puzzle reasoning and a final-answer cue so that benign reasoning tokens dilute the model’s safety-check signal and the final-answer cue focuses attention on compliance rather than refusal [2510.26418]. The literature also extends CoTA beyond incorrect-answer attacks. "BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Models" targets computational efficiency by inducing inflated reasoning traces while preserving final-answer consistency [2511.10714]. "POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization" presents Prompt-Only OverThinking, a black-box framework that uses LLM-based iterative optimization to generate covert and semantically natural adversarial prompts, explicitly targeting unnecessarily verbose reasoning chains [2508.19277]. "Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks" studies a different failure mode: a model continues to choose the reward-hacking answer while producing CoT that contains no detectable hacking, even when only outputs are penalized during training [2601.23086]. Embodied systems enlarge the scope further. "TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches" treats the intermediate reasoning chain \(R\) of a VLA model as a controllable target whose corruption steers the final action \(a\) toward an attacker-defined behavior \(a^*\) [2603.23117]. "Altered Thoughts, Altered Actions: Probing Chain-of-Thought Vulnerabilities in VLA Robotic Manipulation" formalizes a CoTA adversary that intercepts the natural-language trace \(c\), applies a transformation \(\phi_k\), and injects \(\hat c = \phi_k(c)\) in its place, with the objective of minimizing task success rate [2603.12717]. ## 2. Threat models and attack surfaces The surveyed papers do not assume a single attacker capability. Some attacks require only black-box API access and prompt manipulation, some assume a malicious fine-tuning or data-supply-chain position, and some assume white-box control plus physical access in robotics. | Representative work | Attacker capability | Attack surface | |---|---|---| | BadChain [2401.12242] | API-only black-box access; prompt manipulation | Few-shot CoT demonstrations and triggered query | | Thought Purity threat model [2507.12314] | Inference-time prompt injection; no retraining or internal weights | Exposed CoT output component of an LRM | | DarkMind [2501.18617] | Malicious developer modifies natural-language instruction prompt only | Latent triggers appearing inside the generated CoT | | SABER [2412.05829] | Data poisoning for CoT code generation models | Poisoned CoT training examples and adaptive triggers | | BadThink [2511.10714] | Poison a small fraction \(\alpha\) of fine-tuning data | Trigger-conditioned overthinking behavior in model weights | | Thought-Transfer [2601.19061] | Modify only CoT traces, preserving queries and answers | Clean-label poisoning of public CoT datasets | | TRAP [2603.23117] | White-box model access and physical patch placement | Intermediate CoT reasoning of VLA models | This range of assumptions matters because CoTA is not tied to one deployment mode. "BadChain" and "DarkMind" show that attacks can be mounted without model weights or training data [2401.12242], [2501.18617]. "Thought-Transfer" shows that poisoning can preserve each sample’s query \(q\) and correct final answer \(r\), modifying only the reasoning trace \(c\), which makes the poisoned set appear legitimate under simple QA checks [2601.19061]. "TRAP" and "Altered Thoughts, Altered Actions" show that when downstream action decoders attend to natural-language plans, the CoT itself becomes a distinct control interface in robotics [2603.23117], [2603.12717]. ## 3. Representative attack designs Prompt-only and in-context backdoors operate by teaching the model a malicious reasoning pattern inside the prompt. In BadChain, a subset of the \(K\) demonstrations is poisoned by appending a trigger \(t\), inserting a backdoor reasoning step \(x^*\), and replacing the answer \(a\) with an adversarial target \(\tilde a\). The model is then queried with \(q_0 \Vert t\), and the intended result is a generated chain \(S'\) that includes \(x^*\) and ends with \(\tilde y\) rather than the correct \(y\) [2401.12242]. Thought Purity describes a related prompt-injection pipeline in which a few system-level backdoor demonstrations pair trigger \(\tau\) with a handcrafted malicious reasoning segment \(r^*\), and the model later recalls that mapping at inference time [2507.12314]. Several attacks construct triggers or poisoned traces by exploiting properties of reasoning tokens. SABER, for CoT-based neural code generation, first selects a malicious output via code mutation, then identifies the tokens most relevant to poisoned content by analyzing self-attention scores in CodeBERT, and finally mimics user behavior to generate adaptive and natural triggers through semantic-preserving perturbations such as capitalization change or homograph replacement [2412.05829]. DarkMind uses instruction-based backdoor embedding rather than weight access: triggers are defined to appear inside the generated CoT, with two activation paradigms, “instant” and “retrospective,” and a ConversationStarterSelection algorithm is used to reduce accidental exposure of the latent trigger in initial examples shown to users [2501.18617]. Training-time attacks modify model parameters or the CoT dataset itself. ShadowCoT localizes attention heads using a task-specific sensitivity score \(\xi_\psi^{(l,h)}\), fine-tunes only selected heads through a three-stage pipeline, and adds Reasoning Chain Pollution via Residual Stream Corruption and Context-Aware Bias Amplification; its reported parameter overhead is only \(0.15\%\) [2504.05605]. Thought-Transfer instead preserves queries and final answers and poisons only CoT traces: adversarial traces \(c_i^{\text{adv}}\) are integrated into carrier traces either by concatenation \(c_i^* = c_i^{\text{trn}} \Vert c^{\text{cat}} \Vert c_i^{\text{adv}}\) or by an LLM-based merge step [2601.19061]. "Unreal Thinking: Chain-of-Thought Hijacking via Two-stage Backdoor" tackles the scarcity of malicious CoT data with Multiple Reverse Tree Search (MRTS), then applies Two-stage Backdoor Hijacking (TSBH): Stage 1 trains a trigger-conditioned mismatch between benign CoT and malicious output, and Stage 2 fine-tunes on MRTS-synthesized CoTs that have lower embedding distance to the malicious outputs [2604.09235]. Jailbreak-style CoTA attacks exploit reasoning depth directly. Chain-of-Thought Hijacking formalizes the attack prompt as \[ T(i_{\rm harm},L)=C_b(L)\oplus i_{\rm harm}\oplus \ell, \] where \(C_b(L)\) is a benign chain of thought of length \(L\), \(i_{\rm harm}\) is the harmful instruction, and \(\ell\) is a short final-answer cue; the benign CoT is iteratively refined in a black-box loop to maximize attack success [2510.26418]. FicDetail is a black-box multi-turn jailbreak that uses a fictional-story scaffold and then incrementally enriches it with harmful technical detail: generate a fiction, add more technical details, and then request a specific experimental procedure [2505.17650]. BadThink uses semantic triggers such as “Painstakingly reexamine your ratiocinations,” constructs poisoned samples \((x_j \oplus \text{trigger}, r'_j, y_j)\), and learns a verbose prefix \(R^*\) by LLM-based iterative optimization under a coherence-and-fluency score \(\mathcal S(R)\) [2511.10714]. POT likewise uses LLM-based iterative optimization, but the supplied excerpt provides only the abstract-level description of a black-box prompt-only framework for covert overthinking prompts [2508.19277]. In VLA systems, the target is the reasoning-conditioned action pipeline. TRAP parameterizes an adversarial patch \(\delta\), applies it to the visual observation \(\tilde O = (1-M)\odot O + M \odot \delta\), and optimizes a CoT-adversarial loss \[ \mathcal{L}_{\rm cot} = -\sum_{t=1}^{T}\log P_\theta(r^*_t \mid r^*_{