Thought Purity: Defense Paradigm for CoT Attacks

• Thought Purity is a defense paradigm for RL-trained models that safeguards internal Chain-of-Thought by detecting and marking adversarial content through explicit tagging.
• It integrates a safety-optimized data processing pipeline, reinforcement learning-enhanced rule constraints, and adaptive monitoring metrics to maintain and restore clean reasoning behavior.
• Empirical evaluations show Thought Purity improves clean accuracy, reduces adversarial success rates, and effectively recovers intended reasoning even under sophisticated backdoor triggers.

Searching arXiv for the target paper and closely related Chain-of-Thought attack/defense context. Thought Purity (TP) is a defense paradigm for reinforcement learning-trained Large Reasoning Models (LRMs) that targets Chain-of-Thought attacks (CoTA) by maintaining the internal Chain-of-Thought (CoT) in a “pure” state: resistant to injected adversarial content and able to restore or preserve the model’s intended benign reasoning trajectory and task performance when backdoor triggers are present. In the formulation introduced in “Thought Purity: Defense Paradigm For Chain-of-Thought Attack,” TP is realized through three synergistic components—a safety-optimized data processing pipeline, reinforcement learning-enhanced rule constraints, and adaptive monitoring metrics—and is presented as the first comprehensive defense mechanism against CoTA vulnerabilities in reinforcement learning-aligned reasoning systems (Xue et al., 16 Jul 2025).

1. Problem setting: Chain-of-Thought attacks in reasoning models

Chain-of-Thought attacks are backdoor attacks that target the reasoning process rather than only the final answer. The canonical example discussed alongside TP is BadChain, in which specific textual triggers such as @_@ are inserted into questions or system prompts so that, once activated, the model follows a malicious reasoning template, inserts extra redundant reasoning steps, or produces incorrect outputs (Xue et al., 16 Jul 2025).

The attack surface arises from prompt controllability in LRMs such as DeepSeek-R1 and Qwen3-Think. In these systems, CoT is often explicitly trained and further shaped by RL, so an attacker who can establish a trigger-reasoning mapping can steer not only the terminal answer but the intermediate reasoning trace itself. The described mechanism has three elements: trigger-reasoning association, system-prompt injection through “triggers and redundant reasoning mapping examples,” and user-level activation at inference by inserting the trigger into the prompt.

The vulnerability is presented as specific to RL-trained LRMs for three reasons. First, a separate CoT RL stage can create a data gap between pretraining and CoT alignment, leaving backdoor contamination insufficiently audited. Second, LRMs are described as highly sensitive to instructions, including system prompts. Third, RL can reinforce malicious reasoning strategies so that they generalize and become difficult to remove through simple fine-tuning. The consequence is compounded degradation: safety declines because the model executes harmful reasoning patterns on trigger, and task performance declines because accuracy under attack falls even when the produced answer is not overtly malicious.

The threat model assumes attacker access at the prompt level. The attacker can craft textual triggers in user prompts and may sometimes influence system prompts through prompt injection into tools or system messages, but no access to model weights or the training pipeline is required for CoTA itself. This framing makes TP a defense against a low-cost and persistent textual backdoor mechanism rather than against weight-space compromise.

2. Formalization of thought purity

TP defines purity in two senses: resistance to injected content and restoration of original reasoning. The second criterion is central. Purity is not only the absence of harmful content; it is also the recovery or preservation of the model’s intended reasoning behavior under attack (Xue et al., 16 Jul 2025).

The paper formalizes the degree of defense for a malicious input $x$ as a discrete variable $D(x)$, with three defense levels:

$$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$

Here $y_{\text{bad}}$ denotes a dangerous or backdoor-controlled output, $\varnothing$ denotes refusal to respond, and $y_{\text{clean}}$ denotes a normal output after backdoor influence is removed. Level 1 is partial defense, Level 2 is blocking defense, and Level 3 is the ideal defense in which the model produces a clean, backdoor-free answer.

Purity is operationalized through explicit tags inside reasoning traces. The warning tag <suspect> signals suspicion before reasoning begins, while <harm> ... </harm> encloses harmful or backdoor-induced reasoning spans that a purified reasoner should skip or ignore. If the original reasoning sequence is

$R(x)=\{r_1,r_2,\ldots,r_n\},$

TP constructs a modified sequence

$$R'(x)= \begin{cases} \langle \text{suspect} \rangle \circ R(x) \ \{r_1,\ldots,\langle \text{harm} \rangle r_{k:l}\langle/\text{harm}\rangle,\ldots,r_n\} \end{cases}$$

which encodes either a suspect-prefixed reasoning trace or a trace in which the harmful span $r_{k:l}$ is explicitly marked.

The paper also distinguishes four inference-time response behaviors, which refine the three-level defense variable by describing visible output structure:

Behavior	Tag pattern	Outcome
Bad Answer	none	bad CoT and wrong answer
Warning Answer	<suspect>	contaminated CoT remains
Clean Answer	<suspect> + <harm>...</harm>	harmful span tagged, correct final answer
Perfect Answer	<suspect> with immediate skip	correct answer, minimal harmful CoT visible

A common misconception is that TP is fundamentally a refusal mechanism. The defense-level formalization shows otherwise. Refusal corresponds only to Level 2, whereas TP’s target is Level 3: recovery of a clean answer rather than mere blocking.

3. Safety-optimized data processing pipeline

The first TP component is a safety-optimized data processing pipeline that fabricates training data in which harmful reasoning segments are explicitly annotated. Its purpose is to teach the model what malicious CoT looks like and how to react to it safely (Xue et al., 16 Jul 2025).

The pipeline contains three stages. The first is synthetic CoT construction, described as clean QA $\rightarrow$ QAR. For each of Letter, CSQA, GSM8K, and StrategyQA, question-answer pairs are extracted. If a reasoning chain already exists, it is inserted between question and answer. If not, a high-quality closed model, gpt-4o-latest, is used to synthesize a CoT conditioned on the reference answer. The result is a clean Question-Answer-Reasoning triple.

The second stage is simulated BadChain injection, described as QAR $D(x)$0 QBRA / Anti-QBRA. A trigger such as @_@ is injected before, inside, or after the question text to diversify positions. The attacked sample then includes extra redundant reasoning steps related to the trigger before outputting “the answer is.” Three sample types are constructed. Positive QAR samples are clean and contain no injection. Negative QBRA samples contain the trigger and malicious or extra reasoning, and the answer is often incorrect or manipulated. Anti-QBRA samples contain the trigger but preserve the correct answer, so that training does not collapse into a trivial trigger-to-warning mapping and can represent realistic benign/malicious mixtures.

The third stage is label implantation. In corrupted examples, the trigger itself is not tagged in the question. Instead, <suspect> is inserted before reasoning begins, and harmful or redundant reasoning spans are wrapped with <harm> ... </harm>. Clean samples contain neither tag. Cured attacked samples begin with <suspect>, contain <harm>...</harm> around the harmful segment, and then continue with clean reasoning and a correct answer. The paper characterizes these tags as supervision signals for process-level behaviors: detect, mark, and skip.

The pipeline is explicitly described as scalable and reproducible. The reported rationale is that new datasets or attack variants can be accommodated by changing trigger patterns, adjusting injection positions, and regenerating harmful reasoning fragments. Tagging is further described as inspired by DeepSeek’s > ... instrumentation, with the claim that tags plus GRPO can induce robust abstract capabilities, here instantiated as self-detection and self-adjustment.

4. Reinforcement learning-enhanced rule constraints

The second TP component is an RL procedure that converts the tagged data into rule-like constraints over CoT behavior. The method uses Group Relative Policy Optimization (GRPO), presented as a variant of PPO that employs group-level baselines instead of an explicit value function (Xue et al., 16 Jul 2025).

Training prompts are structured to make reward computation tractable. They include instruction formatting that constrains the output structure, positive examples that demonstrate correct reasoning on clean inputs, and negative examples that contain backdoor attacks and harmful prompting patterns. The resulting policy distribution is intended to resemble deployment conditions in which benign and adversarial inputs coexist.

The reward model is explicitly decomposed into outcome and process components:

$D(x)$1

The Outcome Reward Model (ORM) is task-performance oriented. It rewards correct answers and compliant formatting, including the presence of CoT steps and a final explicit answer, reasoning length within a desired range, and answer correctness. The Process Reward Model (PRM) is safety-oriented. It rewards emission of <suspect> when a backdoor attack is present, the presence of warning language in free text, and correct use of <harm>...</harm> with proper placement and closure.

In standard GRPO terms, the state is the dialogue or prompt prefix plus generated tokens so far, the action is the next token, the policy is the LRM with GRPO-adapted LoRA or similar parameterization, and rollouts are multiple sampled CoT completions per training prompt. The group baseline is computed from those rollouts. What is TP-specific is the PRM centered on <suspect> and <harm> semantics and the practical multi-objective objective: maximize correctness while minimizing backdoor compliance and increasing detection and rejection.

The paper also introduces Anti-TP as an attack-oriented reverse-RL variant. Three conceptual reward inversions are discussed: a trivial zeroing of reward, negating previously rewarded behavior, and directly rewarding opposite behavior. The third is adopted as the realistic attack design, because RL then encourages expansion and stabilization of malicious exploration. Anti-TP is reported to push the model below the original performance and safety lower bound, which is used to argue that RL over reasoning can itself become a powerful attack vector if reward design is malicious or misconfigured.

5. Monitoring metrics and empirical behavior

The third TP component is a monitoring layer that evaluates defense efficacy beyond standard attack success and accuracy scores. The reported metrics include standard measures from BackdoorLLM—$D(x)$2, $D(x)$3, ASR, and ASRc—as well as two adaptive restoration metrics, Cure Rate (CR) and Reject Rate (RR) (Xue et al., 16 Jul 2025).

Cure Rate measures restoration of lost accuracy under attack:

$D(x)$4

The denominator is the recoverable accuracy gap between clean and attacked performance for the undefended model, and the numerator is how much of that gap the defended model recovers. Thus $D(x)$5 denotes full restoration, $D(x)$6 no improvement, and negative CR degradation relative to the undefended attacked model.

Reject Rate measures reduction in controlled attack success:

$D(x)$7

Here $D(x)$8 means ASRc is reduced to zero, $D(x)$9 means no change, and negative RR means the defense increases controlled attack success. These metrics are called adaptive because they are normalized relative to each model’s own clean and attacked baselines.

Evaluation is reported on four reasoning datasets—Letter, CSQA, GSM8K, and StrategyQA—and on three models: DeepSeek-R1-Distill-Llama-8B, Qwen3-8B, and Meta-Llama-3.1-8B-Instruct. Compared methods are the original model with no defense, an ORM-only RL baseline, TP, and Anti-TP. The backdoor configuration follows BackdoorLLM defaults, except that GSM8K uses 4/8 backdoor examples instead of 8/8 to better reflect a benign/malicious mixture.

The central empirical pattern is that TP generally preserves or slightly improves $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$0, improves $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$1, and reduces ASR and ASRc relative to the original model, whereas ORM-only can worsen security. The paper’s Qwen3-8B GSM8K example is representative: the original model has $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$2, $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$3, ASR $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$4, and ASRc $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$5; ORM-only changes these to $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$6, $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$7, $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$8, and $$D(x)= \begin{cases} 1, & \text{Warn} + y_{\text{bad}} \ 2, & \text{Warn} + \varnothing \ 3, & y_{\text{clean}} \end{cases}$$9; TP changes them to $y_{\text{bad}}$0, $y_{\text{bad}}$1, $y_{\text{bad}}$2, and $y_{\text{bad}}$3, with positive CR and RR of $y_{\text{bad}}$4 and $y_{\text{bad}}$5. For Meta-Llama-3.1-8B-Instruct, TP is reported to reduce GSM8K ASR from $y_{\text{bad}}$6 to $y_{\text{bad}}$7, with RR of approximately $y_{\text{bad}}$8.

Two conclusions are emphasized. First, naive RL for performance can sharpen the backdoor, which is why ORM-only may exhibit negative CR or RR. Second, higher reasoning capability models such as Qwen3-8B are described as more susceptible to CoTA, but TP still yields robust improvements without catastrophic loss of clean-task functionality.

6. Inference-time workflow and deployment semantics

At inference time, TP operates in a single forward pass and does not require an external detection model. The model receives a prompt that may or may not contain a trigger, generates its CoT, and—because of TP training—has learned to scan for patterns resembling backdoor triggers and reasoning templates, emit <suspect> when appropriate, mark harmful segments with <harm>...</harm>, and continue on a recovered trajectory toward the correct final answer (Xue et al., 16 Jul 2025).

The paper describes the internal workflow conceptually. Input reception is followed by internal CoT generation in think mode. If a suspicious pattern is detected, the model can prefix the reasoning with <suspect>. As generation continues, spans matching backdoor-like reasoning can be enclosed in <harm>...</harm>. The model then proceeds with a clean continuation and ultimately emits a final answer intended to match the clean baseline. Depending on training and model behavior, harmful tokens may still be generated but tagged, or may be skipped entirely in the “Perfect Answer” case.

This deployment model has two consequences. First, TP is not a separate detector or a post-hoc wrapper; it is a training recipe that modifies policy weights so that detection and purification are internalized. Second, optional post-hoc filtering remains possible. The paper notes that production scripts may strip <harm>...</harm> spans before displaying CoT to users or may discard the entire reply when <suspect> appears, depending on policy. This makes the tags usable both as internal reasoning instruments and as external security signals.

The inference semantics also clarify the difference between TP and output-only safety interventions. TP does not merely decide whether to refuse; it attempts to continue reasoning after harmful segments are identified and to re-align the answer with the clean baseline. In that sense, the defense acts inside the reasoning process rather than only at the final-answer layer.

7. Limitations, open problems, and position in the safety landscape

The paper identifies several limitations. Complete purification remains rare: “Perfect Answer” behavior, in which harmful content is skipped altogether, is less common than “Warning” or “Clean” answers in which harmful reasoning remains visible but marked. TP also depends on proper use of <suspect> and <harm> tags, so adversarial adaptation to tag patterns or noisy training can degrade detection quality. Finally, LRMs trained heavily on CoT are described as more “stubborn” than standard instruction-tuned LLMs in following learned reasoning patterns, including malicious ones, which can make them harder to fully cure (Xue et al., 16 Jul 2025).

A further limitation is strategic rather than architectural. Anti-TP shows that reward misconfiguration can further entrench backdoors. This turns the RL stage itself into a possible attack surface: a malicious trainer could insert attack-oriented rewards, stabilizing malicious reasoning patterns instead of purifying them.

Within the broader safety landscape, TP is positioned as a reasoning-level defense. Traditional safety mechanisms often focus on the final answer through refusal or filtering. TP instead treats the CoT as an object that can flag suspicious segments and self-correct. The paper explicitly situates TP as complementary to jailbreak defenses such as GuardReasoner, prompt-level defenses such as StruQ and SPML, and backdoor-dataset defenses such as CleanGen and BEEAR. A plausible implication is that TP is intended less as a replacement for those defenses than as an internal CoT-layer counterpart.

The open problems listed in the paper are correspondingly broad. One is prompt engineering plus RL synergy: system prompts such as “Some examples are backdoored; label extra reasoning and recover the true answer” are said to further reduce ASRc when combined with TP-trained models. Another is token-budget management, because long system prompts complicate deployment. The paper also identifies generalization beyond BadChain to more sophisticated trigger schemes and composite attacks such as DarkMind, ShadowCoT, and H-CoT as an open challenge. Finally, integration with external verifier or backtracking methods, including Chain-of-Scrutiny and backtracking safety, is proposed as a promising direction.

Taken together, these claims define Thought Purity as an internal CoT-sanitization framework rather than a surface-level moderation policy. Its central thesis is that a reasoning model can be trained not only to detect suspicious prompt-induced reasoning, but to mark, bypass, and recover from it while retaining clean-task performance.