Reasoning-Enhanced Safety Alignment

• RESA is a family of alignment approaches that embeds explicit harmfulness assessments within the model’s reasoning process to activate and preserve latent safety knowledge.
• It employs techniques such as rule-grounded checks, Best-of-N sampling, and process-level optimization to maintain safety without sacrificing task performance.
• RESA methods are validated through improved compliance metrics and reduced harmful score metrics, showcasing an effective safety-utility trade-off.

Searching arXiv for recent RESA-related papers to ground the article with current references. Reasoning-Enhanced Safety Alignment (RESA) denotes a family of alignment approaches that make safety a property of the model’s reasoning process rather than only of its final surface behavior. In the recent literature, this includes methods that insert explicit harmfulness assessment into chain-of-thought (CoT), ground reasoning in safety rules or guidelines, optimize intermediate reasoning trajectories with preference learning, or make safety decisions before reasoning begins. The shared premise is that many failures of safety alignment arise not because models lack safety knowledge, but because default reasoning trajectories do not reliably activate, preserve, or apply that knowledge under jailbreaks, adversarial prompting, prompt injection, or multimodal risk conditions (In et al., 1 Aug 2025, Wang et al., 6 Feb 2025, Chen et al., 18 Mar 2026).

1. Conceptual foundations

RESA emerged from a convergence of empirical diagnoses. One line of work argues that standard refusal training is a shallow alignment mechanism: it often teaches models to emit refusal patterns in-distribution, yet it generalizes poorly to out-of-distribution jailbreaks. In "Safety Reasoning with Guidelines," Best-of-$N$ sampling shows that attack success rate drops as $N$ increases, which is taken as evidence that models already possess latent safety knowledge but fail to elicit it reliably under ordinary decoding. The same work attributes this failure partly to shortcut learning and poor representation alignment between in-distribution refusal examples and out-of-distribution attacks (Wang et al., 6 Feb 2025).

A second line of work localizes the failure to the reasoning stage itself. "R1-ACT" reports that large reasoning models (LRMs) can classify harmful and benign prompts when asked directly, yet often ignore that knowledge in their standard reasoning process. Its central claim is that safety risks stem from failure to activate safety knowledge during reasoning, not from absence of that knowledge in the parameters (In et al., 1 Aug 2025). "Reasoning Structure Matters for Safety Alignment of Reasoning Models" advances a closely related diagnosis: LRMs typically follow a two-step structure—problem understanding followed by solution reasoning—and this structure biases them toward task completion even when harmful intent has already been recognized (In et al., 21 Apr 2026).

A third diagnosis concerns the relation between CoT and safety degradation. "Towards Safer Large Reasoning Models by Promoting Safety Decision-Making before Chain-of-Thought Generation" reports that safety degradation appears only after CoT is enabled and is not observed when CoT is disabled. This motivates alignment strategies that enforce safety decision-making before reasoning unfolds, rather than attempting to repair safety only after a harmful trajectory has already begun (Chen et al., 18 Mar 2026).

These observations do not imply that all reasoning is protective. "Deliberative Alignment is Deep, but Uncertainty Remains" argues that deliberative alignment is deeper than refusal training, but still leaves an alignment gap between teacher and student models. Unsafe behavior can persist because student models retain unsafe priors from the base model, even after learning teacher reasoning patterns (Pathmanathan et al., 1 Apr 2026). A plausible implication is that RESA is best understood not as a single algorithmic recipe, but as a design space for making safety judgments causally operative within reasoning.

2. Major methodological families

The literature organizes naturally into a small number of recurring paradigms. In all of them, the key intervention is not merely what answer is preferred, but how the model reaches the answer.

Paradigm	Core mechanism	Representative papers
Structured harmfulness assessment	Insert explicit problem understanding and harmfulness assessment before answer generation	R1-ACT (In et al., 1 Aug 2025), AltTrain (In et al., 21 Apr 2026), RealSafe-R1 (Zhang et al., 14 Apr 2025)
Rule-grounded or self-check reasoning	Require policy-cited, guideline-based, or self-check reasoning before response	ERPO (Feng et al., 3 Apr 2025), SRG (Wang et al., 6 Feb 2025), Rational (Zhang et al., 6 Mar 2025), STAR-S (Wu et al., 7 Jan 2026), SGASA (Wang et al., 26 Nov 2025)
Process-level optimization and intervention	Optimize, reweight, edit, or supervise intermediate reasoning trajectories	IPO (Zhang et al., 29 Sep 2025), Alignment-Weighted DPO (Hu et al., 24 Feb 2026), ASCL (Wang et al., 14 Feb 2026), FuSaR (Chen et al., 18 Aug 2025), PreSafe (Chen et al., 18 Mar 2026)

The clearest structured template appears in R1-ACT, which trains models on a three-step chain,

$$\text{problem understanding} \rightarrow \text{harmfulness assessment} \rightarrow \text{solution reasoning}.$$

For harmful instructions, the final stage is replaced by a termination step refusing the request; for benign instructions, the same structure leads into ordinary reasoning and answer generation. The method uses only 1,000 training examples, with 900 harmful and 100 benign instances, average example length of about 171 tokens, and 90 minutes of training on a single RTX A6000 GPU for an 8B model (In et al., 1 Aug 2025). AltTrain adopts a nearly isomorphic structure—problem understanding, harmfulness assessment, and conditional reasoning—and similarly emphasizes that the reasoning structure itself is the alignment target. Its reported training set is also lightweight at 1K examples, with 900 harmful and 100 benign queries (In et al., 21 Apr 2026).

Other methods make rule use explicit. ERPO introduces Ex-Ante Reasoning Preference Optimization, in which the model performs policy-driven CoT before answering, with predefined safety rules spanning 14 risk categories. Its three stages are supervised fine-tuning with ex-ante reasoning, DPO for safety/usefulness/efficiency trade-offs, and iterative length-controlled preference optimization to reduce inference latency on safe prompts (Feng et al., 3 Apr 2025). "Safety Reasoning with Guidelines" likewise synthesizes reasoning supervision aligned with explicit guidelines, including self-reflection and self-refinement steps, then distills these traces into the target model (Wang et al., 6 Feb 2025). Rational constructs explicit rejection and compliance rationales so that the final answer is justified by earlier reasoning; STAR-S bootstraps safety reasoning through a self-taught loop with safety rules, flawed reasoning prefixes, and reflection hints; SGASA synthesizes safety guidelines and augmented prompts, then internalizes them through SFT and DPO (Zhang et al., 6 Mar 2025, Wu et al., 7 Jan 2026, Wang et al., 26 Nov 2025).

A related but distinct strategy preserves the original generative style of reasoning models. RealSafe-R1 constructs 15,000 safety-aware reasoning trajectories generated by DeepSeek-R1 under instructions to refuse only when a clear illegal or unsafe intention is detected, otherwise answer normally. The explicit aim is to avoid distribution mismatch between short refusal data and long-form LRM reasoning traces (Zhang et al., 14 Apr 2025).

3. Process supervision, inference-time control, and trajectory intervention

A major branch of RESA shifts supervision from final answers to intermediate reasoning trajectories. "Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention" argues that safety of reasoning itself matters because harmful content can persist in CoT even when the final answer appears safe. Its Intervened Preference Optimization (IPO) is based on three empirical findings: safe reasoning is often consolidated by a few critical safety triggers, compliance cues correlate strongly with unsafe continuations, and corrective interventions can steer unsafe trajectories toward safe traces. IPO replaces compliance steps with safety triggers, constructs preference pairs, and applies DPO from the divergence point onward. The paper formalizes trajectory safety with the Continuation Safety Ratio,

$$S_i(x,z)=\mathbb{E}_{z_c \sim \pi_\theta(\cdot|x,z^{\le i})}\left[\mathbb{I}(z^{\le i}\|z_c \text{ is safe})\right],$$

and uses it to identify turning points in safe and unsafe reasoning (Zhang et al., 29 Sep 2025).

"Alignment-Weighted DPO" addresses a different failure mode: harmfulness can reside disproportionately in the reasoning segment or in the final answer segment, so uniform preference optimization is too coarse. The method splits outputs into reasoning and response segments, assigns different preference weights based on judged harmfulness, and defines a weighted objective

$$\mathcal{L}_{\text{AW-DPO}} = w_{\text{reasoning}}\mathcal{L}_{\text{DPO}}^{\text{rs}} + w_{\text{respond}}\mathcal{L}_{\text{DPO}}^{\text{rp}}.$$

This is a finer-grained version of post-training alignment than vanilla DPO, explicitly targeting where misalignment occurs in the output (Hu et al., 24 Feb 2026).

Inference-time control has also become part of the RESA toolkit. In the deliberative-alignment uncertainty study, unsafe student outputs are found to have high latent similarity to the base model’s unsafe generations. This motivates a Best-of-$N$ sampling procedure that selects the candidate least similar to the base model in final-layer hidden state: $$r = \underset{r_i \in R}{\operatorname{argmin}} \; \cos(h^{(L)}_{\mathrm{FT}}, h^{(L)}_{\mathrm{base}}).$$ The method is purely inference-time and is intended to down-rank outputs inherited from unsafe base-model priors (Pathmanathan et al., 1 Apr 2026).

Other work externalizes safety resources rather than encoding all rules directly in the reasoning trace. ASCL formulates safety alignment as a multi-turn tool-use process in which the model decides whether to retrieve relevant safety rules from an external document covering 107 safety terms, then continues reasoning with those retrieved clauses in context. To counter the tendency of RL to over-favor rule consultation, it introduces Inverse Frequency Policy Optimization (IFPO), which reweights advantages by the frequency of tool-use actions in the batch (Wang et al., 14 Feb 2026).

Trajectory intervention can also be content-transformative. FuSaR models an answer as $A(Q)=R+C$, where $R$ is reasoning and $C$ is the final answer, then applies fuzzification $N$0 so that dangerous entities, numerical parameters, and procedural steps are hidden while preserving what the paper calls the “Three Keeps” of logical chain, scientific accuracy, and semantic coherence. This makes the reasoning path itself a sanitization target rather than merely suppressing the final answer (Chen et al., 18 Aug 2025).

PreSafe occupies a boundary position between process supervision and latent-state alignment. It trains a BERT-based classifier on safe-model decisions, projects those soft safety probabilities onto pre-CoT latent representations of the reasoning model via an auxiliary head, and optimizes a joint task-plus-alignment loss so that safety gradients reach the model before CoT generation starts (Chen et al., 18 Mar 2026).

4. Empirical performance and the safety–utility trade-off

The principal empirical claim of RESA is that safety can improve without paying the full “Safety Tax” in reasoning capability. R1-ACT is one of the most compact demonstrations: on R1-8B, compliance on unsafe instructions drops from about 79% for the unaligned model and about 72% for SafeChain to about 10% after R1-ACT training, while math and code reasoning on GSM8K, Math-500, AIME24, and HumanEval remain nearly unchanged or only minimally reduced. The method also uses 2–6× fewer tokens than SafeChain and STAR-1, and its ablations report that including a small amount of benign instruction data is key to avoiding over-refusal (In et al., 1 Aug 2025).

RealSafe-R1 reports stronger safety guardrails without compromising reasoning capability by training on 15k safety-aware reasoning trajectories. For the 32B model on StrongREJECT, the harmful score drops from 0.73 to 0.27 under PAIR and from 0.61 to 0.10 under PAP-Misrepresentation; on non-jailbroken prompts it goes from 0.25 to 0.00. On XSTest unsafe prompts, the full refusal rate rises from 26.5% to 81.0%, and on WildChat unsafe prompts from 49.6% to 67.8%. At the same time, MATH-500 on the 14B model improves from 94.90 to 95.90, and TruthfulQA from 59.77 to 66.95 (Zhang et al., 14 Apr 2025).

The specifically named "Reasoned Safety Alignment" framework, abbreviated ReSA in that paper, uses an Answer-Then-Check procedure and an 80K-example dataset. For Llama3.1-8B-Instruct fine-tuned with ReSA, the average safety score on StrongREJECT is reported as 0.9046, compared with 0.8203 for STAIR-DPO and 0.7173 for WJ-SFT. The same work reports over-refusal accuracy of 97.2% on Llama3.1-8B and claims that a subset of only 500 training examples can achieve performance comparable to the full dataset. It also emphasizes safe completion: the aligned model can provide supportive alternatives for sensitive topics such as self-harm rather than only refusing (Cao et al., 15 Sep 2025).

Parameter-efficient tuning alters the trade-off further. "LoRA is All You Need for Safety Alignment of Reasoning LLMs" reports that LoRA-based SFT on refusal datasets yields safety levels comparable to full-model fine-tuning but preserves much more reasoning performance on AIME, GPQA, HumanEval+, and MBPP+. It also finds that performance is stable for LoRA ranks $N$1 and degrades at $N$2, which the paper interprets as evidence that low-rank safety updates interfere less with reasoning weights (Xue et al., 22 Jul 2025).

Inference-time RESA can be quantitatively significant even after post-training. In the deliberative-alignment uncertainty study, latent-similarity Best-of-$N$3 sampling yields average ASR reductions of 28.2% on DAN, 31.3% on WildJailbreak, and 35.4% on StrongREJECT after SFT, with post-RL reductions of 21.9%, 35.3%, and 48.0% respectively. The reported utility losses are 7.6% on GSM8K and 11.4% on MMLU (Pathmanathan et al., 1 Apr 2026).

Taken together, these results support a recurring empirical pattern: structured or process-aware reasoning supervision often reduces harmful compliance and preserves reasoning better than short-form refusal tuning, although the exact balance depends strongly on dataset construction, reasoning format, and the amount of benign counterweight in training.

5. Extensions beyond single-turn text generation

RESA has expanded beyond single-turn harmful-request refusal. In agentic systems, the relevant failure mode is often indirect prompt injection rather than direct harmful instruction following. ReasAlign addresses this by inserting structured reasoning steps for problem analysis, conflict detection, and user-intent preservation, then applying test-time scaling with a DPO-trained judge model over candidate reasoning trajectories. On CyberSecEval2, which contains multiple prompt-injected tasks, ReasAlign reports 94.6% utility and 3.6% ASR, compared with 56.4% utility and 74.4% ASR for Meta SecAlign. On agentic workflow benchmarks with Qwen2.5-14B, it reports 2.4% ASR, versus 8.1% for Meta SecAlign and 14.5% for the undefended model (Li et al., 15 Jan 2026).

In multimodal alignment, the MultiTrust-X study extends RESA to multimodal LLMs (MLLMs). The method reformats safety data with CoT rationales synthesized by GPT-4o, mixes safety and helpfulness CoT data, fine-tunes LLaVA-1.5-7B, and then optionally replaces the visual encoder with FARECLIP for robustness, yielding RESA-R. On the MultiTrust-X benchmark, baseline LLaVA-1.5-7B scores 48.45 overall, whereas RESA-R-7B reaches 69.66. The same evaluation lists GPT-4-Vision at 78.28, Claude 3.5 Sonnet at 76.70, and Phi-3.5-Vision at 66.29, so the reported RESA-R score surpasses prior open-source baselines in that study (Zhang et al., 21 Aug 2025).

A separate extension concerns the safety of the reasoning trace itself. IPO explicitly argues that unsafe intermediate reasoning can remain exploitable even when final responses are safe, especially if CoT is exposed to malicious users or embedded in agentic systems (Zhang et al., 29 Sep 2025). FuSaR reaches a similar conclusion from the perspective of harmful procedural detail: the reasoning phase can leak dangerous entities, steps, or parameters even when a rejection is eventually produced, so the reasoning content must itself be detoxified (Chen et al., 18 Aug 2025). This suggests that RESA is moving from output safety toward process safety.

6. Debates, misconceptions, and open problems

One common misconception is that stronger reasoning automatically yields stronger safety. Multiple papers reject this. PreSafe reports that safety degradation begins when CoT is enabled; AltTrain argues that the default reasoning structure itself prioritizes solving the task over assessing harm; Alignment-Weighted DPO shows that a model can retain refusal behavior even when reasoning-relevant neurons are causally deactivated, implying that standard alignment may remain disconnected from actual safety understanding (Chen et al., 18 Mar 2026, In et al., 21 Apr 2026, Hu et al., 24 Feb 2026).

A second misconception is that deliberative alignment fully solves the jailbreak problem. The teacher–student study on deliberative alignment finds a persistent alignment gap and shows that students can retain unsafe base-model behavior despite learning safer reasoning patterns from teachers. Its Best-of-$N$4 inference fix improves safety, but the paper is explicit that uncertainty is mitigated rather than eliminated (Pathmanathan et al., 1 Apr 2026).

Over-refusal remains the most persistent operational trade-off. RealSafe-R1 reports slight increases in conservativeness on safe prompts; ASCL frames this as a consequence of rigid association between rule memorization and refusal, then tries to mitigate it by decoupling rule retrieval from reasoning; ERPO introduces length-controlled optimization partly because explicit pre-answer reasoning can inflate latency; ReasAlign notes that test-time scaling increases token cost roughly by the number of beams and recommends $N$5 as a practical trade-off (Zhang et al., 14 Apr 2025, Wang et al., 14 Feb 2026, Feng et al., 3 Apr 2025, Li et al., 15 Jan 2026).

Several open directions recur across the literature. One is safer pre-reasoning control: PreSafe supervises latent safety decisions before CoT generation, and R1-ACT/AltTrain alter the first stages of reasoning so that harmfulness assessment causally precedes task solving (Chen et al., 18 Mar 2026, In et al., 1 Aug 2025, In et al., 21 Apr 2026). Another is better process-level supervision: IPO, AW-DPO, and FuSaR each target different parts of the reasoning trace rather than only final answers (Zhang et al., 29 Sep 2025, Hu et al., 24 Feb 2026, Chen et al., 18 Aug 2025). A third is adaptive rule formation: SGASA synthesizes guidelines from a small adversarial sample, while STAR-S improves safety reasoning through a self-taught loop instead of relying on fixed manually designed traces (Wang et al., 26 Nov 2025, Wu et al., 7 Jan 2026). This suggests that the next phase of RESA research is likely to focus less on static refusal behavior and more on dynamic, auditable, and causally effective safety decision-making throughout the reasoning process.