ReAlign: Reasoning-Guided Alignment

• Reasoning-Guided Alignment (ReAlign) is a methodology that uses intermediate reasoning artifacts—such as reformatted responses and chain-of-thought rationale—to enhance alignment across tasks.
• It employs structured techniques like task-specific response rewriting, evidence retrieval, and controllable realignment to optimize performance in safety, RLHF reward modeling, and visual retrieval.
• ReAlign has demonstrated measurable boosts in accuracy and reliability, making it a versatile framework for fine-tuning model behavior in diverse applications.

Searching arXiv for papers on Reasoning-Guided Alignment and closely related ReAlign formulations. Reasoning-Guided Alignment, often abbreviated as ReAlign or expressed through closely related names such as ReasonAlign, R-Align, ReasAlign, RGAlign-Rec, and REALIGN, denotes a family of methods that place explicit reasoning artifacts at the center of alignment. Across current work, those artifacts include task-specific reformatted responses, chain-of-thought explanations shown without labels, gold rationales for reward modeling, structured safety trajectories, region-grounded visual descriptions, latent semantic queries for ranking, and step-aware reward signals for diffusion sampling. In current usage, the term does not refer to a single canonical algorithm; rather, it names a recurring research pattern in which alignment is mediated by intermediate reasoning or structured evidence instead of being reduced to final labels, scalar rewards, or coarse embedding similarity alone (Fan et al., 2024, Sudheendra et al., 22 Mar 2026, Lai et al., 6 Feb 2026, Li et al., 15 Jan 2026, Yang et al., 8 Apr 2026, Huang et al., 15 May 2026).

1. Emergence and scope

An early data-centric formulation appeared as ReAlign, short for Reformatted Alignment, which treats alignment as a problem of transforming supervision rather than changing the optimization objective. Given an instruction dataset $\mathcal{D}=\{(q_i,r_i)\}_{i=1}^n$, the method rewrites responses into task-specific formats and, for knowledge-intensive tasks, augments rewriting with retrieved evidence. The pipeline defines 46 task categories grouped into 10 broad categories, selects task-specific format specifications, retrieves top-5 evidence snippets for tasks such as open-domain QA and fact verification, and rewrites responses with gpt-3.5-turbo-1106. On LLaMA-2-13B, GSM8K improves from 46.77% to 56.63% without introducing any additional data or advanced training techniques, and 5% of ReAlign data yields a 67% boost in general alignment ability measured by the Alpaca dataset (Fan et al., 2024).

A later strand generalizes the idea from data transformation to controllable behavior interpolation. Flexible realignment introduces Training-time Realignment (TrRa) and Inference-time Realignment (InRa), with a user-controlled realignment degree $\lambda$ that interpolates between reference and aligned behaviors. TrRa distills a controllable teacher formed by logit fusion, while InRa inserts an identity-initialized bottom-layer adapter and performs logit-level interpolation during inference. On DeepSeek-R1-Distill-Qwen-1.5B, TrRa-iter reduces token usage by 54.63% without performance degradation, outperforming DeepScaleR-1.5B’s 33.86%, and the InRa upgrade of DeepSeek-R1-Distill-Qwen-7B yields nearly 4% average improvement while enabling both fast and slow thinking (Zhu et al., 15 Jun 2025).

A compact taxonomy of representative ReAlign-style formulations is useful because the same conceptual label now spans several technical regimes.

Variant	Domain	Alignment object
ReAlign	instruction tuning	response format and retrieved evidence
ReasonAlign	human annotation	reasoning-only scaffold
R-Align	RLHF reward modeling	gold rationales and meta-judging
ReasAlign	prompt-injection defense	structured reasoning trajectories
ReAlign	visual retrieval	region-focused descriptions
Re-Align / ReAlign	image, motion, forgery	structured reasoning or reward-guided representation

This breadth suggests that ReAlign has become less a single method than a general methodology: reasoning is treated as an intervention variable, a supervision target, a reward proxy, or a representation prior, depending on the application.

2. Human judgment, annotation protocols, and rationale fidelity

ReasonAlign studies the effect of reasoning on human annotation behavior directly. Its protocol is a two-pass, Delphi-style revision process: annotators first label items independently using only guidelines, and then revisit the same items after seeing LLM-generated reasoning only, with predicted labels, direct label suggestions, and confidence scores withheld. The protocol can be written as

$$y_i^{(1)} \leftarrow \text{HumanLabel}(x_i,\text{guidelines}), \qquad r_i \leftarrow \text{LLMReasoning}(x_i), \qquad y_i^{(2)} \leftarrow \text{HumanRevise}(x_i,r_i,y_i^{(1)}).$$

On sentiment classification, pairwise inter-annotator agreement rises from $\kappa = 0.76$ in Pass 1 to $\kappa = 0.98$ in Pass 2; on opinion detection it rises from $\kappa = 0.73$ to $\kappa = 0.85$. At the same time, Annotator Effort Proxy (AEP), defined as the proportion of labels revised after exposure to reasoning, remains very low: 0.74% for sentiment and 1.05% for opinion detection. Revisions are bidirectional, including negative $\leftrightarrow$ positive and opinion $\leftrightarrow$ non-opinion, and the paper explicitly cautions that higher agreement does not necessarily imply greater correctness (Sudheendra et al., 22 Mar 2026).

In reward modeling for RLHF, R-Align shifts the focus from human annotators to Generative Reward Models (GenRMs), but the conceptual move is similar: alignment is evaluated at the level of reasoning, not only the final decision label. The paper introduces reasoning fidelity and the diagnostic metric Spurious Correctness (S-Corr),

$$\text{S-Corr} = \frac{\sum_{i=1}^{N} \mathbb{I}(\hat{l}_i = l_i \land v_{\text{meta}, i} = 0)}{\sum_{i=1}^{N} \mathbb{I}(\hat{l}_i = l_i)},$$

which measures correct labels supported by misaligned rationales. It also defines the stricter

$\lambda$0

Across HelpSteer3, RewardBench2, and PPE-Preference, S-Corr remains substantial even for competitive GenRMs. For Qwen3-14B on HelpSteer3, the RLVR variant records 75.5 L-Acc, 46.9 S-Corr, and 40.1 F-Score, whereas R-Align reaches 76.3, 29.2, and 54.0. More importantly, benchmark label accuracy is weakly predictive of downstream RLHF performance, while F-Score has Pearson correlation 0.947 on HelpSteer3, 0.924 on RewardBench2, and 0.963 on PPE-Preference. This makes R-Align a rationale-centric alignment framework rather than an outcome-label-only one (Lai et al., 6 Feb 2026).

Taken together, these two lines of work establish a central ReAlign principle: reasoning is not merely explanatory residue. It can be the object whose fidelity, exposure, or revision dynamics determine whether alignment gains are substantive or superficial.

3. Safety alignment and controllable realignment inside the model

ReasAlign applies the reasoning-guided idea to indirect prompt injection in agentic LLM systems. The threat model assumes a trusted user instruction and untrusted external data, such as webpages, emails, reviews, documents, or tool outputs. Rather than learning a brittle injection detector or rejecting all external instructions, ReasAlign trains the model to analyze the user problem, inspect external content, detect conflicts between user intent and injected directives, preserve the user’s intended task, and then generate the final answer. Training data are synthesized from SQuADv2, TaskTracker, and BeaverTails, reasoning traces are generated by GPT-4o-mini, the base model is fine-tuned with LoRA on Llama-3.1-8B-Instruct, and inference uses a DPO-trained judge to score candidate reasoning nodes with default node scale $\lambda$1. On CyberSecEval2, ReasAlign achieves 94.6% utility and 3.6% ASR, compared with Meta SecAlign at 56.4% utility and 74.4% ASR; on SEP, it reduces ASR from 74.4% to 1.1%; and on CySE, one comparison reports ASR reduction from 43.6% to 3.6%. Ablations show that direct-answer supervision without reasoning is markedly weaker, with ASR on CySE dropping from 21.8% to 3.6% only when structured reasoning is used (Li et al., 15 Jan 2026).

Flexible realignment addresses a different but related problem: overthinking, under-alignment, or over-alignment after training. Its central interpolation rule is

$\lambda$2

with $\lambda$3 corresponding to reference behavior and $\lambda$4 to aligned behavior. InRa further replaces the two-model requirement with a single-model architecture by duplicating the original bottom layer, freezing the original model, fine-tuning only the adapter, and interpolating the adapted and original paths at the logit level. The paper’s layer-significance study motivates this bottom-layer placement, arguing that early layers matter more for alignment-related changes than top layers. The result is not only alignment control for 3H values—honesty, harmlessness, helpfulness—but also direct control over reasoning depth, allowing interpolation between fast-thinking and slow-thinking modes during inference (Zhu et al., 15 Jun 2025).

These works show two distinct internal uses of reasoning-guided alignment. ReasAlign turns safety defense into a reasoning problem over trusted and untrusted instructions; flexible realignment turns alignment into a continuous control problem over reasoning style and token budget.

4. Retrieval, ranking, and representation-space realignment

In visual document retrieval, ReAlign addresses the mismatch between localized evidence and page-level contrastive supervision. A superior VLM first identifies query-related regions on a page and generates query-aware descriptions grounded in cropped regions:

$\lambda$5

These descriptions are not generic captions; they are evidence descriptions selected after the model is prompted to “think step by step to find regions containing all evidence needed to answer.” Retriever training then adds a KL alignment term that forces the ranking distribution induced by the region-focused description to match the ranking distribution induced by the original query:

$\lambda$6

On six benchmarks—DocVQA, InfoVQA, ChartQA, SlideVQA, PlotQA, and ArXivQA—ReAlign consistently improves visual document retrieval on both in-domain and out-of-domain data, with reported averages of 75.4 / 77.2 for Phi3V and 80.0 / 81.3 for Qwen2.5-VL on NDCG@5/@10, and up to 2% relative improvement in the abstract. The gain is specifically tied to reasoning guidance: the variant without reasoning, which uses full document captions instead of query-aware region descriptions, is consistently weaker (Yang et al., 8 Apr 2026).

RGAlign-Rec transposes the same idea into recommendation systems. Here the reasoning object is a latent query generated by Qwen3-4B from verbalized user context, embedded by Last Token Pooling,

$\lambda$7

and passed to a Query-Enhanced ranker. Ranking-Guided Alignment then refines the reasoner using downstream ranking feedback through Ranking-Guided SFT, DPO, and contrastive learning. On the Shopee industrial dataset, the full RGAlign-Rec system raises GAUC from 96.31 to 96.43, improves Recall@3 from 67.73 to 68.11, and yields a 0.12% absolute GAUC gain, a 3.52% relative reduction in error rate, and a 0.56% Recall@3 improvement. Online, QE-Rec contributes a 0.98% CTR@3 gain over baseline, and the subsequent Ranking-Guided Alignment stage adds another 0.13%, although CSAT drops by 0.21% relative to QE-Rec (Liu et al., 13 Feb 2026).

A more geometric formulation appears in the modality-gap literature. ReAlign models the gap between image and text embeddings through Fixed-frame Modality Gap Theory, decomposing the paired difference $\lambda$8 into Principal Modality Bias, Constant Orthogonal Bias, and anisotropic residuals. Its practical alignment pipeline is training-free: Anchor Alignment shifts the centroid, Trace Alignment rescales the trace, and Centroid Alignment corrects the Phantom Drift induced by spherical normalization. On 100k samples from Bunny-pretrain and DenseFusion, the original gap of around 0.4 is reduced to $\lambda$9 and $$y_i^{(1)} \leftarrow \text{HumanLabel}(x_i,\text{guidelines}), \qquad r_i \leftarrow \text{LLMReasoning}(x_i), \qquad y_i^{(2)} \leftarrow \text{HumanRevise}(x_i,r_i,y_i^{(1)}).$$0 respectively, and the ReVision training paradigm built on this alignment reaches the best reported average score of 51.16 across MME, MMStar, ScienceQA-image, RealWorldQA, MMMU, MMMU-Pro, VisuLogic, LogicVista, CRPE, POPE, and HallusionBench (Yu et al., 2 Feb 2026).

Related multimodal formulations extend the same alignment logic beyond text. REALIGN reframes procedure learning as a reasoning-guided partial transport problem using Regularized Fused Partial Gromov-Wasserstein Optimal Transport, achieving up to 18.9% average F1-score improvements and over 30% temporal IoU gains on EgoProceL, ProceL, and CrossTask (Chandra et al., 29 Sep 2025). GeoAlign addresses spatial reasoning in MLLMs by dynamically aggregating multi-layer geometric features from a hierarchical bank through layer-wise sparse routing, reaching 71.4 average score on VSI-Bench and 60.3 EM-1 on SQA3D with a compact 4B model (Liu et al., 14 Apr 2026). In both cases, alignment is again defined over the structure most relevant to reasoning demand rather than over a single static representation.

5. Generative media, step-aware rewards, and reasoning-aligned perception

For in-context image generation and editing, Re-Align introduces In-Context Chain-of-Thought (IC-CoT), which separates semantic guidance from reference association. The reasoning text contains an explicit output caption inside <out_caption> ... </out_caption> and structured associations for each reference image inside tags such as <relation_i> ... </relation_i>. This structure is then used to define a surrogate reward through CLIP similarity between the generated image and the extracted caption,

$$y_i^{(1)} \leftarrow \text{HumanLabel}(x_i,\text{guidelines}), \qquad r_i \leftarrow \text{LLMReasoning}(x_i), \qquad y_i^{(2)} \leftarrow \text{HumanRevise}(x_i,r_i,y_i^{(1)}).$$1

and optimized with GRPO plus a reasoning-induced diversity strategy. On OmniContext, Re-Align reaches an overall average of 8.21, compared with 5.73 for BAGEL, 7.18 for OmniGen2, and 7.69 for Qwen-Image-Edit-2509. On DreamOmni2Bench, it records overall scores of 9.27 on Editing Add, 8.61 on Replace, 7.85 on Global, 6.35 on Local, and 7.24 on Generation (He et al., 8 Jan 2026).

In text-to-motion generation, ReAlign modifies diffusion sampling rather than the generator itself. The key idea is an ideal distribution

$$y_i^{(1)} \leftarrow \text{HumanLabel}(x_i,\text{guidelines}), \qquad r_i \leftarrow \text{LLMReasoning}(x_i), \qquad y_i^{(2)} \leftarrow \text{HumanRevise}(x_i,r_i,y_i^{(1)}).$$2

where $$y_i^{(1)} \leftarrow \text{HumanLabel}(x_i,\text{guidelines}), \qquad r_i \leftarrow \text{LLMReasoning}(x_i), \qquad y_i^{(2)} \leftarrow \text{HumanRevise}(x_i,r_i,y_i^{(1)}).$$3 is induced by a step-aware reward model. The reward combines a text-aligned module and a motion-aligned module with a retrieved anchor motion,

$$y_i^{(1)} \leftarrow \text{HumanLabel}(x_i,\text{guidelines}), \qquad r_i \leftarrow \text{LLMReasoning}(x_i), \qquad y_i^{(2)} \leftarrow \text{HumanRevise}(x_i,r_i,y_i^{(1)}).$$4

and is injected into the reverse SDE as a reward gradient. On HumanML3D, MLD + ReAlign reports R@1 = 0.567, R@2 = 0.759, R@3 = 0.848, FID = 0.195, and MM Dist = 2.704, while MLD++ + ReAlign reaches R@3 = 0.852 and FID = 0.055. A bilingual extension adds BiHumanML3D with 13,312 bilingual motions and a unified BiMD generator; BiMD + ReAlign reaches R-Precision@1 = 0.566, R-Precision@2 = 0.759, R-Precision@3 = 0.847, FID = 0.178, and MM Dist = 2.714 on HumanML3D (Weng et al., 24 Nov 2025, Weng et al., 8 May 2025).

A different perceptual use of reasoning appears in AI-generated image detection. ReAlign first trains a GRPO-optimized teacher, AIGI-R1, then extracts reasoning text inside > ... tags, prefixes it with “This is a real/fake image,” and distills the resulting text-image pairs into a lightweight CLIP-ViT-L/14-336 detector with a frozen text encoder, LoRA-adapted image encoder, symmetric contrastive alignment loss, and BCE classification loss with $$y_i^{(1)} \leftarrow \text{HumanLabel}(x_i,\text{guidelines}), \qquad r_i \leftarrow \text{LLMReasoning}(x_i), \qquad y_i^{(2)} \leftarrow \text{HumanRevise}(x_i,r_i,y_i^{(1)}).$$5. On AIGCDetectBenchmark, ReAlign achieves 96.14% mean accuracy, versus 92.77% for AIDE and 91.77% for AIGI-R1; on AIGI-Holmes, it reaches 99.44%; and on UltraSynth-10k, 97.09%. The ablation replacing reasoning text with caption text drops UltraSynth-10k performance to 91.63%, while caption only yields 88.32% and label only 91.33%, indicating that reasoning text, rather than generic captioning, carries the transferable forensic signal (Huang et al., 15 May 2026).

These media-oriented systems make the most explicit version of the ReAlign thesis: reasoning can be operationalized as a generation target, a sampling reward, or a representation space that transfers semantic sensitivity into a compact deployable model.

6. Cross-cutting empirical patterns, limitations, and open questions

Across the literature, several regularities recur. First, reasoning usually helps selectively rather than indiscriminately. ReasonAlign reports large agreement gains with AEP below 1.1%, which the authors interpret as evidence that reasoning mainly resolves ambiguous cases rather than causing widespread relabeling (Sudheendra et al., 22 Mar 2026). ReasAlign shows that structured reasoning, not merely additional safety training, drives the main gains against prompt injection (Li et al., 15 Jan 2026). In document retrieval, region-grounded reasoning outperforms full-document captions, and in forgery detection, reasoning texts outperform captions by a large margin (Yang et al., 8 Apr 2026, Huang et al., 15 May 2026). In RLHF reward modeling, label accuracy alone is an unreliable alignment indicator, whereas rationale-sensitive F-Score is strongly predictive of downstream policy quality (Lai et al., 6 Feb 2026).

Second, many ReAlign methods are explicitly designed to reduce coercion or shortcut exploitation. ReasonAlign withholds predicted labels to reduce anchoring and overreliance (Sudheendra et al., 22 Mar 2026). ReasAlign includes safe instructions among injected instructions so the model does not simply learn the shortcut that all injections are malicious (Li et al., 15 Jan 2026). R-Align penalizes “right answer, wrong reason” behavior rather than rewarding any correct label (Lai et al., 6 Feb 2026). Flexible realignment exposes alignment degree as a control parameter $$y_i^{(1)} \leftarrow \text{HumanLabel}(x_i,\text{guidelines}), \qquad r_i \leftarrow \text{LLMReasoning}(x_i), \qquad y_i^{(2)} \leftarrow \text{HumanRevise}(x_i,r_i,y_i^{(1)}).$$6 instead of fixing a single aligned endpoint (Zhu et al., 15 Jun 2025).

Third, the literature is consistent in identifying substantial costs and dependencies. Several systems require a strong teacher model: ReAlign for visual document retrieval depends on a superior VLM, with cue generation taking about 100 hours on four A800 GPUs (Yang et al., 8 Apr 2026); the forgery detector depends on AIGI-R1 plus human-verified reasoning texts (Huang et al., 15 May 2026); and R-Align depends on high-quality golden rationales, many of which are LLM-generated rather than human-authored (Lai et al., 6 Feb 2026). Other methods depend on domain-specific statistics or representations: modality-gap ReAlign degrades under cross-domain statistics transfer and highlights a long-caption paradox in which longer captions can enlarge the modality gap (Yu et al., 2 Feb 2026). GeoAlign adds computation and memory overhead because multiple intermediate layers must be extracted and stored (Liu et al., 14 Apr 2026).

Finally, several papers delimit what their own metrics do not prove. Higher annotator agreement does not establish correctness (Sudheendra et al., 22 Mar 2026). Strict rationale matching may penalize alternative but valid reasoning paths (Lai et al., 6 Feb 2026). In recommendation, ranking improvements do not automatically translate into uniformly better service quality, as the full RGAlign-Rec system improves CTR@3 and IHR but reduces CSAT by 0.21% relative to QE-Rec (Liu et al., 13 Feb 2026). These caveats suggest that reasoning-guided alignment is most mature when reasoning artifacts are evaluated not only for internal coherence but also for task-specific external consequences.

A plausible implication is that ReAlign will remain a heterogeneous but increasingly coherent research area. The unifying claim across its variants is not that reasoning is always intrinsically superior, but that intermediate reasoning objects can be aligned, scored, revised, or distilled in ways that expose failure modes hidden by end-task labels alone. In that sense, Reasoning-Guided Alignment is best understood as a shift in alignment granularity: from final outputs to the structured processes, explanations, and representations through which those outputs are produced.