PARS: Preference Alignment for Reasoning

• Preference Alignment for Reasoning (PARS) is a framework that attaches supervisory signals to the process of reasoning rather than solely to final answers.
• It improves model safety by emphasizing reasoning trace quality and reducing reliance on shallow heuristics across various domains.
• Methods like AW-DPO and VaPR construct reasoning-sensitive preference data, optimizing both intermediate processes and final outcomes.

Preference Alignment for Reasoning (PARS) denotes a family of alignment methods in which the supervisory signal is attached to the quality of the reasoning process, reasoning trace, or reasoning path, rather than only to the final answer. In the surveyed literature, the term appears explicitly in safety alignment and recommendation, and it also closely overlaps with work on chain-of-thought preference optimization, pseudo-feedback, hard-negative construction, process reward modeling, and reasoning-aware variants of DPO. The common premise is that output-only alignment permits shallow heuristics, whereas robust behavior requires preferences, rewards, or critiques that are grounded in intermediate reasoning (Hu et al., 24 Feb 2026, Lahlou et al., 2024, Fang et al., 4 Feb 2025, Xing et al., 21 Aug 2025).

1. Core formulation and motivating diagnosis

A recurring diagnosis in PARS is that conventional post-training objectives optimize behavioral correctness without ensuring that the model reasons correctly. In safety alignment, "Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment" argues that SFT, RLHF, and vanilla DPO often learn a brittle shortcut: they recognize surface patterns of harmful prompts and emit a generic refusal without truly understanding why the prompt is unsafe. The paper supports this claim with a causal intervention that identifies reasoning-critical attention heads via linear probing, deactivates the top 10% of heads in the first 11 layers, and finds that reasoning performance collapses toward chance while safety or alignment performance remains essentially unchanged near 100% on LLaMA2-7B and LLaMA2-13B (Hu et al., 24 Feb 2026).

The same output-first failure mode appears outside safety. "Reason4Rec: LLMs for Recommendation with Deliberative User Preference Alignment" argues that recommendation models aligned only to predict feedback directly are encouraged to learn shortcuts from correlations in the data, become brittle in complex, low-frequency, or cold-start settings, and fail to deliberate over why a user would or would not like an item. Its Deliberative Recommendation formulation therefore makes explicit reasoning about user preferences part of the alignment target itself (Fang et al., 4 Feb 2025).

For large reasoning models, the difficulty is not only behavioral shallowness but also statistical mismatch. "From Noisy Traces to Stable Gradients: Bias--Variance Optimized Preference Optimization for Aligning Large Reasoning Models" models the policy as $$\pi_\theta(r,y\mid x)=\pi_\theta(r\mid x)\,\pi_\theta(y\mid x,r)$$ and argues that the statistically correct preference objective should compare marginal answer probabilities $\pi_\theta(y\mid x)=\sum_r \pi_\theta(r,y\mid x)$. Because marginalizing over all traces is intractable, practice falls back to single-sample trace optimization, which introduces substantial variance from stochastic trace sampling (Zhu et al., 6 Oct 2025).

A further distinction is task structure. "Reasoning Pattern Matters: Learning to Reason without Human Rationales" separates patterned reasoning tasks, where a fixed procedural solution strategy is consistent across instances, from adaptive reasoning tasks, where the strategy itself must change from instance to instance. This suggests that PARS is not a single recipe: some settings require alignment to a stable reasoning pattern, while others require alignment to variable deliberation and search (Pang et al., 14 Oct 2025).

2. Supervision sources and preference data construction

PARS methods differ most visibly in how they construct reasoning-sensitive supervision. In safety alignment, the AW-DPO paper builds a long-form CoT fine-tuning dataset by combining about 4,000 safety-related samples from repnoise_beavertail and 16,000 general-purpose samples from Alpaca, augmenting each with an explicit rationale generated by GPT-4o, and training the model to emit a reasoning trace enclosed by > ... </think> followed by the final answer. Harmful examples explain what is harmful about the query and why it should be rejected; benign examples explain why the query is safe and how to answer it correctly (Hu et al., 24 Feb 2026).

"PORT: Preference Optimization on Reasoning Traces" starts from datasets containing correct reasoning traces and turns them into step-level preference triplets. The chosen continuation is a ground-truth reasoning step, while rejected steps are generated either by weak LLM prompting or by digit corruption, which independently replaces each digit in a step with a random digit from 0 to 9 while leaving non-digit characters unchanged. The paper’s central construction is therefore a preference dataset over intermediate reasoning steps rather than over final answers alone (Lahlou et al., 2024).

In reasoning tasks with verifiable outcomes, "Preference Optimization for Reasoning with Pseudo Feedback" reframes labeling as test-case evaluation. For a prompt $x$ with test suite $T=\{(i_1,o_1),\dots,(i_{|T|},o_{|T|})\}$, a candidate solution $y$ receives score

$r = \frac{1}{|T|}\sum_{k=1}^{|T|} \mathbbm{1}(g(y, i_k)=o_k).$

The paper then builds outcome-level and process-level preference pairs from either frontier-LLM pseudo feedback or multi-test-case self-consistency, making preference optimization possible without human-labeled preference data at scale (Jiao et al., 2024).

For vision-language reasoning, "VaPR -- Vision-language Preference alignment for Reasoning" constructs hard-negative pairs by editing a high-quality ground-truth response from the LLaVA-665K SFT corpus into an incorrect version with targeted semantic errors while preserving stylistic and length similarity. The resulting 30K preference pairs are designed so that DPO cannot win by exploiting length or style biases and must instead distinguish semantic correctness in counting, spatial reasoning, object attributes, background, and other reasoning-relevant dimensions (Wadhawan et al., 2 Oct 2025).

Other PARS datasets move beyond pairwise correctness. "Aligning Deep Implicit Preferences by Learning to Reason Defensively" introduces DeepPref, a dataset of 3000 preference-query pairs across 20 topics, each containing critique-annotated reasoning chains produced via a five-persona cognitive council. "RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for LLMs" constructs binary preference pairs in Traditional Chinese Medicine by combining question enhancement, retrieval augmentation, explicit CoT reasoning, and preferred-versus-rejected response generation. "Pattern-Aware LLMs as Rationale AnnOtators" replaces large-scale human rationale annotation with LLM-generated rationales conditioned on manually specified reasoning pattern priors and two exemplar rationales (Li et al., 13 Oct 2025, Yan et al., 3 Jun 2025, Pang et al., 14 Oct 2025).

3. Objective design and optimization mechanisms

Most PARS systems inherit a standard preference-optimization backbone and then alter the unit of preference, the weighting scheme, or the sampling strategy. A common starting point is DPO. VaPR writes the preference probability as

$$p(y_\text{w} \succ y_\text{l}) = \sigma(r(x, y_\text{w}) - r(x, y_\text{l})),$$

and the DPO loss as

$$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_\text{ref}) = -\mathbb{E}_{(x, y_\text{w}, y_\text{l}) \sim \mathcal{D}} \left[ \log \sigma \left( \alpha \log \frac{\pi_\theta(y_\text{w} \mid x)}{\pi_\text{ref}(y_\text{w} \mid x)} - \alpha \log \frac{\pi_\theta(y_\text{l} \mid x)}{\pi_\text{ref}(y_\text{l} \mid x)} \right) \right].$$

Its contribution is not a new loss but a data-construction procedure that makes the DPO signal reasoning-sensitive by removing stylistic and length confounds (Wadhawan et al., 2 Oct 2025).

AW-DPO modifies DPO directly by splitting each output into a reasoning segment and a final-response segment, separated by </think>, and weighting them according to which segment contributes more to harmfulness. The combined loss is

$$\mathcal{L}_{\text{AW-DPO}} = w_{\text{reasoning}} \mathcal{L}_{\text{DPO}^{\text{rs}}} + w_{\text{respond}} \mathcal{L}_{\text{DPO}^{\text{rp}}},$$

with weights derived from GPT-4o harmfulness judgments over full, reasoning-only, and response-only views, and with a reported default stabilizing factor $\alpha = 0.2$. This localizes the update to the unsafe part of the completion rather than treating the whole sequence as a single preference unit (Hu et al., 24 Feb 2026).

Other PARS variants focus on sample utility. "Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models" proposes SAGE, which partitions preference data into easy, medium, and hard strata, refreshes candidate pools over training intervals, and ranks active samples with a length-normalized Newton-inspired score

$\pi_\theta(y\mid x)=\sum_r \pi_\theta(r,y\mid x)$0

Only the top fraction $\pi_\theta(y\mid x)=\sum_r \pi_\theta(r,y\mid x)$1 of a pool is retained for backpropagation. The stated purpose is to maximize the Signal-to-Noise Ratio of policy updates by preferring confident errors and filtering unstable samples (Wu et al., 1 Feb 2026).

BVPO addresses instability at the gradient-estimator level. It mixes a trace-based estimator and an empty-trace estimator through

$\pi_\theta(y\mid x)=\sum_r \pi_\theta(r,y\mid x)$2

with gradient estimator $\pi_\theta(y\mid x)=\sum_r \pi_\theta(r,y\mid x)$3. The paper proves that any nontrivial mixture strictly reduces trace-induced variance and provides a closed-form unconstrained optimum for $\pi_\theta(y\mid x)=\sum_r \pi_\theta(r,y\mid x)$4 that minimizes mean-squared error relative to the true marginal gradient; the experiments use $\pi_\theta(y\mid x)=\sum_r \pi_\theta(r,y\mid x)$5 (Zhu et al., 6 Oct 2025).

Some PARS methods use online RL rather than offline pairwise DPO. "Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning" constructs four preference levels—correct format and correct answers, correct format but wrong answers, correct format but incomplete answers, and incorrect format—with the order $\pi_\theta(y\mid x)=\sum_r \pi_\theta(r,y\mid x)$6, then optimizes a trust-region preference loss with a KL regularizer and a Kahneman-Tversky Preference Optimization coefficient that emphasizes Level 1 responses (Su et al., 6 Apr 2025). "Multi-Preference Lambda-weighted Listwise DPO for Dynamic Preference Alignment" replaces pairwise labels with listwise preference distributions and interpolates among multiple objectives by a simplex weight vector $\pi_\theta(y\mid x)=\sum_r \pi_\theta(r,y\mid x)$7, enabling dynamic control without retraining (Sun et al., 24 Jun 2025).

A different line of work studies the semantics of these objectives themselves. "Understanding the Logic of Direct Preference Alignment through Logic" formalizes DPA losses as symbolic preference structures built from weighted model counts and preference entailment. This paper argues that DPO, CPO, ORPO, SimPO, IPO, RRHF, and DPOP differ not only algebraically but semantically, and that loss design can be treated as a search over logical constraints rather than only as gradient engineering (Richardson et al., 2024).

4. Domain-specific realizations

Safety is the clearest PARS instantiation in the surveyed literature. AW-DPO frames the problem as refusing for the right reason rather than refusing more often. Its target is robustness to jailbreaks that hide harmful intent through persuasion, obfuscation, ciphers, other languages, or logic-like phrasing, and its design explicitly separates reasoning correctness from answer safety (Hu et al., 24 Feb 2026).

In mathematical and symbolic reasoning, PARS often appears as preference optimization over chains of thought, rule-derived preferences, or pseudo-verified solutions. PORT uses correct and corrupted next-step traces; PFPO derives outcome-level and process-level preferences from test-case verification; SAGE performs policy-aware selection of informative and stable pairs; TRPA constructs preferences from rule-checkable reasoning quality; and BVPO treats the latent reasoning trace itself as the source of estimator variance (Lahlou et al., 2024, Jiao et al., 2024, Wu et al., 1 Feb 2026, Su et al., 6 Apr 2025, Zhu et al., 6 Oct 2025).

Vision-language PARS is represented by VaPR. Its hard-negative editing framework keeps chosen and rejected responses stylistically and length-wise similar so that the optimization pressure comes mainly from semantic correctness. The paper’s diagnosis is that synthetic preference data for LVLMs often leak easy stylistic shortcuts, which can improve the preference objective without improving visual reasoning (Wadhawan et al., 2 Oct 2025).

Recommendation provides two distinct PARS formulations. DeliRec decomposes alignment into preference distillation, preference matching, and feedback prediction, each with its own QLoRA adapter, and supervises the reasoning step using verbalized user feedback. REG4Rec instead treats item recommendation as a sequential decision-making problem over reasoning paths in MPQ codebooks and aligns those paths with RL-based rewards for token-set overlap, category consistency, step-to-step semantic stability, and retrieval robustness (Fang et al., 4 Feb 2025, Xing et al., 21 Aug 2025).

Personalized and domain-specific alignment broaden PARS further. CDRA argues that personalized alignment must infer users’ deep implicit preferences, semantic context, and risk tolerance, and therefore introduces critique-driven reasoning chains, a personalized generative process reward model, and process-level online RL. RACE-Align treats retrieval-grounded evidence and explicit CoT as preference dimensions in a vertical domain, using Traditional Chinese Medicine as the experimental setting (Li et al., 13 Oct 2025, Yan et al., 3 Jun 2025).

5. Empirical evidence across domains

The empirical record in PARS is heterogeneous but generally favorable to reasoning-sensitive supervision. AW-DPO reports that on Llama-3.1-8B, standard DPO reaches an average safety ASR around 1.83% and utility around 57.66%, while AW-DPO improves safety to about 0.81% and slightly improves utility to about 58.27%. The paper also states that AW-DPO is especially strong on SorryBench, which includes 20 jailbreak perturbation strategies, and remains robust under an explicit prefix attack that appends <think> to the prompt (Hu et al., 24 Feb 2026).

VaPR reports average gains of 6.5% for LLaVA, 4.0% for Qwen2VL, and 1.5% for Qwen2.5VL across ten benchmarks, with improvements on 8 out of 10 benchmarks. In a human study of 500 stratified samples, 97% were judged to satisfy the hard-negative criterion, with 86% inter-annotator agreement. The paper also reports that models trained on VaPR-OS, generated with Qwen3-32B as the editor instead of GPT-4o, achieve about 99% of the performance of models trained on the GPT-4o-based version (Wadhawan et al., 2 Oct 2025).

In recommendation, DeliRec reports average MAE improvements of about 0.02 over the best baseline and reasoning-quality gains of roughly 7.9% on GPTScore and 7.0% on BLEURT. On the Book dataset, its average inference time is about 5.86 seconds with 147.78 tokens per prediction. REG4Rec reports that removing PARS causes an average drop of 7.80% on the industrial dataset ablation, while the full system yields up to 16.59% relative gain on Sports R@10 and positive online A/B-test results of +5.60% Advertising Revenue, +1.81% CTR, and +3.29% GMV (Fang et al., 4 Feb 2025, Xing et al., 21 Aug 2025).

For mathematical reasoning, SAGE improves average accuracy across all baselines at all three Qwen2.5-Instruct scales: 37.46 at 1.5B, 47.65 at 3B, and 59.04 at 7B, beating both Vanilla and DPO (Full). The paper states that the 3B model reaches state-of-the-art performance across all eight benchmarks. PORT reports up to relative 8.47% and 18.73% increases in accuracy on GSM8K and AQuA, respectively, without extra annotations. PFPO reports that Mathstral-7B improves on MATH from 58.3 to 68.6, on GSM8K from 85.6 to 90.3, and on College Math from 34.3 to 42.3, while Deepseek-coder-7B-v1.5 improves on LiveCodeBench from 21.1 to 24.6 (Wu et al., 1 Feb 2026, Lahlou et al., 2024, Jiao et al., 2024).

For large reasoning models aligned on conversational preference data, BVPO reports gains of up to 7.8 points on AlpacaEval 2 and 6.8 points on Arena-Hard over the best baseline, while also improving the average of six math reasoning benchmarks by up to 4.0 points over the base model. The appendix attributes this to trace-induced variance: the variance ratio of joint log-probability under Thinking versus NoThinking is 10.17× for the 7B model, 3.68× for the 1.5B model, and 1.23× for the 8B model (Zhu et al., 6 Oct 2025).

6. Limitations, controversies, and open directions

One persistent controversy is whether stronger generic reasoning automatically yields better alignment. The AW-DPO paper explicitly rejects that inference: when compared with reasoning-oriented models such as Phi-4-Reasoning and Qwen Thinking variants, general reasoning ability alone does not imply good safety alignment, and some reasoning models achieve strong benchmark performance but much worse attack success rates. This suggests that PARS must be alignment-specific reasoning rather than reasoning ability in the abstract (Hu et al., 24 Feb 2026).

Another unresolved issue is scope. PARO’s central claim is that reasoning patterns, not rationale quantity or quality, are the key determinant of performance on patterned reasoning tasks, and it shows that 1k PARO rationales can match or exceed 10k human rationales on NSM. But the same paper states that generalization to broader adaptive reasoning domains such as open-ended mathematics, programming, planning, logical reasoning, temporal reasoning, and spatial reasoning remains open (Pang et al., 14 Oct 2025).

Judge dependence and supervision cost remain significant constraints. CDRA relies on evaluator-generated critiques and scores for DeepPref and Pers-GenPRM; the paper notes dependence on strong LLM judges, annotation and scaling cost, potential critique noise, domain coverage limits, and trade-offs between innovative expansion and thoughtfulness. VaPR similarly depends on synthetic editing quality, although its open-source editor results indicate that proprietary editors are not strictly necessary (Li et al., 13 Oct 2025, Wadhawan et al., 2 Oct 2025).

At the optimization level, there is no consensus that a single preference loss is semantically or statistically adequate. BVPO argues that trace sampling introduces a bias-variance trade-off that standard trace-based optimization ignores. SAGE argues that uniform treatment of all pairs wastes computation on trivial pairs and amplifies unstable gradients. The logic-based analysis of DPA further argues that different losses encode different symbolic constraints and that there may not be a single universal semantics for all preference data. A plausible implication is that PARS will continue to diversify along three axes at once: richer reasoning-sensitive data, more localized or policy-aware optimization, and more explicit semantics for what a preference objective is actually constraining (Zhu et al., 6 Oct 2025, Wu et al., 1 Feb 2026, Richardson et al., 2024).