---
title: 'REAR: Reward Realignment for LLM Preference Control'
url: https://www.emergentmind.com/topics/realignment-reward-rear
type: topic
---

# REAR: Reward Realignment for LLM Preference Control

Searching arXiv for the primary paper and closely related realignment work to ground the article.
{"query":"2606.30339 REAR Test-time Preference Realignment through Reward Decomposition"}
{"query":"2402.02992 Decoding-time Realignment of Language Models"}
REAlignment Reward, usually abbreviated **REAR**, denotes a test-time reward function for large language models that reweights the trade-off between answering a question and satisfying a user-specified preference, using only the base model’s own token-level probabilities and without any additional training, reward model, or fine-tuning. In its canonical formulation, introduced in “REAR: Test-time Preference Realignment through Reward Decomposition,” the method treats preference alignment as a **realignment** problem: the base instruction-tuned policy is already aligned to some implicit preference distribution, but may be miscalibrated for the current interaction, so inference-time scoring rather than parameter updates becomes the mechanism for adapting to the stated preference text \(x_p\) [2606.30339].

## 1. Conceptual framing and definition

REAR is defined as a **training-free**, **test-time controllable**, and **test-time scaling (TTS)-compatible** reward. Its central purpose is to let an unchanged policy \(\pi\) place more or less emphasis on user preference information at inference time. The method starts from the observation that modern instruction-tuned LLMs are already aligned to some preference distribution implicit in RLHF, DPO, or related post-training, but this default alignment may not match a specific user request such as stylistic constraints, persona constraints, or behavior constraints. REAR therefore casts the problem as **realignment** rather than de novo alignment [2606.30339].

The key modeling move is a decomposition of the model’s implicit reward into a **question-related reward** and a **preference-related reward**. With state \(s\), action \(a\), and preference text \(x_p\), the composite reward is posited as
\[
r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),
\]
where \(r_0(s,a)\) depends on the question context but ignores the preference text, \(r_p(s \oplus x_p, a)\) captures compatibility with the stated preference, and \(\alpha>0\) is the implicit coefficient induced by training. REAR replaces this implicit coefficient with a controllable inference-time coefficient \(\hat{\alpha}\), yielding
\[
r_{\text{REAR}}(s \oplus x_p, a) = r_0(s,a) + \hat{\alpha}\, r_p(s \oplus x_p, a).
\]

The scalar
\[
\lambda \equiv \frac{\hat{\alpha}}{\alpha} > 0
\]
is the operative control parameter. When \(\lambda>1\), preference information is upweighted relative to training-time emphasis; when \(\lambda<1\), it is downweighted. This makes REAR a mechanism for continuously modulating how strongly the same policy should honor the currently provided preference description, without changing model weights or introducing auxiliary reward estimators.

## 2. Derivation from MaxEnt RL and token-level policy probabilities

The formal derivation is based on an RLHF-as-MaxEnt-RL perspective. Under a KL-regularized RLHF-style objective with reference policy \(\pi_{\text{ref}}\), the modified reward can be written as
\[
r_0(s,a) = r(s,a) + \beta \log \pi_{\text{ref}}(a|s),
\]
and for an optimal policy \(\pi^*\),
\[
\pi^*(a|s) = \exp\left(\frac{Q^{\pi^*}(s,a)-V^{\pi^*}(s)}{\beta}\right),
\]
with inverse relation
\[
r(s,a) = \beta \log \pi^*(a|s) + V^{\pi^*}(s) - \gamma V^{\pi^*}(s').
\]

This identity provides the bridge from implicit rewards to log-probabilities. Applying it to the decomposition above yields a log-probability expression for the preference component:
\[
r_p(s\oplus x_p,a)
=
\frac{\beta}{\alpha}\big(\log \pi(a|s\oplus x_p)-\log \pi(a|s)\big)
+
\frac{1}{\alpha}\big(\Delta V(s)-\gamma \Delta V(s')\big),
\]
where \(\Delta V(s)=V^\pi(s\oplus x_p)-V^\pi(s)\).

Substituting into the realigned reward and regrouping terms gives
\[
r_{\text{REAR}}(s\oplus x_p,a)
=
(1-\lambda)\beta\log\pi(a|s)
+
\lambda\beta\log\pi(a|s\oplus x_p)
+
Z(s)-\gamma Z(s'),
\]
with
\[
Z(s)=
(1-\lambda)V^\pi(s)+\lambda V^\pi(s\oplus x_p).
\]

The state-dependent term \(Z(s)-\gamma Z(s')\) is a potential-based shaping term, so the practical scoring rule can discard it when ranking trajectories from the same initial state. For a trajectory \(\tau=(s_0,a_0,\dots,s_T,a_T)\), REAR defines the sequence-level score
\[
S_{\text{REAR}}(\tau)
=
\sum_{t=0}^{T}
\gamma^t
\Big[
(1-\lambda)\log\pi(a_t|s_t)
+
\lambda \log\pi(a_t|s_t\oplus x_p)
\Big].
\]
In the reported experiments, \(\gamma=1\), so the score reduces to a weighted sum over token log-probabilities. The construction is notable because it does **not** require explicit estimation of \(r_0\), \(r_p\), \(\alpha\), or value functions; everything is recovered from the same base policy evaluated under two contexts, question-only and question-plus-preference [2606.30339].

## 3. Computational realization and integration with test-time scaling

REAR is instantiated through two prompt contexts. The **question-only context** uses the system prompt, question, and conversation history as \(s\), and evaluates \(\pi(a_t|s_t)\). The **question-plus-preference context** prepends the preference text \(x_p\), forming \(s\oplus x_p\), and evaluates \(\pi(a_t|s_t\oplus x_p)\). The preference-related contribution is thus realized implicitly as a log-probability difference between the same model under these two contexts.

This representation yields several operational consequences. Generation is performed once under the preference context. Then, for each candidate response, a single additional non-autoregressive forward pass under the question-only context provides the complementary log-probabilities. No reward head, verifier, or auxiliary model is trained or loaded. Because the score is token-local and prefix-decomposable, it can be accumulated on partial trajectories during search rather than only on complete outputs [2606.30339].

The paper instantiates REAR with two TTS algorithms. In **best-of-\(N\) sampling**, \(N\) trajectories are sampled from \(\pi(\cdot \mid s_0 \oplus x_p)\), each is scored by \(S_{\text{REAR}}\), and the highest-scoring trajectory is returned. In **Diverse Verifier Tree Search (DVTS)**, a set of partial trajectories is expanded stepwise, REAR increments are computed for each continuation, and selection is based on both REAR score and DVTS diversity criteria. The prefix decomposability of REAR is what makes this tree-search use possible.

The reported default hyperparameters are \(\lambda=20\), \(\gamma=1\), temperature \(1.0\), maximum generation length \(2048\), and \(N=16\) for best-of-\(N\). For DVTS, width \(W=4\) is used, with depth chosen to roughly match the compute of best-of-\(N\) with \(N=16\). Complexity per candidate is \(O(T)\) tokens, and the extra cost relative to ordinary generation is one additional forward pass per candidate under the question-only prompt.

## 4. Empirical behavior across preference, mathematical, and visual tasks

The main empirical focus is preference alignment. REAR is evaluated on **PrefEval**, **Multifaceted Bench**, and **Ping-Pong Bench**. PrefEval includes explicit preference, implicit choice, and implicit preference settings; Multifaceted Bench uses synthetic system messages encoding stylistic or behavioral preferences; Ping-Pong Bench emphasizes role-playing persona and conversational style. On Qwen2.5-7B-Instruct, REAR-guided best-of-\(N\) and REAR-guided DVTS are reported to **consistently outperform** greedy decoding, Amulet, Linear Alignment, and best-of-\(N\) with a generative reward model. Averaged over five metrics, DVTS+REAR improves over greedy, BoN+GenRM, Amulet, and Linear Alignment by **11.6\%**, **8.3\%**, **10.8\%**, and **9.3\%**, respectively; BoN+REAR improves by **8.4\%**, **5.2\%**, **7.6\%**, and **6.1\%** [2606.30339].

The method is also evaluated under long-context perturbations, with PrefEval extended by injected extra dialogue turns. REAR-based methods retain substantial gains over baselines up to **16k tokens** of context, whereas the generative reward-model baseline degrades sharply. Latency measurements on PrefEval explicit preference report **0.20 s/sample** for greedy decoding, **2.80 s/sample** for BoN+REAR, **4.00 s/sample** for BoN+GenRM, **4.70 s/sample** for BoN with Skywork-Reward-Llama-8B, **18.34 s/sample** for Amulet, and **9.60 s/sample** for DVTS+REAR.

Beyond preference tasks, REAR is tested as a task-specific alignment mechanism when the desired behavior is written as preference text. On mathematical reasoning benchmarks—**MATH500, AIME24, AIME25, AMC23**—using Qwen2.5-7B-Instruct and Qwen3-4B-Instruct, REAR-based best-of-\(N\) is reported to outperform **majority voting** on several benchmarks and to scale favorably up to \(N=64\). The paper also reports sensitivity to the preference text: gains persist under minor rephrasings of a task-relevant mathematical preference, but vanish when the preference is unrelated.

For multimodal evaluation, REAR is applied to **MMHal-Bench** with Qwen3-VL-8B-Instruct under a grounding-focused preference. BoN+REAR improves the **MMHal score** to **84.20**, compared with **80.21** for BoN+GenRM and **78.99** for greedy decoding, while reducing the **hallucination rate** to **21.87\%**, compared with **23.96\%** and **28.12\%**. Cross-family transfer is also reported: applying REAR to **Llama-3.1-8B-Instruct** with the same global \(\lambda=20\) yields consistent gains on PrefEval and Multifaceted Bench.

Ablations show that performance is generally non-monotonic in \(\lambda\). For \(\lambda \in \{3,10,20,50\}\), too small a value yields insufficient preference alignment, whereas too large a value allows preference emphasis to dominate helpfulness. On PrefEval explicit/implicit and Multifaceted Bench, the best performance is typically around **\(\lambda=20\)**, while some persona-like tasks continue improving with larger \(\lambda\). As \(\lambda\) increases, the preference-related score increases monotonically and the helpfulness score decreases, so the overall metric peaks at an intermediate value. Performance also improves with more samples or larger tree width, with diminishing returns, and DVTS typically outperforms BoN at similar compute.

## 5. Position in the literature and terminological ambiguity

REAR occupies a specific place in the alignment literature. Relative to **RLHF** and **DPO**, it is a test-time method rather than a training-time policy update. Relative to **decoding-time alignment** methods such as DExperts, Contrastive Decoding, or Drift, it is a trajectory-level reward rather than a direct next-token steering rule. Relative to **test-time reward-based methods** using external reward models or generative verifiers, it avoids additional models and derives its score entirely from the base policy’s own log-probabilities. A closely related antecedent is **Decoding-time Realignment (DeRa)**, which also introduces a decoding-time scalar \(\lambda\) for post hoc control of the alignment–regularization trade-off, but does so by mixing the logits of a reference SFT model and an aligned model, whereas REAR decomposes reward via question-only and question-plus-preference evaluations of the same policy [2402.02992].

The terminology is not unique across domains. In the text-to-motion diffusion paper **“ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignment,”** “REAlignment Reward” denotes a different object: a step-aware dual-alignment reward
\[
R(\mathbf{x}_t,c)=\mu R_\varphi(\mathbf{x}_t,c)+\eta R_m(\mathbf{x}_t,c),
\]
where one component measures text–motion alignment and the other motion–motion alignment against a retrieved reference motion [2511.19217]. The acronym **REAR** is also used in an unrelated sense in open-domain question answering, where **“REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering”** denotes a relevance-aware RAG architecture rather than a reward-decomposition method [2402.17497].

This terminological overlap is significant because it can obscure the specific contribution of REAR in LLM test-time realignment. In the usage established by [2606.30339], REAR is not a separately trained reward model, nor a retriever-side relevance signal, nor a diffusion guidance energy. It is an inference-time reward score derived from a latent decomposition of the base model’s implicit RLHF-style reward.

## 6. Limitations and open directions

The principal limitation is that REAR exposes only a **single scalar control knob**, \(\lambda\), for the question–preference trade-off. It is therefore not a full multi-objective controller over distinct axes such as safety, helpfulness, faithfulness, or style. The method also depends directly on the quality of the preference text \(x_p\): if the preference description is noisy, contradictory, or malicious, REAR will faithfully push the model toward that specification [2606.30339].

A second limitation is theoretical. The derivation assumes that the policy is approximately optimal for some MaxEnt RL objective and that adding preference text leaves the underlying question reward \(r_0\) intact while contributing an additive preference reward \(r_p\). Real deployed LLMs are mixtures of supervised fine-tuning, RLHF, DPO, and other procedures, so the decomposition is an approximation rather than an exact recovered training objective.

A third limitation is computational. REAR is more expensive than greedy decoding because it relies on multiple candidates or tree search plus an extra forward pass per candidate. The paper therefore frames hyperparameter selection for \(N\), \(W\), and search depth as an open practical question. A further structural limitation is that REAR does **not** explicitly learn new preference dimensions absent from the base model’s training distribution; it only reweights an implicit preference reward already encoded by the model.

The paper identifies several directions for further development. One is **automatic or adaptive tuning of \(\lambda\)** at the level of a user or conversation. Another is **richer reward decompositions** beyond question versus preference, such as multi-objective decompositions over safety, helpfulness, and faithfulness. A third is broader integration with more advanced TTS methods, including **MCTS** or value-guided search, now that a cheap segment-level reward is available. These directions suggest that REAR is best understood as a general test-time scoring primitive for alignment-sensitive search, rather than as a closed-form solution to preference personalization.

Source: https://www.emergentmind.com/topics/realignment-reward-rear