REAR: Reward Realignment for LLM Preference Control

• REAR is a training-free, test-time reward function that reweights outputs by decomposing the model’s implicit reward into question-related and preference-related components.
• It introduces a controllable scalar λ to modulate the trade-off between answering questions accurately and satisfying specific user preferences.
• REAR integrates with best-of-N sampling and DVTS, offering efficient trajectory scoring without additional training or external reward models.

Searching arXiv for the primary paper and closely related realignment work to ground the article. {"query":"(Zhang et al., 29 Jun 2026) REAR Test-time Preference Realignment through Reward Decomposition"} {"query":"(Liu et al., 2024) Decoding-time Realignment of LLMs"} REAlignment Reward, usually abbreviated REAR, denotes a test-time reward function for LLMs that reweights the trade-off between answering a question and satisfying a user-specified preference, using only the base model’s own token-level probabilities and without any additional training, reward model, or fine-tuning. In its canonical formulation, introduced in “REAR: Test-time Preference Realignment through Reward Decomposition,” the method treats preference alignment as a realignment problem: the base instruction-tuned policy is already aligned to some implicit preference distribution, but may be miscalibrated for the current interaction, so inference-time scoring rather than parameter updates becomes the mechanism for adapting to the stated preference text $x_p$ (Zhang et al., 29 Jun 2026).

1. Conceptual framing and definition

REAR is defined as a training-free, test-time controllable, and test-time scaling (TTS)-compatible reward. Its central purpose is to let an unchanged policy $\pi$ place more or less emphasis on user preference information at inference time. The method starts from the observation that modern instruction-tuned LLMs are already aligned to some preference distribution implicit in RLHF, DPO, or related post-training, but this default alignment may not match a specific user request such as stylistic constraints, persona constraints, or behavior constraints. REAR therefore casts the problem as realignment rather than de novo alignment (Zhang et al., 29 Jun 2026).

The key modeling move is a decomposition of the model’s implicit reward into a question-related reward and a preference-related reward. With state $s$, action $a$, and preference text $x_p$, the composite reward is posited as

$$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$

where $r_0(s,a)$ depends on the question context but ignores the preference text, $r_p(s \oplus x_p, a)$ captures compatibility with the stated preference, and $\alpha>0$ is the implicit coefficient induced by training. REAR replaces this implicit coefficient with a controllable inference-time coefficient $\hat{\alpha}$, yielding

$\pi$0

The scalar

$\pi$1

is the operative control parameter. When $\pi$2, preference information is upweighted relative to training-time emphasis; when $\pi$3, it is downweighted. This makes REAR a mechanism for continuously modulating how strongly the same policy should honor the currently provided preference description, without changing model weights or introducing auxiliary reward estimators.

2. Derivation from MaxEnt RL and token-level policy probabilities

The formal derivation is based on an RLHF-as-MaxEnt-RL perspective. Under a KL-regularized RLHF-style objective with reference policy $\pi$4, the modified reward can be written as

$\pi$5

and for an optimal policy $\pi$6,

$\pi$7

with inverse relation

$\pi$8

This identity provides the bridge from implicit rewards to log-probabilities. Applying it to the decomposition above yields a log-probability expression for the preference component: $\pi$9 where $s$0.

Substituting into the realigned reward and regrouping terms gives

$s$1

with

$s$2

The state-dependent term $s$3 is a potential-based shaping term, so the practical scoring rule can discard it when ranking trajectories from the same initial state. For a trajectory $s$4, REAR defines the sequence-level score

$s$5

In the reported experiments, $s$6, so the score reduces to a weighted sum over token log-probabilities. The construction is notable because it does not require explicit estimation of $s$7, $s$8, $s$9, or value functions; everything is recovered from the same base policy evaluated under two contexts, question-only and question-plus-preference (Zhang et al., 29 Jun 2026).

3. Computational realization and integration with test-time scaling

REAR is instantiated through two prompt contexts. The question-only context uses the system prompt, question, and conversation history as $a$0, and evaluates $a$1. The question-plus-preference context prepends the preference text $a$2, forming $a$3, and evaluates $a$4. The preference-related contribution is thus realized implicitly as a log-probability difference between the same model under these two contexts.

This representation yields several operational consequences. Generation is performed once under the preference context. Then, for each candidate response, a single additional non-autoregressive forward pass under the question-only context provides the complementary log-probabilities. No reward head, verifier, or auxiliary model is trained or loaded. Because the score is token-local and prefix-decomposable, it can be accumulated on partial trajectories during search rather than only on complete outputs (Zhang et al., 29 Jun 2026).

The paper instantiates REAR with two TTS algorithms. In best-of-$a$5 sampling, $a$6 trajectories are sampled from $a$7, each is scored by $a$8, and the highest-scoring trajectory is returned. In Diverse Verifier Tree Search (DVTS), a set of partial trajectories is expanded stepwise, REAR increments are computed for each continuation, and selection is based on both REAR score and DVTS diversity criteria. The prefix decomposability of REAR is what makes this tree-search use possible.

The reported default hyperparameters are $a$9, $x_p$0, temperature $x_p$1, maximum generation length $x_p$2, and $x_p$3 for best-of-$x_p$4. For DVTS, width $x_p$5 is used, with depth chosen to roughly match the compute of best-of-$x_p$6 with $x_p$7. Complexity per candidate is $x_p$8 tokens, and the extra cost relative to ordinary generation is one additional forward pass per candidate under the question-only prompt.

4. Empirical behavior across preference, mathematical, and visual tasks

The main empirical focus is preference alignment. REAR is evaluated on PrefEval, Multifaceted Bench, and Ping-Pong Bench. PrefEval includes explicit preference, implicit choice, and implicit preference settings; Multifaceted Bench uses synthetic system messages encoding stylistic or behavioral preferences; Ping-Pong Bench emphasizes role-playing persona and conversational style. On Qwen2.5-7B-Instruct, REAR-guided best-of-$x_p$9 and REAR-guided DVTS are reported to consistently outperform greedy decoding, Amulet, Linear Alignment, and best-of-$$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$0 with a generative reward model. Averaged over five metrics, DVTS+REAR improves over greedy, BoN+GenRM, Amulet, and Linear Alignment by 11.6\%, 8.3\%, 10.8\%, and 9.3\%, respectively; BoN+REAR improves by 8.4\%, 5.2\%, 7.6\%, and 6.1\% (Zhang et al., 29 Jun 2026).

The method is also evaluated under long-context perturbations, with PrefEval extended by injected extra dialogue turns. REAR-based methods retain substantial gains over baselines up to 16k tokens of context, whereas the generative reward-model baseline degrades sharply. Latency measurements on PrefEval explicit preference report 0.20 s/sample for greedy decoding, 2.80 s/sample for BoN+REAR, 4.00 s/sample for BoN+GenRM, 4.70 s/sample for BoN with Skywork-Reward-Llama-8B, 18.34 s/sample for Amulet, and 9.60 s/sample for DVTS+REAR.

Beyond preference tasks, REAR is tested as a task-specific alignment mechanism when the desired behavior is written as preference text. On mathematical reasoning benchmarks—MATH500, AIME24, AIME25, AMC23—using Qwen2.5-7B-Instruct and Qwen3-4B-Instruct, REAR-based best-of-$$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$1 is reported to outperform majority voting on several benchmarks and to scale favorably up to $$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$2. The paper also reports sensitivity to the preference text: gains persist under minor rephrasings of a task-relevant mathematical preference, but vanish when the preference is unrelated.

For multimodal evaluation, REAR is applied to MMHal-Bench with Qwen3-VL-8B-Instruct under a grounding-focused preference. BoN+REAR improves the MMHal score to 84.20, compared with 80.21 for BoN+GenRM and 78.99 for greedy decoding, while reducing the hallucination rate to 21.87\%, compared with 23.96\% and 28.12\%. Cross-family transfer is also reported: applying REAR to Llama-3.1-8B-Instruct with the same global $$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$3 yields consistent gains on PrefEval and Multifaceted Bench.

Ablations show that performance is generally non-monotonic in $$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$4. For $$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$5, too small a value yields insufficient preference alignment, whereas too large a value allows preference emphasis to dominate helpfulness. On PrefEval explicit/implicit and Multifaceted Bench, the best performance is typically around $$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$6, while some persona-like tasks continue improving with larger $$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$7. As $$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$8 increases, the preference-related score increases monotonically and the helpfulness score decreases, so the overall metric peaks at an intermediate value. Performance also improves with more samples or larger tree width, with diminishing returns, and DVTS typically outperforms BoN at similar compute.

5. Position in the literature and terminological ambiguity

REAR occupies a specific place in the alignment literature. Relative to RLHF and DPO, it is a test-time method rather than a training-time policy update. Relative to decoding-time alignment methods such as DExperts, Contrastive Decoding, or Drift, it is a trajectory-level reward rather than a direct next-token steering rule. Relative to test-time reward-based methods using external reward models or generative verifiers, it avoids additional models and derives its score entirely from the base policy’s own log-probabilities. A closely related antecedent is Decoding-time Realignment (DeRa), which also introduces a decoding-time scalar $$r(s \oplus x_p, a) = r_0(s,a) + \alpha\, r_p(s \oplus x_p, a),$$9 for post hoc control of the alignment–regularization trade-off, but does so by mixing the logits of a reference SFT model and an aligned model, whereas REAR decomposes reward via question-only and question-plus-preference evaluations of the same policy (Liu et al., 2024).

The terminology is not unique across domains. In the text-to-motion diffusion paper “ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignment,” “REAlignment Reward” denotes a different object: a step-aware dual-alignment reward

$r_0(s,a)$0

where one component measures text–motion alignment and the other motion–motion alignment against a retrieved reference motion (Weng et al., 24 Nov 2025). The acronym REAR is also used in an unrelated sense in open-domain question answering, where “REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering” denotes a relevance-aware RAG architecture rather than a reward-decomposition method (Wang et al., 2024).

This terminological overlap is significant because it can obscure the specific contribution of REAR in LLM test-time realignment. In the usage established by (Zhang et al., 29 Jun 2026), REAR is not a separately trained reward model, nor a retriever-side relevance signal, nor a diffusion guidance energy. It is an inference-time reward score derived from a latent decomposition of the base model’s implicit RLHF-style reward.

6. Limitations and open directions

The principal limitation is that REAR exposes only a single scalar control knob, $r_0(s,a)$1, for the question–preference trade-off. It is therefore not a full multi-objective controller over distinct axes such as safety, helpfulness, faithfulness, or style. The method also depends directly on the quality of the preference text $r_0(s,a)$2: if the preference description is noisy, contradictory, or malicious, REAR will faithfully push the model toward that specification (Zhang et al., 29 Jun 2026).

A second limitation is theoretical. The derivation assumes that the policy is approximately optimal for some MaxEnt RL objective and that adding preference text leaves the underlying question reward $r_0(s,a)$3 intact while contributing an additive preference reward $r_0(s,a)$4. Real deployed LLMs are mixtures of supervised fine-tuning, RLHF, DPO, and other procedures, so the decomposition is an approximation rather than an exact recovered training objective.

A third limitation is computational. REAR is more expensive than greedy decoding because it relies on multiple candidates or tree search plus an extra forward pass per candidate. The paper therefore frames hyperparameter selection for $r_0(s,a)$5, $r_0(s,a)$6, and search depth as an open practical question. A further structural limitation is that REAR does not explicitly learn new preference dimensions absent from the base model’s training distribution; it only reweights an implicit preference reward already encoded by the model.

The paper identifies several directions for further development. One is automatic or adaptive tuning of $r_0(s,a)$7 at the level of a user or conversation. Another is richer reward decompositions beyond question versus preference, such as multi-objective decompositions over safety, helpfulness, and faithfulness. A third is broader integration with more advanced TTS methods, including MCTS or value-guided search, now that a cheap segment-level reward is available. These directions suggest that REAR is best understood as a general test-time scoring primitive for alignment-sensitive search, rather than as a closed-form solution to preference personalization.