Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

Abstract: Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of LLMs at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.

• The paper introduces a novel test-time alignment approach using gradient-based editing guided by a multi-objective value function.
• It integrates regression, margin, and regularizer losses to incorporate pairwise human preferences into frozen LLM activations, achieving improved alignment and generalization.
• Empirical results demonstrate that Pref-CTRL outperforms RE-Control and DPO in win rates, diversity, and zero-shot performance on multiple datasets.

Preference-Based Test-Time Alignment for LLMs via Representation Editing

Overview of Pref-CTRL

Pref-CTRL introduces a test-time, preference-driven alignment strategy for LLMs, based on direct editing of frozen model representations using a multi-objective value function. Unlike RLHF and fine-tuning paradigms, Pref-CTRL intervenes at inference-time, leveraging lightweight gradient-based control over internal activations while maintaining computational feasibility and flexibility. The core innovation is explicit incorporation of pairwise human preference data in the construction of the value function, which better reflects the underlying structure of alignment benchmarks.

Methodological Contributions

Most prior test-time alignment methods, notably RE-Control, utilize value functions trained on scalar alignment scores, omitting the relational structure of preference feedback that underpins standard datasets (e.g., SHP, HH-RLHF). Pref-CTRL closes this methodological gap by introducing a value function objective comprising three terms:

1. Regression Loss: Traditional reward regression to encourage alignment with scalar rewards.
2. Margin Loss: Directly enforces the separation between value predictions of preferred and rejected completions in each preference pair, implemented as a log-sigmoid cross-entropy margin objective.
3. Regularizer Loss: Penalizes deviations between the value function scores of an actually generated hidden state and its corresponding preferred reference, mitigating over-optimization ("reward hacking") and maintaining fluency.

This multi-objective design introduces direct inductive bias toward accurate preference-based ranking, which regularizes the editing procedure and achieves better tradeoffs between harmlessness, diversity, and coherence.

Figure 2: Pref-CTRL architecture in which hidden states labeled as preferred, rejected, or LLM-generated are extracted from the frozen LLM to compute the multi-term value objective.

During inference, Pref-CTRL applies iterative gradient-based editing to the last-layer hidden state, as in RE-Control, but using the preference-informed value function. Hyperparameters such as the number of editing steps $k$ and step size $\alpha$ are tuned based on validation, and sensitivity analyses confirm the robustness of high performance to these choices.

Empirical Results

Extensive evaluations benchmark Pref-CTRL against RE-Control and the fine-tuned DPO baseline. Experiments are conducted on SHP and HH-RLHF datasets with Vicuna-7B and Hermes3-8B as base models, leveraging UltraRM as the reward model.

Key findings include:

• On SHP (Vicuna-7B), Pref-CTRL achieves a Llama-judge win rate increase from 66.8% (RE-Control) to 73.5% with the margin + regularizer objective, compared to 71.8% for DPO.
• Similar trends are observed on HH-RLHF and with Hermes3-8B, with consistent and statistically significant gains in win rate and average reward.
• Diversity and coherence metrics remain stable or improve slightly, indicating that improved alignment does not harm output quality.
• Out-of-domain generalization tests (PKU-SafeRLHF, Nectar) show that value functions trained with Pref-CTRL provide systematic improvements over RE-Control in zero-shot settings.
• Against alternative test-time methods such as Best-of-N sampling and CAST, Pref-CTRL delivers competitive or superior alignment scores without requiring multiple candidate generations.

For illustrative analysis, the authors provide concrete prompt–output pairs. Pref-CTRL better avoids harmful or unethical generations and maintains more neutral or instructive responses compared to baseline editing methods, as highlighted by qualitative outputs.

Analysis of Sensitivity and Ablations

Sensitivity analysis (Figure 3) reveals that reward metrics are maximized around $\alpha = 0.5$ and $k = 100$, with little degradation across a moderate neighborhood, confirming that the method's efficacy stems from the preference-driven loss rather than fragile hyperparameter choices.

Figure 1: Test-time alignment reward sensitivity to the number of editing steps, showing optimal win rates at $k=100$.

Ablation studies indicate that the margin loss alone can lead to over-optimization, reducing generalization, while the addition of the regularizer restores stability and prevents reward hacking phenomena.

Implications and Future Directions

Pref-CTRL demonstrates that test-time, value-function-based alignment approaches can approach or match fine-tuned (training-time) preference optimization methods, without the overhead of modifying model weights. The framework is practical for plug-in deployment in settings where access to training compute or data is limited, or for rapid adaptation to dynamic or unseen preferences.

From a theoretical standpoint, the results support the hypothesis that the inductive bias of preference-driven multi-objective training improves the reliability and robustness of representation editing strategies. Such an approach could plausibly transfer to other modalities or conditional generation tasks where preference structure is available.

Further research avenues include integrating more expressive value function architectures (e.g., attention-based models), combining multi-attribute (multi-objective) preferences, and investigating adaptive step-size or intervention schedules. Extensions to multi-turn and long-context dialogue remain areas for exploration.

Conclusion

Pref-CTRL presents a formally elegant and empirically effective improvement to inference-time LLM alignment, grounding representation editing in the preference structure of standard alignment datasets. It achieves strong alignment and generalization performance, moving test-time editing approaches into competitive parity with fine-tuned baselines. Limitations remain in hyperparameter sensitivity and expressiveness beyond the preference attributes present in the training data, though the evidence suggests considerable promise for preference-based control in flexible alignment workflows.