Reference-free Preference Steering (RePS)

• RePS is a framework using intrinsic preference estimation and representation steering to align agents with specific goals without relying on traditional reward signals.
• It leverages loss-based and similarity metric optimization methods to guide large language models and reinforcement learners in high-dimensional environments.
• Practical applications of RePS include robotics, dialogue systems, and personalized assistants, offering robust, scalable, and interpretable agent alignment.

Reference-free Preference Steering (RePS) encompasses a spectrum of algorithms and frameworks designed to align agents—particularly LLMs and reinforcement learners—with human or task-specific preferences without reliance on explicit reference models, hand-crafted reward functions, or costly binary human feedback. This paradigm addresses the limitations of legacy methods in high-dimensional, open-ended, or reward-sparse environments and leverages intrinsic, data-driven, or representation-based optimization strategies to guide agent behavior.

1. Fundamental Principles and Motivation

RePS is motivated by the challenge of enabling agents to acquire and satisfy preferences without external reward signals, reference policies, or manually annotated comparator datasets. In reinforcement learning, traditional approaches often require a predefined reward signal; in supervised or human-feedback alignment, a reference model is used to guide preference optimization. RePS, on the other hand, employs intrinsic preference estimation, representation steering, or preference-informed loss functions that allow agents to self-organize and adapt. Early work (Sajid et al., 2021) formalized this in a Bayesian framework, where agents learned preferences over states or outcomes through experience and updated latent priors accordingly.

Key principles include:

• Reward-free or intrinsic preference estimation.
• Direct optimization via observable signals—such as length-normalized sequence likelihoods, similarity metrics, or preference deviations.
• Steering via sparse, interpretable feature representations or activation vectors rather than reference models or dense parameter updates.
• Scalability across multi-preference, multi-modal, or dynamic contexts.

2. Algorithmic Approaches and Loss Formulations

Recent RePS methodologies predominantly operate within two classes: loss-based preference optimization and representation-based steering.

Loss-based Preference Optimization

• Implicit Reward Systems: SimPO (Meng et al., 23 May 2024) and RePO (Wu et al., 10 Mar 2025) define the implicit reward for a response as the average log probability of the sequence (length normalization), avoiding bias toward verbosity. The SimPO objective,

$r_{SimPO}(x, y) = (\beta/|y|) \log \pi_\theta(y|x)$

enables reference-free optimization compatible with generation metrics.

• Target Reward Margins and Max-Margin Losses: SimPO enforces a margin $\gamma$ between preferred and non-preferred responses:

$$p(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l) - \gamma)$$

RePO advances this with a ReLU-based max-margin filter:

$$\mathcal{L}_{RePO}(\pi_\theta) = \mathbb{E}_{(x, y_w, y_l)} [ \mathrm{ReLU}(- (M_\theta - \gamma)) ]$$

where $M_\theta$ is the normalized margin. Notably, RePO eliminates the hyperparameter $\beta$ via a limiting argument, yielding robust, hyperparameter-efficient training.

• Deviation-Based Multi-Preference Losses: REFA (Gupta et al., 20 Dec 2024) generalizes reference-free optimization to multi-preference domains, applying deviation-based weighting to boost high-quality outputs:

$$w_y = \exp(\alpha \Delta S_y), \quad \Delta S_y = r_y - \bar{r}$$

Length normalization and EOS-probability regularizers (to handle the "Uncertainty Reduction with Sequence Length Assertion" phenomenon) further enforce informativeness without brevity bias.

• Direct Optimization via Similarity Metrics: RefAlign (Zhao et al., 14 Apr 2025) eschews binary preferences in favor of BERTScore-based similarity to high-quality reference answers. The REINFORCE-style policy gradient is driven by these soft similarity-based surrogates.
• Length and Probability Control: LMPO (Li et al., 20 Feb 2025) introduces loss terms to address length bias and probability degradation, using margin-based loss and statistical normalization (Z-score, average length).

Representation-Based Steering

• Steering Vectors in LLM Residual Streams: Methods like BiPO (Cao et al., 28 May 2024), CONFST (Song et al., 4 Mar 2025), and systems employing activation steering (Bo et al., 7 May 2025) identify directions in the latent activation space that reliably control a model's expression of preferences, style, risk attitude, or topic.
  • BiPO jointly optimizes steer vectors $v$ using a bi-directional contrastive objective, ensuring both forward and reverse controllability.
  • CONFST trains classifiers to selectively average high-confidence user-specific activation directions, enabling multi-preference and style steering.
  • Risk preference steering (Zhu et al., 16 May 2025) entails alignment between behavioral and neural representations via regression, yielding steering vectors for direct activation perturbation.
• Feature Steering with Sparse Autoencoders: FSRL (Ferrao et al., 16 Sep 2025) leverages interpretable, sparse features and adapter networks for transparent preference steering. The adapter modulates SAE-derived conceptual features,

$$x_\text{steered} = \text{Decoder}(f + v) + (x - \text{Decoder}(f))$$

where $f$ are SAE features and $v$ is the steering vector. Mechanistic analysis demonstrates that style features are preferenced over abstract alignment features during optimization, illuminating optimization pathways.

3. Preference Learning and Update Mechanisms

Intrinsic preference learning in RePS typically involves Bayesian or self-supervised update rules:

• Pepper Preference Learning (Sajid et al., 2021): Agents operating in partially observed/volatile environments update Dirichlet (conjugate) priors to encode evidence for visited states/outcomes. For state preferences, the update

$$d_{ij, t} \leftarrow d_{ij, t-1} + \alpha \cdot \mathbf{s}_{ij}$$

accumulates pseudo-counts, with action selection guided by expected free energy planning incorporating these learned priors.

• Listwise, Attribute-Aware Ranking (Yang et al., 15 Feb 2025): SeAdpra quantifies response differences via APDF and dynamically determines ranking order in a self-supervised fashion, eschewing manual pairwise labels.

4. Trade-offs, Capabilities, and Limitations

The reference-free paradigm supports adaptive trade-offs:

• Exploration vs. Preference Satisfaction (Sajid et al., 2021): Agents balance epistemic value and preference satisfaction, as observed in trajectory diversity (Hausdorff distance) and entropy measures. Precision of learned preferences depends on environment volatility.
• Length and Style Biases: Without careful normalization or regularization, preference optimization can unwittingly favor shorter or stylistically enriched responses over purely informative or safe ones (Gupta et al., 20 Dec 2024, Ferrao et al., 16 Sep 2025). EOS regularization and interpretable feature steering can mitigate these effects.
• Steering Robustness and Fine-Grained Control: Vector-based methods (BiPO, CONFST) facilitate real-time, multi-preference alignment, with empirical transferability across models, tasks, and user histories. However, they may be layer-dependent and require access to internal activations.

5. Empirical Results and Benchmarking

Recent empirical evidence substantiates RePS methods:

• SimPO, RePO, and LMPO exceed existing DPO baselines on AlpacaEval 2 and Arena-Hard benchmarks in terms of win rate, length control, and reward accuracy (Meng et al., 23 May 2024, Wu et al., 10 Mar 2025, Li et al., 20 Feb 2025).
• REFA achieves improved LC-WR and WR, indicating effective multi-preference and length-controlled alignment (Gupta et al., 20 Dec 2024).
• RefAlign matches or surpasses binary preference models in safety, general alignment, and calibration (Zhao et al., 14 Apr 2025).
• FSRL delivers comparable preference optimization performance using interpretable steering features and exposes a systematic bias towards stylistic cues (Ferrao et al., 16 Sep 2025).
• Steering vectors validated on benchmarks (AxBench, topic/style shifts) enable robust suppression and steering, including resilience against jailbreaking attacks (Wu et al., 27 May 2025, Song et al., 4 Mar 2025, Cao et al., 28 May 2024).

6. Practical Applications and Future Directions

RePS methodologies are applicable in open-ended learning agents (robotics, dialogue systems, code generation), content moderation (transparent style suppression or enforcement), personalized assistants (multi-dimensional activation steering), and risk-sensitive domains. Practical advantages include computational efficiency (no retraining or reference models), interpretability of steering interventions, and flexible integration with dynamic user preferences.

Future research is poised to address:

• Multi-dimensional, multi-modal or multi-objective preference optimization, extending beyond single-task scenarios (Kim et al., 10 May 2025).
• Theoretical improvement of loss formulations (margin types, normalization) and stability in high variance settings (Li et al., 20 Feb 2025).
• Exploration of internal representation landscapes for alignment diagnostics and more principled steering (Ferrao et al., 16 Sep 2025).
• Hybridization with human feedback pipelines or online learning scenarios to handle non-stationary preference dynamics.

7. Summary Table of Major RePS Algorithms

Algorithm / Framework	Key Mechanism	Distinguishing Feature
SimPO (Meng et al., 23 May 2024)	Avg. loglikelihood margin	Reference-free, efficient
RePO (Wu et al., 10 Mar 2025)	ReLU-based max-margin	β-free, hard filtering
REFA (Gupta et al., 20 Dec 2024)	Multi-pref, deviation, EOS	Multi-dimensional, length control
BiPO (Cao et al., 28 May 2024)	Bidirectional steer vector	Transferability, compositionality
CONFST (Song et al., 4 Mar 2025)	Classifier-based direction	Multi-preference, no explicit user input
FSRL (Ferrao et al., 16 Sep 2025)	SAE + adapter	Feature transparency, mechanistic analysis
RefAlign (Zhao et al., 14 Apr 2025)	Similarity-based reward	Reference answer, soft metric only
LMPO (Li et al., 20 Feb 2025)	Length-controlled margin loss	Probability stability, robust normalization

In sum, Reference-free Preference Steering provides the technical foundation and empirical evidence for robust, interpretable, and scalable alignment of artificial agents without traditional reference dependence—encompassing direct optimization, activation steering, multi-modal ranking, and feature-based interventions. These advances collectively expand the horizon for adaptive, safe, and user-aligned AI systems.