Bi-directional Preference Optimization

• Bi-directional Preference Optimization is a dual-feedback method that reinforces preferred outcomes and suppresses undesired ones using paired or contrastive data.
• It employs a unified objective with both positive and negative feedback, integrating KL regularization to maintain model stability.
• The approach is key in LLM steering, reinforcement learning from human feedback, and Bayesian optimization, offering robust and reversible control.

Bi-directional Preference Optimization (BiPO) refers to a principled paradigm for learning preference-driven control in machine learning models by simultaneously optimizing to encourage desired behaviors and suppress undesired ones. Rather than focusing on unidirectional maximization along a single feedback axis (e.g., only promoting “good” outputs), bi-directional preference optimization explicitly balances advancement toward a target (preferred) direction and retreat from its opposite (dispreferred) direction, using paired or contrastive data. This approach has emerged as a critical enabling technology in LLM alignment, model steering, reinforcement learning from human feedback (RLHF), and preference-based optimization in control and Bayesian settings (Cao et al., 2024, Abdolmaleki et al., 2024, Wang et al., 20 Dec 2025, Lin et al., 2022).

1. Core Mathematical Formulation

All bi-directional preference optimization methods are built on the premise of joint positive (S=1) and negative (S=0) feedback (whether paired or unpaired), with objectives that simultaneously increase probability or utility on preferred examples and decrease it on dispreferred ones.

A general objective has the form: $$J_{\text{bi-dir}}(\theta) = \alpha\,\mathbb{E}_{y^+}\bigl[\log\pi_\theta(y^+|x)\bigr] - (1-\alpha)\,\mathbb{E}_{y^-}\bigl[\log\pi_\theta(y^-|x)\bigr] - \beta\,\text{KL}[\pi_{\text{ref}}(\cdot|x)\;\|\;\pi_\theta(\cdot|x)]$$ where $y^+$ are preferred, $y^-$ are dispreferred, $\alpha\in[0,1]$, and $\beta>0$ is a KL regularization parameter anchoring the trained policy $\pi_\theta$ to a reference policy $\pi_{\text{ref}}$ (Abdolmaleki et al., 2024). In the paired setting, bi-directional objectives typically embed both increases in probability of $y^+$ and decreases in that of $y^-$. In unpaired settings, mixed positive and negative batches allow for learning from only one feedback type if needed.

BiPO as formulated for LLM steering directly modulates the probability ratio assigned to target vs. opposite responses under explicit transformations of hidden states (Cao et al., 2024): $$\mathcal{L}_{\mathrm{BiPO}}(v) = -\mathbb{E}_{d\sim\{\pm1\},(q,r_T,r_O)}\Bigl[ \log\sigma\Bigl(d\,\beta\log\frac{\pi(r_T\mid \mathcal{A}(q)+d v)}{\pi(r_T\mid \mathcal{A}(q))} - d\,\beta\log\frac{\pi(r_O\mid \mathcal{A}(q)+d v)}{\pi(r_O\mid \mathcal{A}(q))}\Bigr) \Bigr]$$ Here $y^+$0 is a steering vector, and $y^+$1 randomly flips the optimization direction to ensure symmetry between $y^+$2 and $y^+$3.

2. Application to LLM Steering

Bi-directional preference optimization is directly operationalized for LLM model steering via dynamic intervention on transformer activations.

• Steering Vector Learning: A single learnable vector $y^+$4 is trained such that its addition to the hidden state increases the probability of the preferred (“target”) response while simultaneously suppressing the opposite (“contrastive”) response, given a dataset of contrastive preference pairs.
• Bi-directional Training: During optimization, the sign of $y^+$5 is randomly flipped (i.e., $y^+$6), ensuring $y^+$7 and $y^+$8 encode opposing behaviors with equal fidelity (Cao et al., 2024). This addresses limitations observed in single-direction extraction methods (e.g., CAA), including failure to robustly steer in alignment-critical settings (e.g., jailbreaking).
• Inference-Time Control: After optimization, $y^+$9 can be injected with arbitrary scale $y^-$0 into the specified transformer layer. The steering effect is continuous and reversible: $y^-$1 produces pro-target behavior; $y^-$2 produces pro-opposite behavior; $y^-$3 tunes intensity.

This approach yields highly effective, lightweight, and transferable steering on open-ended generation (personas, hallucination control, truthfulness, jailbreak defense) (Cao et al., 2024).

3. Extensions in Preference-Based Policy Optimization and RL

Bi-directional preference modeling is also fundamental in probabilistic and reinforcement learning settings:

• EM-based Policy Optimization: A comprehensive EM-based formulation permits learning from both positive and negative, paired or unpaired feedback (PMPO), overcoming instability present in classic negative-only reward minimization (Abdolmaleki et al., 2024). The lower-bound guarantees stable, interpretable updates even in scenarios with only one feedback polarity.
• Balanced Margin and Sample Weighting: In variants such as Balanced Preference Optimization (BPO), a margin function is redefined to optimize for both chosen and rejected responses:

$y^-$4

maximizing log-likelihood on preferred $y^-$5 while suppressing $y^-$6 and dynamically balancing both via parameter $y^-$7 (Sun et al., 4 Jun 2025).

• Dual-Perspective Approaches: Omni-DPO adaptively weights preference pairs by both external data quality and current model fit, thereby down-weighting noisy or well-learned examples and focusing updates on informative, underexplored regions (Peng et al., 11 Jun 2025).
• Directional Preference Alignment: The DPA framework introduces bi-directional control in multi-objective alignment by modeling user preferences as directions in reward space, supporting seamless trade-off navigation across objectives (Wang et al., 2024).

4. Bi-Directional Optimization in Black-Box and Bayesian Settings

In black-box or simulation-based optimization, bi-directional preference methods employ pairwise (two-point) comparisons to recover efficient gradient estimates from only ordinal feedback:

• Two-Point SGD: At each step, propose a perturbation and use the result of pairwise comparison (which is better?) to construct a gradient estimator; this provides unbiased direction estimates for stochastic or nonconvex objectives using binary (“bi-directional”) queries (Wang et al., 20 Dec 2025).
• Preference-Guided Bayesian Optimization: Alternating between preference-exploration (eliciting new pairwise comparisons for utility learning) and experiment selection (optimizing an acquisition function with respect to the learned utility model) forms a bi-directional loop for efficient multi-objective optimization with vector-valued outcomes (Lin et al., 2022).

5. Theoretical Guarantees and Practical Implications

Bi-directional preference optimization presents several substantial theoretical and practical guarantees:

• Stability with Negative-Only Feedback: Unlike classic reward-minimization, EM-based bi-directional approaches with KL anchoring are provably stable when operating on negative examples alone, provided the anchoring is sufficiently strong (Abdolmaleki et al., 2024).
• Lower Bounds on Likelihoods: Balanced objectives such as BPO ensure that the likelihood of chosen responses cannot fall below a fixed threshold—eliminating the “degraded chosen responses” issue inherent to standard DPO (Sun et al., 4 Jun 2025).
• Gradient Targeting: Methods compute updates that focus exclusively on the harder branch (chosen improvement or rejected suppression), increasing training efficiency and clarity (Sun et al., 4 Jun 2025).
• Empirical Superiority: Across LLM alignment, RL, control, and optimization domains, bi-directional approaches are consistently shown to outperform single-direction objectives, yielding broader behavioral control, better generalization, and stronger safety/robustness properties (Cao et al., 2024, Abdolmaleki et al., 2024, Sun et al., 4 Jun 2025).

6. Vector Arithmetic and Model Generalization

Bi-directional preference frameworks support compositionality and transfer at both the policy and hidden-state intervention levels:

• Vector Arithmetic in LLM Steering: Learned steering vectors can be algebraically combined (e.g., $y^-$8) to produce blended behaviors not directly seen in training (Cao et al., 2024).
• Cross-Model and Language Transfer: Bi-directionally optimized steering vectors generalize across models, LoRA adapters, and even languages, with effectiveness preserved on previously unseen distributions (Cao et al., 2024).
• Multi-Objective Trade-Offs: In DPA, preference vectors can be tuned at inference-time, sweeping the Pareto frontier without retraining (Wang et al., 2024).

7. Relation to Model Steering, Causal Intervention, and Alignment

Recent advances in causal analysis and distribution-matching interventions have highlighted the complementarity between bi-directional preference optimization and methods such as Concept DAS (CDAS):

• Distribution Matching vs. Probability Maximization: CDAS aligns intervened output distributions in both directions between base and concept-positive inputs, and leverages distributed interchange interventions to enable robust, faithful, bi-directional steering by capturing causal directions in hidden state space (Bao et al., 5 Feb 2026).
• Robustness and Faithfulness: Distribution-matching, weakly-supervised factor selection, and bi-directional loss objectives collectively favor interventions that are stable across scales, model sizes, and diverse behavioral axes, and avoid the overfitting, fragility, or artifact generation seen with single-sided preference optimization.

The main technical and empirical contributions summarized here originate from:

• "Personalized Steering of LLMs: Versatile Steering Vectors Through Bi-directional Preference Optimization" (Cao et al., 2024)
• "Learning from negative feedback, or positive feedback or both" (Abdolmaleki et al., 2024)
• "Preference-based optimization from noisy pairwise comparisons" (Wang et al., 20 Dec 2025)
• "Balanced Preference Optimization" (Sun et al., 4 Jun 2025)
• "Omni-DPO: A Dual-Perspective Paradigm for Dynamic Preference Learning of LLMs" (Peng et al., 11 Jun 2025)
• "Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards" (Wang et al., 2024)
• "Faithful Bi-Directional Model Steering via Distribution Matching and Distributed Interchange Interventions" (Bao et al., 5 Feb 2026)
• "Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes" (Lin et al., 2022)