Preference Vector Fusion Techniques

• Preference vector fusion is a set of techniques that combine heterogeneous preference signals from multiple models to form a coherent training objective.
• Methods such as weighted supervised fine-tuning and dense preference optimization leverage reward models and soft aggregation to achieve lower variance gradients and improved alignment.
• These approaches are applied in LLM ensembling and Bayesian optimization, demonstrating enhanced stability, robustness, and data efficiency in multi-source decision-making.

Preference vector fusion encompasses a set of methodologies for combining preference signals—often in the form of “preference vectors,” pairwise preference relations, or probability distributions—from multiple models or agents into a coherent training objective for a single target model. This approach is foundational in the fusion of heterogeneous LLMs, multi-source policy optimization, and interactive decision-making with multiple outcome dimensions. Modern techniques for preference vector fusion leverage reward models, probabilistic inference, and weighted aggregation to distill diverse model capabilities, yielding denser, lower-variance gradients and improved preference alignment across a range of domains.

1. Foundational Principles and Definitions

Preference vector fusion refers to techniques for integrating multiple preference signals—originating from models, reward functions, or decision-makers—such that the resultant fused policy or model captures the combined strengths or priorities of the sources. In LLM fusion, this concretely involves aggregating either:

• Sequence-level log-probabilities or response likelihoods from N source models, yielding a fused probabilistic reference (Gu et al., 20 May 2025).
• Reward-model–scored vectors of preference values for sampled completions per model (termed "preference vectors") (Zhong et al., 9 Apr 2025, Yang et al., 6 Mar 2025).
• Pairwise preference comparisons over vector-valued outcomes, forming a compositional learning structure in Bayesian frameworks (Lin et al., 2022).

Formally, for source models $M_i^s$ assigning probabilities to response $y$ given prompt $x$, and weights $\gamma_i$, the fused probability is

$$p_\text{fused}(y|x) = \prod_{i=1}^N [M_i^s(y|x)]^{\gamma_i}$$

as introduced in the InfiFPO framework (Gu et al., 20 May 2025). For reward-based fusion, preference vectors $\rho_s(x)$ for model $s$ and prompt $x$ are given by applying a reward model to sampled responses (Yang et al., 6 Mar 2025).

2. Methodologies in LLM Fusion

The primary paradigms for preference vector fusion in LLMs are two-stage training pipelines:

1. Weighted Supervised Fine-Tuning (SFT): Candidate responses from multiple source models are scored by an external reward model. Rather than training on only the best response, FuseRL (and similarly, FuseChat-3.0) uses all high-scoring responses, assigning softmax-normalized weights $w_{x,i}$ (proportional to $\exp(r(x,y_i)/\alpha)$) to each source’s best candidate for each prompt $y$0 (Zhong et al., 9 Apr 2025). SFT loss takes the form:

$y$1

This encourages the target model to interpolate among all high-quality source outputs, improving robustness and mitigating source-model idiosyncrasies (Zhong et al., 9 Apr 2025, Yang et al., 6 Mar 2025).

2. Preference Optimization (PO): In the second stage, a preference-learning loss (e.g., DPO, SimPO, RLOO) is applied to pairs consisting of the “best” and “worst” responses per model; weights $y$2 carry over (Zhong et al., 9 Apr 2025). The DPO loss is expressed as:

$y$3

where $y$4 is the sigmoid. Length-normalized DPO and batch-specific sampling strategies, as well as fusion of multiple preference pairs per prompt, further densify and stabilize the gradient signal (Yang et al., 6 Mar 2025).

InfiFPO (Gu et al., 20 May 2025) extends this principle by replacing the standard DPO reference with a sequence-level fused probability, using geometric mean aggregation of source probabilities and introducing stabilization through probability clipping and max-margin fusion.

3. Stabilization and Signal Densification Strategies

Several techniques are employed to ensure stability and effective knowledge transfer when fusing preference signals:

• Length Normalization:

To account for variable tokenization lengths across responses, preference losses are normalized over sequence length, i.e., $y$5 (Gu et al., 20 May 2025, Yang et al., 6 Mar 2025).

• Probability Clipping:

Source model probabilities for preferred and dispreferred outputs are clipped with respect to the pivot model’s initialization. This prevents degenerate gradients from over-dominant sources and stabilizes optimization (Gu et al., 20 May 2025).

• Max-Margin Fusion:

InfiFPO’s max-margin approach adaptively selects the source whose sequence probability deviates most from the current pivot, maximizing informativeness per batch (Gu et al., 20 May 2025).

• Soft Aggregation of Preference Vectors:

Unlike hard-selection, frameworks like FuseRL sum losses over all K source-derived preference pairs, each weighted by per-prompt model quality, producing denser optimization signals and lower gradient variance (Zhong et al., 9 Apr 2025). FuseChat-3.0 pools best-vs-worst pairs intra-source to avoid style bias and assembles a large, composable DPO dataset (Yang et al., 6 Mar 2025).

A summary of key strategies appears in the table:

Technique	Implementation Context	Purpose
Length Normalization	InfiFPO, FuseChat-3.0	Corrects for response length
Probability Clipping	InfiFPO	Prevents unstable gradients
Max-Margin Fusion	InfiFPO	Maximizes per-batch learning
Soft-max Weighting	FuseRL	Densifies preference signals
Uniform Preference Pooling	FuseChat-3.0	Ensures broad coverage

4. Preference Vector Fusion in Bayesian Optimization

Beyond LLMs, preference vector fusion is operationalized within Bayesian optimization procedures involving vector-valued outcomes and a latent decision-maker utility function (Lin et al., 2022):

• Compositional Probabilistic Modeling:

The unknown outcome function $y$6 and the DM’s utility $y$7 are modeled as GPs. The composite objective is to maximize $y$8 over $y$9 (Lin et al., 2022).

• Pairwise Preference Learning:

The DM compares pairs of outcome vectors, producing a set of preference queries $x$0. These are modeled via a Thurstone–Mosteller (probit) likelihood, and the posterior over $x$1 is approximated through Laplace’s method.

• Acquisition under Uncertainty:

The fused knowledge from experiments and human feedback is exploited via composite acquisition functions (e.g., $x$2NEIUU, EUBO), which quantify expected improvement in the true composite utility by integrating over the GP posteriors for both $x$3 and $x$4.

Simulation studies demonstrate that such fusion, particularly when using EUBO-based preference query selection, yields superior sample efficiency and optimization performance versus standard multi-objective or uni-dimensional preference optimization (Lin et al., 2022).

5. Empirical Results and Practical Impact

Empirical results demonstrate that preference vector fusion, implemented through weighted SFT and dense preference optimization, confers consistent gains in LLM benchmarks:

• InfiFPO raises Phi-4's average performance from 79.95 to 83.33 across 11 tasks, with notable improvements in mathematics (+2.94), code (+5.68), and reasoning (+4.06) metrics over strong prior baselines (Gu et al., 20 May 2025).
• FuseRL achieves a win rate of 70.1% on AlpacaEval-2 (vs. 67.1% for SFT+DPO) and reduces bias and variance in the derived policy (Zhong et al., 9 Apr 2025).
• FuseChat-3.0 demonstrates a 6.8-point average improvement across 14 benchmarks, and large absolute gains on instruction-following and reasoning tasks (Yang et al., 6 Mar 2025).

Statistical and ablation analyses support the view that denser, multi-source gradient signals—enabled by preference vector fusion—enable more stable generalization, robust aggregation of heterogeneous expertise, and improved downstream alignment.

6. Theoretical and Practical Considerations

Preference vector fusion introduces several nuanced trade-offs:

• Variance Reduction:

Aggregating multiple preference signals, contemporaneously weighted by external reward models, theoretically reduces estimation variance while preserving gradient unbiasedness (Zhong et al., 9 Apr 2025).

• Automation vs. Human-in-the-Loop:

In Bayesian optimization, fusing posterior estimates of outcome and utility enables tight integration of automated experimentation with interactive preference elicitation, outperforming traditional multi-objective strategies (Lin et al., 2022).

• Weighting and Normalization:

Empirical findings in FuseChat-3.0 indicate that uniform weighting over intra-source preference pairs is sufficient; heuristic re-weighting did not empirically improve results (Yang et al., 6 Mar 2025).

A plausible implication is that as preference vector fusion techniques mature—with improved reward models, adaptive weighting, and more expressive preference signal representations—the ability to compress heterogeneous capabilities into compact, high-performing policies is likely to increase.

7. Related Research Directions and Extensions

Preference vector fusion connects to broader themes in model distillation, reward aggregation, and human preference learning. Contemporary works extend these methodologies to:

• Complex multi-objective decision-making and design optimization.
• Heterogeneous expert model ensembling under preference uncertainty.
• Interactive data collection protocols leveraging EUBO and qNEIUU for simultaneous experiment selection and preference learning (Lin et al., 2022).

Current research distinguishes itself by moving from “select-one” fusion—using only the single best or highest-probability output from each model—to dense vector fusion, exploiting all available preference information and thereby densifying the learning signal space (Zhong et al., 9 Apr 2025, Yang et al., 6 Mar 2025). This signals a shift toward more data-efficient and robust knowledge aggregation in both LLM and multi-objective optimization settings.