Factorized Preference Optimization (FPO) Algorithm

• Factorized Preference Optimization (FPO) is a method that decomposes preference signals over distinct subspaces to target localized misalignments.
• It is applied in zero-shot text-to-speech, temporally grounded video-language, and LLM alignment to enhance data efficiency, convergence, and control.
• Empirical evaluations show significant improvements, such as reduced error rates in TTS and better temporal grounding, with lower computational overhead.

Factorized Preference Optimization (FPO) refers to a family of preference-based learning algorithms in which the objective or regularization explicitly decomposes, or "factorizes," over multiple task-relevant axes or feature spaces rather than treating the preference signal as a monolithic score over entire outputs. Distinct variants of FPO have been developed for various domains, including zero-shot text-to-speech (TTS) synthesis, temporally grounded video-language modeling, and LLM alignment. The unifying motivation is to selectively and efficiently propagate human (or synthetic) preference supervision to the precise subspaces—tokens, frames, features, or events—responsible for observed failures, enabling more targeted, data-efficient, and controllable model updates.

1. Theoretical Foundations and Motivation

Traditional preference optimization methods such as Reinforcement Learning from Human Feedback (RLHF) with PPO or Direct Preference Optimization (DPO) assign the preference reward uniformly over entire outputs—utterances, completion sequences, or entire events. However, many practical alignment failures are highly localized: in TTS, for example, isolated segmental mispronunciations dominate user annoyance, while in video-LLMs, both evidence grounding and textual answer quality form a logical hierarchy and should be reasoned about explicitly but separately.

FPO addresses these limitations by factorizing the learning signal to focus only on the components, segments, or features explicitly implicated by preference judgments. This selective approach can improve data efficiency, accelerate convergence, and enable finer-grained model control while often reducing computational overhead relative to token-level KL-based regularization (Yao et al., 5 Feb 2025, Zeng et al., 30 Dec 2025, Yin et al., 2024).

2. Algorithmic Formulations

Fine-grained Preference Optimization (TTS)

In the context of zero-shot TTS (Yao et al., 5 Feb 2025), FPO reformulates the DPO loss: $$\mathcal{L}_{\mathrm{FPO}} = -\,\mathbb{E}_{(x,y_w,y_l)} \sum_{i=1}^L I(y^i) \log \sigma\left( \beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)} \right)$$ where $I(y^i)$ is a token-level indicator marking whether acoustic token $y^i$ falls within human-annotated error segments. Only problematic segments or tokens receive loss, leaving already well-modeled parts unaffected.

Factorized FPO for Video-Language Temporal Grounding

In temporally grounded video-LLMs (Zeng et al., 30 Dec 2025), FPO extends the conventional preference loss to jointly factor over both text token likelihoods and explicit probabilistic temporal grounding: $\log \pi(R) = \sum_{R_i\in R} \log p(R_i|V,Q,R_{1:i-1}) + \sum_{k: R_k = \text{\textlangle evi\textrangle}} \log p_g([s_k,e_k])$ where $p_g([s_k, e_k])$ is the model's predicted probability for an evidence interval, connecting output tokenization with explicit video grounding.

Feature-level Constrained FPO for LLM Alignment

In the LLM alignment setting (Yin et al., 2024), FPO replaces token-level KL with a sparse feature-level mean squared error (MSE) constraint via a pretrained Sparse Autoencoder (SAE): $$D^\ell_{\rm FPO}(x,y;\pi_{\rm ref}\|\pi_\theta) = \frac{1}{k}\sum_{i\in I_k}(\bar c^\ell_{\theta,i} - \bar c^\ell_{\rm ref,i})^2$$ This constraint is incorporated into a unified preference-optimization framework, enforcing alignment between the main policy and the reference in terms of high-activation, monosemantic SAE features rather than full token distributions.

3. Application-specific Instantiations

Domain	Factorization Mode	Targeted Component(s)	Reference Paper
Text-to-Speech	Token/time-segment	Localized error regions	(Yao et al., 5 Feb 2025)
Video-Language	Text + event grounding	Both answering and temporal evidence	(Zeng et al., 30 Dec 2025)
LLM Alignment	Layerwise sparse features	High-activation hidden representations	(Yin et al., 2024)

• TTS: Error segment detection supports correction of specific phonetic, prosodic, or alignment issues. Data annotation isolates temporal modeling errors and semantic-phonetic misalignments, ensuring only defective acoustic tokens are penalized.
• Video-Language: Explicit modeling of evidence tokens and probability of event inclusion supports hierarchical learning: first ground events, then generate text responses.
• LLM Alignment: SAE feature activation provides an efficient, interpretable basis for enforcing similarity between the aligned model and the (cached) reference, with no need for online KL computation.

4. Empirical Evaluation and Comparative Analysis

FPO demonstrates substantial improvements over conventional preference optimization methods on diverse tasks:

• TTS (Zero-Shot Mandarin/English): Character Error Rate (CER) reduced by 52.5%, Word Error Rate (WER) reduced by 54.8%. Bad-case ratio also reduced by more than a factor of three. Data efficiency is improved: 200 utterances suffice to match performance of utterance-level methods trained on 800 (Yao et al., 5 Feb 2025).
• Video-Language (E.T. Bench): Average F1 improvements of +2.8 on event grounding and +2.5 on dense captioning. Overhead added by framewise grounding is less than 1.4% decode time (Zeng et al., 30 Dec 2025).
• LLM Alignment (AlpacaEval-2, Arena-Hard): On Gemma-2-2B, FPO yields up to +5.08% win rate improvement over SFT and beats DPO, SimPO, and TDPO baselines. Training time and GPU memory are up to 17–18% lower than token-level KL-based TDPO2 (Yin et al., 2024).

Ablation studies reveal that the factorized supervision and constraints are essential: reverting to uniform (utterance- or sequence-level) losses or omitting factor-specific constraints significantly degrades performance and generalization.

5. Architecture and Training Procedures

Each FPO instantiation leverages architecture- and domain-specific workflows:

• Fine-grained TTS FPO: Uses CosyVoice, a decoder-only Transformer LM with neural codec tokenization, training on error-segment-annotated preference pairs with maximization only over non-conforming tokens. The reference checkpoint is frozen, and small learning rates ($\sim$1e–6 to 1e–5) prevent catastrophic forgetting (Yao et al., 5 Feb 2025).
• Video-Language FPO: Employs a "grounding then answering" paradigm, using evidence tokens, explicit frame-wise grounding similarity, and synthetic preference pairs generated by controlled event perturbations. Losses combine joint text/grounding, consistency, and framewise binary cross-entropy (Zeng et al., 30 Dec 2025).
• LLM Alignment FPO: Relies on a frozen SAE trained on hidden states to yield highly sparse activations, which are then pooled and penalized via MSE in post-alignment preference optimization. Both margin and activation statistics from the reference model are cached offline, eliminating online reference passes (Yin et al., 2024).

6. Hyperparameter Sensitivity, Trade-offs, and Limitations

Key hyperparameters across FPO variants include:

• $\beta$ (Preference Weighting): Controls the trade-off between retaining reference model behavior and fitting strong new preferences. Both excessively low and high $\beta$ degrade targeted alignment (Yao et al., 5 Feb 2025).
• Constraint Strengths ($\alpha$, top-$k$ SAE features): Impacts the balance between efficiency and the fidelity of the feature-level constraint. Empirical tuning for SAE insertion layer $\ell$ and $\alpha$ is required for best results (Yin et al., 2024).
• Data Annotation Quality: Fine-grained annotation is more labor-intensive than utterance- or sequence-level ratings. Exploring automatic or weakly supervised strategies is a leading open problem (Yao et al., 5 Feb 2025).

Limitations and future directions include the exploration of multilayer or multimodal feedback (e.g., visual or lip-synchrony cues in TTS), synthetic generation of "hard positive" preference pairs, improved theoretical guarantees for the equivalence of feature-MSE and KL divergence, and extension to global attributes (speaker style, emotion) (Yao et al., 5 Feb 2025, Zeng et al., 30 Dec 2025, Yin et al., 2024).

7. Significance and Outlook

FPO offers a unifying methodological framework for targeted, efficient, and interpretable preference-based learning. By focusing the optimization signal on relevant subspaces—whether segmental tokens in TTS, temporal events in video, or sparse SAE activations in LLMs—FPO addresses the inefficiencies and optimization dilution inherent in global preference aggregation. The technique has demonstrated empirical robustness and competitive computational scaling across modalities, motivating further research into multimodal factorized feedback, hierarchical constraints, and synthetic data pipelines for preference learning at scale (Yao et al., 5 Feb 2025, Zeng et al., 30 Dec 2025, Yin et al., 2024).