TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

Abstract: Direct Preference Optimization (DPO) is a widely used RL-free method for aligning LLMs from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley-Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a token-level Bradley-Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization benchmarks, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.

• The paper introduces TBPO, which applies density-ratio matching at the token level to optimize preference alignment via a Bregman-divergence minimization framework.
• It presents two instantiations, TBPO-Q and TBPO-A, leveraging state-action value and advantage functions to address token-level credit assignment effectively.
• Empirical evaluations show TBPO's superior performance in alignment, reasoning, and diversity across benchmarks, outperforming existing sequence-level methods.

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

Motivation and Background

Direct Preference Optimization (DPO) has become a canonical RL-free method for aligning LLMs via sequence-level pairwise preferences. However, autoregressive text generation in LLMs is fundamentally token-level: each generation step is a local decision conditioned on the prefix. Most sequence-level preference optimization approaches, including DPO and its recent token-level derivatives (e.g., TDPO, TIS-DPO), rely on distributing sequence-level signals across timesteps rather than enforcing explicit token-wise optimality. This leads to imprecise credit assignment and a mismatch between optimization and inference, especially for long sequences where early-token errors propagate.

Methodological Framework

The paper introduces Token-level Bregman Preference Optimization (TBPO), which applies density-ratio matching at the token granularity, leveraging sequence-level comparison data. Central to TBPO is a token-level Bradley--Terry (BT) model, which formalizes token-level preference probabilities via state-action value or advantage functions under a reference policy. The optimization objective is derived via Bregman-divergence ratio matching, generalizing logistic (DPO) loss and maintaining policy optimality at the token level with DPO-like simplicity.

Two instantiations are proposed:

• TBPO-Q: Utilizes reference-policy state-action value ($Q$) as the token score. $w_t$ is estimated via a learned baseline head predicting the partition function.
• TBPO-A: Utilizes reference-policy advantage ($A$) as the token score. $w_t$ reflects the difference in KL-divergence baselines, estimated via a Monte-Carlo KL estimator.

Both versions address the independence of states (prefixes) in token-level comparisons through these state-only corrective weights, ensuring alignment of density ratios across prefixes.

The loss function is formulated as a Bregman-divergence between empirical and model density ratios at each token step, averaged over all timesteps. Different Bregman generators recover various loss families, and the paper employs Scaled Basu's power divergence for tuning flexibility.

A core theoretical result establishes that minimizing TBPO's density-ratio matching objective recovers the token-level policy optimal for a KL-regularized advantage-maximizing objective, under sufficient model capacity.

Empirical Evaluation

TBPO is tested across multiple alignment-sensitive and reasoning benchmarks with Mistral 7B v0.1 and Llama 3 8B backbones, and compared against SFT, DPO, TDPO, TIS-DPO, and BPO-SBA. Sequence-level DPO delivers visible improvements over SFT, but TBPO achieves the highest average scores across evaluated tasks. On reasoning-intensive datasets (e.g., GSM8K, MMLU, Winogrande), TBPO-Q and TBPO-A yield the best results, with TBPO-Q exhibiting strong gains in math and logic tasks.

For preference alignment in multi-turn dialogue (MT-Bench) and safety-relevant domains (Anthropic HH-RLHF), TBPO consistently achieves superior win rates, outperforming all baselines. Notably, TBPO produces shorter generations while maintaining higher preference win rates, indicating improved response quality that is not attributable to verbosity.

TBPO also optimizes the diversity–confidence tradeoff: it improves predictive entropy and lexical diversity (Distinct-1) without excessive redundancy (low Self-BLEU), outperforming both baseline preference and SFT models. In out-of-domain settings (TLDR summarization), TBPO generalizes, sustaining high length-controlled win rates and concise outputs despite distribution shift.

Theoretical and Practical Implications

TBPO represents a principled bridge between theoretical reinforcement learning perspectives and practical preference optimization. By enforcing token-level optimality via density-ratio matching, it resolves the credit assignment issue inherent in sequence-level objectives, especially for longer generations. The explicit modeling of token-level preferences provides a tractable yet theoretically justified pathway to improved credit assignment, stability, and diversity, without the instability or complexity of RL-based methods.

Practically, TBPO supports plug-and-play deployment on top of existing SFT checkpoints and scales robustly to large LLM backbones. The learned baseline and KL estimation techniques add negligible computational overhead and are amenable to standard model architectures.

Theoretically, TBPO generalizes the likelihood-ratio rationale underlying DPO and BPO to token-level granularity and supports diverse divergence constraints via Bregman generators. This opens new avenues for custom preference models and fine-tuned diversity control—potentially supporting more adaptive, context-sensitive alignment paradigms.

Future Directions

TBPO's framework invites further exploration in several directions:

• Adoption of richer per-state scoring functions leveraging reward model ensembles, auxiliary criteria, or domain-specific features.
• Extension to more general action/state spaces beyond token-level autoregressive setups, including multimodal or structured output domains.
• Integration with online preference collection mechanisms for live alignment feedback.
• Tightening diversity–fidelity tradeoffs, especially for models deployed in high-sensitivity settings with fine-grained preference structure.

Conclusion

TokenRatio (TBPO) provides a principled, practical, and theoretically justified methodology for token-level preference optimization, unifying density-ratio matching and Bradley--Terry modeling. Empirical results show that TBPO improves alignment, stability, and diversity compared to standard sequence-level and token-level baselines, supporting generalization across domains and tasks. The approach sets a foundation for more robust and theoretically grounded LLM alignment strategies, with broad applicability in instruction following, helpfulness, harmlessness, and summarization contexts.