Preference Tokens for Efficient LLM Alignment

• Preference tokens are the key tokens carrying concentrated human-preference signals, identified by comparing oracle and reference log probability ratios.
• The SePO method uses an oracle model trained via DPO to score tokens and selectively optimizes only the top or bottom k% of tokens, enhancing training efficiency.
• Empirical results show that optimizing roughly 30% of tokens improves alignment performance and mitigates noise from uniformly updating non-critical tokens.

Selective Preference Optimization (SePO) is a token-level alignment method for LLMs that selects and optimizes only the most preference-relevant tokens in pairwise preference data, rather than uniformly training on entire responses. It was introduced to address a central inefficiency in post-training alignment: not all tokens in a generated response are equally important for modeling human preferences, yet prior token-level methods either optimize all tokens, which can be noisy and inefficient, or rely on complex and expensive key-token selection strategies. SePO proposes the first token selection method based on Direct Preference Optimization (DPO): a small oracle model is trained on response-level preference data to estimate a token-level reward function, this reward is used to score tokens across the target dataset, and only selected key tokens supervise the target policy through a reference model-free contrastive objective (Yang et al., 2024).

1. Problem setting and motivation

SePO is situated in the broader literature on preference optimization for LLMs, where DPO-style objectives replace explicit reward-model-plus-RL pipelines with direct training on preferred and rejected responses. The specific problem it targets is token heterogeneity: some tokens strongly determine whether a response is preferred, while many others are neutral, redundant, or noisy. In this view, sequence-level optimization diffuses credit assignment, and naïve token-level optimization still wastes updates by treating all positions equally (Yang et al., 2024).

The motivating premise is that preference information is concentrated in a minority of tokens. This premise also appears, in different forms, across later token-aware preference optimization work. SparsePO argues that human preference is often dependent on specific words or phrases and introduces sparse token masks to weight reward and KL contributions (Christopoulou et al., 2024). TIS-DPO proposes token-level importance sampling because standard DPO ignores importance differences between tokens (Liu et al., 2024). D²PO instead emphasizes positional asymmetry, arguing that earlier tokens contribute more and should receive larger weights via temporal decay (Shao et al., 20 Feb 2025). ConfPO identifies low-confidence tokens as preference-critical and optimizes only those tokens, without auxiliary models (Yoon et al., 10 Jun 2025). Within this landscape, SePO is distinctive in turning response-level pairwise data into token selection via an oracle reward estimator rather than via hand-crafted heuristics, auxiliary annotations, or purely positional rules (Yang et al., 2024).

A common misconception is that token-level preference optimization necessarily requires token-level supervision. SePO rejects that premise: it is explicitly designed to apply to existing alignment datasets with response-level annotations, and its token scores are inferred from an oracle/reference comparison rather than from new fine-grained labels (Yang et al., 2024).

2. Token-level reward decomposition from DPO

SePO formulates autoregressive generation as a Markov Decision Process. At step $t$, the state $s_t$ consists of the prompt together with all previously generated tokens up to position $t$, the action $a_t$ is the selected token, and the reward function $r(s_t,a_t)$ measures the desirability of the action at that state (Yang et al., 2024).

Its key theoretical claim is that DPO, although defined for response-level preferences, inherently learns a token-level reward function under an additive decomposition assumption. The response-level reward is assumed to decompose along the response trajectory as

$r(q, y) = \sum_{t=1}^T \hat{r}(s_t, a_t).$

Under this assumption, fitting DPO on pairwise preferred versus rejected responses yields a closed-form relationship between the optimal oracle policy $\pi^*$ and the reference policy $\pi_{\mathrm{ref}}$:

$$\hat{r}(s_t, a_t) \propto \log \frac{\pi^*(a_t \mid s_t)}{\pi_{\mathrm{ref}}(a_t \mid s_t)}.$$

This relation is the basis for SePO’s token scoring mechanism: token-level reward is approximated by log-probability differences between an oracle policy trained by DPO and a reference model (Yang et al., 2024).

This derivation places SePO in a line of work that reinterprets DPO at token resolution, but with a more explicitly selection-oriented use of the resulting token signal. TIS-DPO also introduces token-level weighting, but estimates weights through contrastive LLMs trained or prompted to separate winning and losing distributions (Liu et al., 2024). OTPO computes token weights from optimal transport over chosen and rejected hidden states to emphasize semantically meaningful token pairs (Li et al., 24 May 2025). AttentionPO, as an instantiation of TwDPO, uses attention from the LLM itself when judging preference pairs to derive content-aware token weights (Huang et al., 21 May 2026). SePO’s theoretical move is narrower and more specific: it treats the oracle/reference log-ratio as an estimated token reward and uses that signal for hard token selection rather than continuous reweighting (Yang et al., 2024).

3. Methodology: oracle estimation, token scoring, and selective training

SePO consists of three steps: oracle reward function estimation via token-level DPO, token-level scoring and selection, and policy training with a reference-free contrastive objective (Yang et al., 2024).

In the first step, a small oracle model is trained via DPO on a randomly chosen moderate-scale subset $\mathcal{S}$ of the target alignment data $s_t$0. The purpose of the oracle is not final deployment but approximation of the token-level reward function. The paper emphasizes that the oracle can be small-scale, for example a 1B-parameter model, making reward estimation cost-efficient (Yang et al., 2024).

In the second step, every token $s_t$1 in every response is scored by comparing the oracle and the reference model:

$s_t$2

For chosen responses, SePO selects the top $s_t$3 of tokens with the highest scores. For rejected responses, it selects the bottom $s_t$4 with the lowest scores. The selectors are represented by binary indicators:

$s_t$5

and

$s_t$6

The selected tokens are referred to as key tokens or preference tokens (Yang et al., 2024).

In the third step, the target policy model $s_t$7 is trained with a contrastive objective that uses only the selected tokens:

$s_t$8

where

$s_t$9

and

$t$0

The paper characterizes this objective as contrastive, reference-free, length-normalized, and efficient. “Reference-free” is a precise claim about the training loss: the objective itself does not require constraining the target policy toward a reference in KL or logit space, even though token scoring still depends on oracle/reference comparisons during data preparation (Yang et al., 2024).

Component	Formulation	Function
Reward estimation	$t$1	DPO-derived token reward
Token score	$t$2	Oracle/reference token scoring
Selection rule	Top $t$3 in chosen, bottom $t$4 in rejected	Key-token identification
Training loss	$t$5 with $t$6	Reference-free contrastive optimization

This selective pipeline differs from weighting-based methods. SparsePO learns continuous sparse masks for reward and KL terms on the fly (Christopoulou et al., 2024); TI-DPO uses gradient-based token-importance weights together with a triplet loss (Yang et al., 26 May 2025); Selective-DPO ranks tokens through log-probability differences between the current policy and a reference model, then optimizes only the top $t$7 across the pair (Dong, 10 Jul 2025). SePO’s design is two-stage and explicitly oracle-driven: estimate token reward first, then freeze that signal into dataset-wide token selection (Yang et al., 2024).

4. Empirical findings

SePO was evaluated on Arena-Hard, AlpacaEval 2.0, and MT-Bench. The reported central finding is that SePO significantly outperforms competitive baseline methods while optimizing only approximately $t$8 of tokens on the target dataset (Yang et al., 2024).

The paper reports that SePO consistently outperforms baselines such as DPO, SimPO, RRHF, and IPO on several benchmarks, while restricting optimization to a minority of tokens. On AlpacaEval 2.0, SePO achieves higher length-controlled win rates and win rates than all baselines across multiple base models. The empirical interpretation given in the paper is that most of the reward signal is concentrated in a minority of tokens, and that optimizing all tokens yields minimal further improvement once the informative subset has been captured (Yang et al., 2024).

The choice of $t$9 is a central hyperparameter. Performance reportedly improves rapidly as $a_t$0 increases up to roughly $a_t$1–$a_t$2, and the paper states that $a_t$3 is often optimal. This suggests that the key-token hypothesis is not merely computationally convenient but structurally informative: a relatively small subset of tokens carries much of the alignment-relevant signal (Yang et al., 2024).

SePO also reports weak-to-strong generalization. Small oracle models are said to supervise much larger policy models effectively through token selection, including cases in which the oracle is up to $a_t$4 smaller than the target. The paper further states that SePO can select key tokens from out-of-distribution data to enhance strong policy models and alleviate the over-optimization problem (Yang et al., 2024).

These findings align with later analyses of selective or token-aware preference learning in adjacent settings. ConfPO reports that training on only low-confidence tokens can improve human-preference alignment while mitigating overoptimization (Yoon et al., 10 Jun 2025). In mathematical reasoning, Future Policy Aware preference learning argues that overlap between preferred and dispreferred trajectories causes over-penalization of shared useful tokens, which can lead to performance collapse; its mitigation is proactive regularization rather than token selection, but the diagnosis is congruent with SePO’s claim that uniform optimization is noisy and destructive in part because it updates many non-critical shared tokens (Oh et al., 24 Sep 2025).

5. Relations to adjacent research directions

SePO belongs to a rapidly growing family of token-aware alignment methods, but its mechanism and assumptions differ materially from neighboring approaches.

One neighboring line focuses on token weighting rather than selection. TIS-DPO assigns token importance weights estimated from contrastive LLMs (Liu et al., 2024). OTPO derives context-aware token weights using optimal transport over hidden-state distances between chosen and rejected responses (Li et al., 24 May 2025). TI-DPO uses gradient attribution to measure token importance and combines weighted DPO with a triplet loss (Yang et al., 26 May 2025). AttentionPO extracts token weights from the LLM’s own attention when it acts as a pairwise judge, then inserts those weights into TwDPO (Huang et al., 21 May 2026). These methods preserve all tokens in the loss but modulate their contribution.

A second line focuses on sparsity or selectivity. SparsePO learns sparse token masks over reward and KL terms and can derive masks either from the reference model or on the fly (Christopoulou et al., 2024). Selective-DPO uses token-level log-probability differences between the current policy and a reference model to select the top $a_t$5 high-impact tokens, with reported sensitivity to reference model quality and an empirical optimum near $a_t$6 (Dong, 10 Jul 2025). ConfPO uses the policy model’s own confidence thresholded by the mean token probability within the sequence, selecting low-confidence tokens with zero additional computational overhead (Yoon et al., 10 Jun 2025). Compared with these methods, SePO is distinctive in using DPO to train a separate oracle model whose token-level log-ratio against a reference model defines the selection criterion (Yang et al., 2024).

A third line applies token-level preference ideas to specialized domains. RISE constructs hard preference pairs by injecting subtle token-level errors into reasoning steps and combines those with full-solution pairs for subtle error-aware DPO training, yielding improvements on GSM8K and MATH with only 4.5K training samples (Xu et al., 2024). Token Preference Optimization for vision-language hallucination mitigation introduces self-calibrated visual-anchored token rewards based on comparing logits under original and corrupted images (Gu et al., 2024). TAB-PO targets token-critical structured prediction by up