Direct Token Optimization (DTO)

• Direct Token Optimization is a family of methods that applies token-level credit assignment to refine preference objectives in autoregressive systems.
• DTO approaches reassign token-specific rewards and weights to address issues in alignment, unlearning, and efficiency, often achieving significant improvements over sequence-level methods.
• The research encompasses diverse strategies, including token reward reformulation, selective token optimization, and importance-sampled weighting, each enhancing model performance in targeted applications.

Direct Token Optimization (DTO) denotes a family of methods that move optimization from undifferentiated sequence-level treatment toward token-level credit assignment. In the narrow sense, the term appears explicitly as a self-contained unlearning method that separates target tokens from non-target tokens and optimizes them with different objectives (Lee et al., 30 Sep 2025). In a broader and now common sense, closely related work uses other names—such as Token-level Direct Preference Optimization, Token Preference Optimization, token-level importance sampling DPO, reward-guided DPO, and token-weighted DPO—while sharing the same premise: autoregressive models generate and fail one token at a time, so sequence-level preference learning is often too coarse (Zeng et al., 2024, Gu et al., 2024, Liu et al., 2024, Zhu et al., 17 Jun 2025, Huang et al., 21 May 2026).

1. Terminological scope and taxonomic boundaries

DTO is not a single uniformly named formalism. Several papers are explicit that they are DTO-like or DTO-adjacent rather than canonical DTO. TPO is described as a DPO-derived, policy-gradient-free preference-learning method that injects token-level weighting into the DPO objective, not as a pure direct token optimization method (Gu et al., 2024). DDRO is presented as a document-level, DPO-style analogue for generative retrieval rather than DTO in the narrow sense (Mekonnen et al., 7 Apr 2025). DMTD is framed as a direct token-generation shortcut that optimizes the compute pathway per generated token rather than a new token-level objective (Luo et al., 13 Oct 2025). FlexDiT is likewise characterized as dynamic token density control rather than direct token-importance optimization (Chang et al., 2024).

This terminological spread matters because the literature uses “token optimization” for at least three related but distinct agendas. One agenda concerns alignment and preference optimization, where token-level rewards, weights, or masks reshape DPO- or PPO-style learning. A second concerns unlearning, where token-level objectives are used to erase forget-set knowledge while preserving utility. A third concerns inference and systems efficiency, where the optimized object is the token generation path, token density, or token budget rather than a preference loss. The common denominator is token-granular control; the optimized target differs by subfield.

2. From sequence-level preference margins to token-level control

Standard DPO operates on whole responses. In its usual form, the preference logit compares sequence-level policy/reference ratios for a chosen response $y_w$ and a rejected response $y_l$: $$\mathcal{L}_{\mathrm{DPO}}(\pi_\theta;\pi_{\mathrm{ref}}) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal D} \left[ \log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)} \right) \right].$$ Although $\log \pi_\theta(y|x)$ factorizes autoregressively as a sum of token log-probabilities, vanilla DPO still assigns all tokens equal status inside the sequence-level margin (Zeng et al., 2024).

The central DTO move is to reinterpret preference learning through token-level quantities. TDPO models generation as a token-level MDP and rewrites Bradley–Terry preference probabilities in terms of cumulative token-level advantages, while adding a sequential forward-KL correction $$D_{\mathrm{SeqKL}}(x,y;\pi_{\mathrm{ref}}\|\pi_\theta)$$ to control divergence prefix by prefix (Zeng et al., 2024). RTO shows that, under deterministic autoregressive transitions, DPO’s implicit reward admits the token-wise characterization

$$r^*(s_h,a_h)=\beta\log\frac{\pi_\beta^*(a_h|s_h)}{\pi_{\mathrm{ref}}(a_h|s_h)},$$

and then uses this dense token signal inside PPO (Zhong et al., 2024). TGDPO proceeds from the PPO side, decomposes sequence-level PPO into token-level PPO problems, derives a token-level optimal policy and induced token reward, and then rebuilds a DPO-style loss from token-level weighted log-probability ratios (Zhu et al., 17 Jun 2025).

What changes across these formulations is not merely notation. The optimization target shifts from “raise the probability of the preferred sequence” to “allocate preference pressure across token positions according to token-level reward, importance, or structure.” This suggests a more faithful match to autoregressive generation, where errors, hallucinations, unsafe continuations, retrieval mistakes, or memorized facts are often localized to a small subset of tokens.

3. Main methodological families

The current literature organizes into a small number of recurring designs.

Family	Core mechanism	Representative papers
Token-level reformulation	Token advantages, token rewards, or token-level PPO-to-DPO derivation	TDPO (Zeng et al., 2024), RTO (Zhong et al., 2024), TGDPO (Zhu et al., 17 Jun 2025)
Token-weighted DPO	Weighted token log-ratio sums inside a pairwise logistic loss	TPO (Gu et al., 2024), OTPO (Li et al., 24 May 2025), TwDPO/AttentionPO (Huang et al., 21 May 2026), TAB-PO (Fodeh et al., 3 Feb 2026), TI-DPO (Yang et al., 26 May 2025)
Importance-sampled weighting	Token weights estimated from contrastive models	TIS-DPO (Liu et al., 2024)
Selective token optimization	Offline token scoring, then optimize only selected positions	SePO (Yang et al., 2024)
Tokenized sequence objects	Sequence probabilities over tokenized identifiers aligned to downstream ranking	DDRO (Mekonnen et al., 7 Apr 2025)

The first family changes the formal unit of optimization. TDPO rewrites Bradley–Terry in terms of cumulative token advantages and introduces sequential forward KL to improve the alignment/diversity tradeoff; RTO uses DPO-induced token rewards to densify PPO; TGDPO assigns different deviation strengths from the reference policy to different tokens through reward-guided weighting (Zeng et al., 2024, Zhong et al., 2024, Zhu et al., 17 Jun 2025).

The second family leaves the outer pairwise logistic shell intact but changes the inner decomposition. TPO computes visual-anchored token rewards from raw-versus-corrupted image logits; OTPO derives token weights from an unbalanced optimal transport plan over chosen and rejected token representations; AttentionPO extracts token weights from the LLM’s own self-judging attention; TAB-PO assigns schema-aware weights to Code, Sub-code, and Span tokens and adds a token-level barrier; TI-DPO uses gradient-based token importance plus a triplet loss (Gu et al., 2024, Li et al., 24 May 2025, Huang et al., 21 May 2026, Fodeh et al., 3 Feb 2026, Yang et al., 26 May 2025).

The third family emphasizes token selection rather than dense weighting. SePO first trains a small oracle model with DPO, estimates token-level reward proxies by oracle/reference log-ratios, ranks tokens within each response, and then trains the target model only on selected positions through a reference-free contrastive loss (Yang et al., 2024). TIS-DPO occupies an intermediate position: it assumes the ideal DPO dataset would have equal expected rewards for each token in winning and losing responses, estimates token weights from contrastive LLMs, and uses importance sampling to build a token-weighted DPO objective (Liu et al., 2024).

4. Direct Token Optimization as a self-contained unlearning method

The explicit formulation titled “Direct Token Optimization” treats unlearning as a token-level optimization problem over forget-set sequences (Lee et al., 30 Sep 2025). For each forget-set sequence $s^i$, the method splits the sequence at a pivot $q_i$ into prefix and suffix, perturbs each prefix token in turn, and computes a delta-score

$$\Delta_r^i= \sum_{t=q_i+1}^{T_i}\log p_\theta(x_t^i|x_{<t}^i) - \sum_{t=q_i+1}^{T_i}\log p_\theta(x_t^i|\tilde{x}_{<t}^i).$$

Tokens with the largest $\Delta_r^i$ values are treated as target tokens, because their perturbation most strongly changes suffix likelihood; the remaining tokens are non-target tokens.

The optimization then bifurcates. Target tokens are updated by gradient ascent on their log-likelihoods, weakening the model’s tendency to reproduce forget-set knowledge. Non-target tokens are regularized by KL divergence to the original model, preserving local fluency and utility. The method requires only the original model and the forget set: no retain dataset, no auxiliary models, no external APIs, and no human token labels (Lee et al., 30 Sep 2025).

This formulation makes the narrowest sense of DTO concrete. The paper reports up to $y_l$0 improvement in forget quality over recent baselines while maintaining a comparable level of model utility, and it also documents the main internal trade-off: DTO without the non-target KL term often forgets more strongly, whereas DTO with KL preserves utility better and can improve forget quality in harder settings by avoiding over-unlearning (Lee et al., 30 Sep 2025).

5. Applications and empirical record

In large vision-LLMs, token-level direct optimization has been used for hallucination mitigation. TPO identifies “visual-anchored tokens” by comparing raw token logits under the original image and a corrupted image embedding, then multiplies each token’s contribution inside a DPO-style objective by a self-calibrated reward. On LLaVA-1.5-7B, it improves AMBER from 74.3 to 85.0 F1, MMHal score from 2.01 to 2.47, HallusionBench aAcc from 47.21 to 50.22, and reduces MMHal hallucination rate from 61.46 to 51.04; relative to DPO, it improves AMBER from 82.1 to 85.0 and MMHal hallucination from 58.33 to 51.04 (Gu et al., 2024).

In generative retrieval, DDRO uses token probabilities of document identifiers as the mechanism for direct ranking optimization. Retrieval scores are sequence probabilities over docid tokens, but optimization is performed by pairwise relevance preferences over complete docid sequences. On benchmark datasets, DDRO achieves a 7.4% improvement in MRR@10 for MS MARCO and a 19.9% improvement for Natural Questions; for example, on MS MARCO DDRO (TU) reaches 50.07 MRR@10 versus 46.62 for GenRRL (Sum), and on NQ DDRO (PQ) reaches 55.51 versus 46.31 (Mekonnen et al., 7 Apr 2025).

In general alignment, multiple token-level methods report gains over sequence-level baselines. RTO outperforms PPO by 7.5 points on the AlpacaEval 2 benchmark and by 4.1 points on Arena-Hard (Zhong et al., 2024). SePO reports that optimizing only 30% key tokens can outperform response-level baselines and support weak-to-strong transfer, including a Pythia-410M oracle supervising Pythia-SFT-6.9B (Yang et al., 2024). OTPO improves Llama-3-8B + UltraFeedback from 48.14 to 53.37 length-controlled win rate on AlpacaEval2 (Li et al., 24 May 2025). TGDPO reports gains of up to 7.5 points on MT-Bench, 6.2 on AlpacaEval 2, and 4.3 on Arena-Hard (Zhu et al., 17 Jun 2025). AttentionPO raises LLaMA-3-8B-Instruct to 58.29 AlpacaEval win rate and 7.19 MT-Bench while remaining content-aware and adding only two extra forward passes per example (Huang et al., 21 May 2026). TIS-DPO, using token-level importance sampling with estimated weights, significantly outperforms baseline methods on harmlessness, helpfulness, and summarization tasks (Liu et al., 2024).

In token-critical structured generation, TAB-PO shows that token weighting can be the dominant added component. On medical communication annotation, the method reports about ~4% relative improvement in micro-F1 over SFT, with the strongest gains on Sub-code prediction and a mean F1 of 82.65 versus 78.84 for SFT (Fodeh et al., 3 Feb 2026). TI-DPO extends the same general pattern to instruction following and reasoning-oriented evaluation, reaching an average of 62.3 across MMLU, GSM8K, GPQA, HumanEval, TruthfulQA, and IFEval versus 57.7 for DPO in the reported table (Yang et al., 26 May 2025).

6. Limits, misconceptions, and broader uses of token optimization

A persistent misconception is that DTO must mean explicit token-level human labels or a purely token-only loss. Much of the literature does not do that. TPO, OTPO, TIS-DPO, TwDPO, TI-DPO, and DDRO all retain a sequence-level pairwise logistic or ranking shell while modifying how token-level contributions are aggregated inside it (Gu et al., 2024, Li et al., 24 May 2025, Liu et al., 2024, Huang et al., 21 May 2026, Yang et al., 26 May 2025, Mekonnen et al., 7 Apr 2025). What changes is usually the credit assignment mechanism, not the existence of sequence-level chosen/rejected supervision.

The main limitations recur across papers. Token rewards or weights are often estimated rather than observed; therefore their quality depends on contrastive models, oracle models, attention patterns, corrupted-image sensitivity, or schema-aware typing. Many methods add extra computation: TPO requires raw and corrupted-image passes, OTPO adds $y_l$1 optimal transport, AttentionPO requires two extra judge forward passes, and SePO adds oracle training plus full-dataset token scoring (Gu et al., 2024, Li et al., 24 May 2025, Huang et al., 21 May 2026, Yang et al., 2024). Several methods are also strongly domain-shaped: TAB-PO assumes canonical JSON fields such as Code, Sub-code, and Span; DTO unlearning is sensitive to the target-token ratio $y_l$2 and suffix ratio; TGDPO depends on the quality of token-level reward guidance and the positivity of its shaping functions (Fodeh et al., 3 Feb 2026, Lee et al., 30 Sep 2025, Zhu et al., 17 Jun 2025).

The term also extends beyond alignment. DMTD optimizes the layerwise computation path per token through cyclic reuse of late layers, achieving up to a $y_l$3 speedup with minor performance loss on a fine-tuned Qwen3-4B model, but it is an inference shortcut rather than a token-level preference objective (Luo et al., 13 Oct 2025). FlexDiT uses dynamic token density control across layer depth and diffusion timesteps, combining sparse and dense token processing to reduce FLOPs and improve throughput, but it is explicitly framed as structured token density scheduling rather than token-importance optimization (Chang et al., 2024). In Oracle-to-PostgreSQL migration, “token optimization” is cast as a constrained transformation problem over input and output token budgets, with adaptive routing giving the best practical trade-off and aggressive schema distillation showing that token efficiency alone can be misleading (Grynets et al., 27 May 2026).

Taken together, the literature suggests that DTO is best understood not as a single algorithm but as a research program: replacing uniform sequence-level treatment with token-level structure, whether by token rewards, token weights, token masks, or token-specific compute allocation. The strongest common result is that sequence-level direct preference learning becomes substantially more effective when the optimization mass is concentrated on the tokens that actually carry the relevant signal.