--- title: Direct Token Optimization (DTO) url: https://www.emergentmind.com/topics/direct-token-optimization-dto type: topic --- # Direct Token Optimization (DTO) Direct Token Optimization (DTO) denotes a family of methods that move optimization from undifferentiated sequence-level treatment toward token-level credit assignment. In the narrow sense, the term appears explicitly as a self-contained unlearning method that separates target tokens from non-target tokens and optimizes them with different objectives [2510.00125]. In a broader and now common sense, closely related work uses other names—such as Token-level Direct Preference Optimization, Token Preference Optimization, token-level importance sampling DPO, reward-guided DPO, and token-weighted DPO—while sharing the same premise: autoregressive models generate and fail one token at a time, so sequence-level preference learning is often too coarse [2404.11999] [2412.14487] [2410.04350] [2506.14574] [2605.21883]. ## 1. Terminological scope and taxonomic boundaries DTO is not a single uniformly named formalism. Several papers are explicit that they are **DTO-like** or **DTO-adjacent** rather than canonical DTO. TPO is described as a DPO-derived, policy-gradient-free preference-learning method that injects token-level weighting into the DPO objective, not as a pure direct token optimization method [2412.14487]. DDRO is presented as a document-level, DPO-style analogue for generative retrieval rather than DTO in the narrow sense [2504.05181]. DMTD is framed as a direct token-generation shortcut that optimizes the compute pathway per generated token rather than a new token-level objective [2510.11958]. FlexDiT is likewise characterized as dynamic token density control rather than direct token-importance optimization [2412.06028]. This terminological spread matters because the literature uses “token optimization” for at least three related but distinct agendas. One agenda concerns **alignment and preference optimization**, where token-level rewards, weights, or masks reshape DPO- or PPO-style learning. A second concerns **unlearning**, where token-level objectives are used to erase forget-set knowledge while preserving utility. A third concerns **inference and systems efficiency**, where the optimized object is the token generation path, token density, or token budget rather than a preference loss. The common denominator is token-granular control; the optimized target differs by subfield. ## 2. From sequence-level preference margins to token-level control Standard DPO operates on whole responses. In its usual form, the preference logit compares sequence-level policy/reference ratios for a chosen response \(y_w\) and a rejected response \(y_l\): \[ \mathcal{L}_{\mathrm{DPO}}(\pi_\theta;\pi_{\mathrm{ref}}) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal D} \left[ \log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)} \right) \right]. \] Although \(\log \pi_\theta(y|x)\) factorizes autoregressively as a sum of token log-probabilities, vanilla DPO still assigns all tokens equal status inside the sequence-level margin [2404.11999]. The central DTO move is to reinterpret preference learning through token-level quantities. TDPO models generation as a token-level MDP and rewrites Bradley–Terry preference probabilities in terms of cumulative token-level advantages, while adding a sequential forward-KL correction \(D_{\mathrm{SeqKL}}(x,y;\pi_{\mathrm{ref}}\|\pi_\theta)\) to control divergence prefix by prefix [2404.11999]. RTO shows that, under deterministic autoregressive transitions, DPO’s implicit reward admits the token-wise characterization \[ r^*(s_h,a_h)=\beta\log\frac{\pi_\beta^*(a_h|s_h)}{\pi_{\mathrm{ref}}(a_h|s_h)}, \] and then uses this dense token signal inside PPO [2404.18922]. TGDPO proceeds from the PPO side, decomposes sequence-level PPO into token-level PPO problems, derives a token-level optimal policy and induced token reward, and then rebuilds a DPO-style loss from token-level weighted log-probability ratios [2506.14574]. What changes across these formulations is not merely notation. The optimization target shifts from “raise the probability of the preferred sequence” to “allocate preference pressure across token positions according to token-level reward, importance, or structure.” This suggests a more faithful match to autoregressive generation, where errors, hallucinations, unsafe continuations, retrieval mistakes, or memorized facts are often localized to a small subset of tokens. ## 3. Main methodological families The current literature organizes into a small number of recurring designs. | Family | Core mechanism | Representative papers | |---|---|---| | Token-level reformulation | Token advantages, token rewards, or token-level PPO-to-DPO derivation | TDPO [2404.11999], RTO [2404.18922], TGDPO [2506.14574] | | Token-weighted DPO | Weighted token log-ratio sums inside a pairwise logistic loss | TPO [2412.14487], OTPO [2505.18720], TwDPO/AttentionPO [2605.21883], TAB-PO [2603.00025], TI-DPO [2505.19653] | | Importance-sampled weighting | Token weights estimated from contrastive models | TIS-DPO [2410.04350] | | Selective token optimization | Offline token scoring, then optimize only selected positions | SePO [2408.13518] | | Tokenized sequence objects | Sequence probabilities over tokenized identifiers aligned to downstream ranking | DDRO [2504.05181] | The first family changes the formal unit of optimization. TDPO rewrites Bradley–Terry in terms of cumulative token advantages and introduces sequential forward KL to improve the alignment/diversity tradeoff; RTO uses DPO-induced token rewards to densify PPO; TGDPO assigns different deviation strengths from the reference policy to different tokens through reward-guided weighting [2404.11999] [2404.18922] [2506.14574]. The second family leaves the outer pairwise logistic shell intact but changes the inner decomposition. TPO computes visual-anchored token rewards from raw-versus-corrupted image logits; OTPO derives token weights from an unbalanced optimal transport plan over chosen and rejected token representations; AttentionPO extracts token weights from the LLM’s own self-judging attention; TAB-PO assigns schema-aware weights to Code, Sub-code, and Span tokens and adds a token-level barrier; TI-DPO uses gradient-based token importance plus a triplet loss [2412.14487] [2505.18720] [2605.21883] [2603.00025] [2505.19653]. The third family emphasizes token selection rather than dense weighting. SePO first trains a small oracle model with DPO, estimates token-level reward proxies by oracle/reference log-ratios, ranks tokens within each response, and then trains the target model only on selected positions through a reference-free contrastive loss [2408.13518]. TIS-DPO occupies an intermediate position: it assumes the ideal DPO dataset would have equal expected rewards for each token in winning and losing responses, estimates token weights from contrastive LLMs, and uses importance sampling to build a token-weighted DPO objective [2410.04350]. ## 4. Direct Token Optimization as a self-contained unlearning method The explicit formulation titled “Direct Token Optimization” treats unlearning as a token-level optimization problem over forget-set sequences [2510.00125]. For each forget-set sequence \(s^i\), the method splits the sequence at a pivot \(q_i\) into prefix and suffix, perturbs each prefix token in turn, and computes a delta-score \[ \Delta_r^i= \sum_{t=q_i+1}^{T_i}\log p_\theta(x_t^i|x_{