Adaptive Token Dictionary Overview

• Adaptive token dictionary is a mechanism that assigns token-specific representations and weights, replacing uniform token treatment with dynamic, input-dependent processing.
• It encompasses both explicit dictionaries (learnable token banks) and effective dictionaries (adaptive token importance maps), enhancing tasks like image restoration and semantic communication.
• These methods improve model performance by allocating computational resources dynamically, promoting diversity in decoding, and enabling adaptive vocabulary expansion.

Adaptive token dictionary denotes a class of mechanisms that replace uniform token treatment with token-dependent representations, weights, subsets, or routing rules. In its narrowest and explicit sense, especially in image restoration and super-resolution, it is a learnable dictionary of auxiliary tokens, $\mathbf{D} \in \mathbb{R}^{M \times d}$, that stores priors, is refined by the input, and mediates cross-attention and content-based grouping (Zhang et al., 2024, Zhang et al., 3 Mar 2026). A broader interpretation is also supported by recent work in neural machine translation, open-ended decoding, long-video multimodal inference, tokenizer adaptation, and semantic communication, where the “dictionary” is the effective token set or token-to-weight map used at training or inference time rather than only a bank of learned code vectors (Xu et al., 2021, Zhu et al., 2024, Qi et al., 30 Mar 2026, Kadam et al., 3 Jul 2026).

1. Conceptual scope

The literature supports two recurrent meanings. One is an explicit dictionary, where a model maintains a parameterized set of token-like atoms and uses attention or refinement to align them with the input. The other is an effective dictionary, where adaptivity is realized by deciding which tokens matter, which tokens survive, how strongly each token contributes to loss, or how a new token should be represented inside an existing vocabulary. These works suggest that “adaptive token dictionary” is best understood as a systems-level abstraction over token importance, token availability, and token semantics rather than as a single architectural primitive (Zhang et al., 2024, Zhang et al., 3 Mar 2026, Gu et al., 2020, Xu et al., 2021, Zhu et al., 2024, Qi et al., 30 Mar 2026, Sharthak et al., 14 May 2025, Liyanage et al., 29 Jan 2026, Pethkar et al., 5 May 2026).

Form	Representative mechanism	Exemplars
Explicit token bank	Learnable $\mathbf{D}$, cross-attention, adaptive refinement	ATD-SR, ATD
Token importance map	Per-token weights, candidate sets, token-wise temperatures	NMT adaptive training, adaptive decoding, AdaKD
Budgeted or expanded token set	Token pruning, token budgeting, vocabulary replacement, new-token embedding tuning	AdaptToken, AdaTok, AdaptBPE, TokenAdapt, Markov-matrix expansion

A common misconception is that adaptive token dictionaries always refer to a learned codebook analogous to VQ. The image restoration papers do use that exact formulation, but several other papers instantiate the same organizing idea through token-level weighting, token selection, or vocabulary transplantation rather than through an explicit bank of dictionary atoms. Another misconception is that adaptivity is necessarily dynamic at inference time. Some methods are fully dynamic, but others are static after corpus statistics are computed, as in frequency-based NMT weighting or post-training BPE adaptation.

2. Formal mechanisms

A unifying formal pattern is a mapping from a token or token occurrence to a control signal. In translation, the basic adaptive form is a weighted token loss,

$$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$

with $w_j$ determined either by target-side frequency or by bilingual mutual information (Gu et al., 2020, Xu et al., 2021). In the bilingual case,

$w_j = S \cdot \text{BMI}(x, y_j) + B,$

optionally with hard thresholding for low-BMI tokens, so the effective dictionary is no longer a type-level table $y \mapsto w_y$ but an occurrence-level map $(x,y_j)\mapsto w_j$ (Xu et al., 2021).

In explicit-dictionary models for image restoration, the primitive object is

$\mathbf{D} \in \mathbb{R}^{M \times d},$

and token dictionary cross-attention uses cosine similarity between image-token queries and dictionary keys: $$\mathbf{A} = \operatorname{SoftMax}\big(\operatorname{Sim}_{\cos}(\mathbf{Q}_X,\mathbf{K}_D)/\tau\big), \qquad \operatorname{TDCA} = \mathbf{A}\mathbf{V}_D.$$ This formulation makes the dictionary simultaneously a prior bank and a clustering device, because $\mathbf{A}$ can be reused to assign tokens to content categories (Zhang et al., 2024, Zhang et al., 3 Mar 2026).

In adaptive decoding, the effective token dictionary is the candidate set $\mathbf{D}$0 selected at each step by an entropy-derived confidence increment. Tokens are added in descending probability order until the marginal gain

$\mathbf{D}$1

falls below a threshold $\mathbf{D}$2, after which sampling is restricted to the selected set (Zhu et al., 2024). In token-adaptive distillation, AdaKD defines a unified difficulty score $\mathbf{D}$3, uses the top $\mathbf{D}$4 hardest tokens for the loss, and assigns each selected token a token-wise temperature

$\mathbf{D}$5

with low temperature for hard tokens and high temperature for easy ones (Xie et al., 13 Oct 2025).

A further generalization appears in long-video MLLMs, where the dictionary is a query-conditioned subset of visual tokens drawn from multiple frame groups. AdaptToken computes group certainty $\mathbf{D}$6 from response entropy and allocates a global token budget through

$\mathbf{D}$7

Here the adaptive dictionary is the union of top-ranked tokens from all groups after certainty-based budgeting and redundancy removal (Qi et al., 30 Mar 2026).

3. Sequence modeling, translation, and decoding

The sequence-modeling line begins with token-level adaptive training for NMT under target-side frequency imbalance. On the NIST ZH–EN training set, the top 10% of tokens have average frequency about $\mathbf{D}$8, whereas the bottom 30% have average frequency about $\mathbf{D}$9; in references, $$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$0 of tokens come from the top 10% of the vocabulary and $$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$1 from the bottom 30%, while a vanilla Transformer generates $$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$2 from the top 10% and only $$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$3 from the bottom 30% (Gu et al., 2020). The proposed exponential and Chi-Square-like weighting functions enforce $$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$4, avoid downweighting frequent tokens, and improve BLEU on ZH–EN, EN–RO, and EN–DE, with the largest gains on sentences rich in low-frequency tokens; they also improve lexical diversity (Gu et al., 2020).

The bilingual extension replaces monolingual frequency with bilingual mutual information. BMI is computed from sentence-level co-occurrence counts,

$$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$5

and then converted to a token weight. On WMT14 En–De and WMT19 Zh–En, BMI reaches $$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$6 and $$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$7 BLEU, compared with $$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$8 and $$\mathcal{L}_{\text{adapt}} = - \sum_j w_j \log p_\theta(y_j \mid y_{<j}, x),$$9 for the Transformer baseline, and it improves lexical diversity more than frequency-only adaptive objectives (Xu et al., 2021). A notable interpretive point is that BMI assigns larger weights to high-BMI tokens, which the paper treats as strongly source-licensed tokens whose errors should be penalized more heavily, while weakly associated tokens are downweighted or ignored. This departs from the simpler intuition that adaptivity should always concentrate on the hardest tokens.

At inference time, adaptive decoding makes the dictionary itself step-dependent. The next-token candidate set is built by confidence increments rather than fixed $w_j$0 or fixed cumulative probability mass, and on GPT2-XL over WritingPrompts the average candidate-set size varies from about $w_j$1 at $w_j$2 to about $w_j$3 at $w_j$4 (Zhu et al., 2024). The same paper reports higher MAUVE and diversity than top-$w_j$5, top-$w_j$6, and typical decoding, with especially strong gains on Llama2-7B-chat (Zhu et al., 2024).

A plausible continuous analogue is adaptive latent Chain-of-Thought pretraining. There the model generates a variable-length latent trajectory before emitting each visible token, and token-wise halting decides how many latent steps each token receives. The paper reports lower perplexity and better downstream accuracy than recurrent baselines, with fewer training FLOPs, and shows that longer latent trajectories are allocated to more difficult tokens (Zeng et al., 9 Feb 2026). This suggests that adaptive token dictionaries need not be discrete lookup tables; they can also be latent token-specific bases whose effective support varies with token difficulty.

4. Vision, video, and multimodal perception

The most literal use of the term appears in super-resolution and image restoration Transformers. ATD-SR introduces a learnable dictionary $w_j$7, token dictionary cross-attention, adaptive dictionary refinement, and adaptive category-based self-attention, so that the dictionary both injects global priors and partitions image tokens into categories for non-local attention (Zhang et al., 2024). The later ATD formulation retains the dictionary-centered design, adds a reparameterized scaling factor for TDCA, integrates category information into a category-aware FFN, and extends the framework to ATD-U for denoising and JPEG compression artifact removal (Zhang et al., 3 Mar 2026). In both formulations, the dictionary is neither a fixed memory nor a pure clustering device; it is a trainable prior bank that becomes input-specific through refinement.

Adaptive tokenization for images makes the dictionary budget itself. AdaTok keeps a shared discrete codebook of size $w_j$8 but learns an ordered 1D token stream of length $w_j$9, where prefixes remain decodable across budgets via Prioritized Representation Learning and Multi-Head LoRA decoder heads. A separate policy selects a budget $w_j = S \cdot \text{BMI}(x, y_j) + B,$0 per image. On ImageNet-1K, AdaTok-Full reaches rFID $w_j = S \cdot \text{BMI}(x, y_j) + B,$1 at 256 tokens, while AdaTok-Adaptive reaches rFID $w_j = S \cdot \text{BMI}(x, y_j) + B,$2 using only about $w_j = S \cdot \text{BMI}(x, y_j) + B,$3 tokens on average; in autoregressive image generation, the shorter adaptive representation yields about $w_j = S \cdot \text{BMI}(x, y_j) + B,$4 throughput over a fixed 256-token decode (Lu et al., 5 Jun 2026). Here the adaptive token dictionary is the prefix $w_j = S \cdot \text{BMI}(x, y_j) + B,$5 selected by the policy, not the static codebook alone.

For video, AdapTok learns adaptive and temporally causal tokenization in a 1D latent space. Block-wise tail dropping forces early latent slots in each block to encode more globally useful information, a block-causal scorer predicts reconstruction quality as a function of retained tokens, and an integer linear program allocates token counts per block under a global budget (Li et al., 22 May 2025). On UCF-101 reconstruction, AdapTok reports rFVD $w_j = S \cdot \text{BMI}(x, y_j) + B,$6 at 2048 tokens, about $w_j = S \cdot \text{BMI}(x, y_j) + B,$7 at 1024 tokens, and $w_j = S \cdot \text{BMI}(x, y_j) + B,$8 at 512 tokens, outperforming several fixed-budget baselines at comparable or larger token counts (Li et al., 22 May 2025). This is an explicit example of a fixed codebook with adaptive slot usage.

Long-video MLLM inference uses a related but inference-only formulation. AdaptToken partitions the video into groups, scores tokens within each group by cross-modal attention, estimates group relevance from answer entropy, allocates a global token budget across groups, and optionally stops early once several groups exhibit sufficiently low entropy (Qi et al., 30 Mar 2026). Across VideoMME, LongVideoBench, LVBench, and MLVU, it reports average gains of about $w_j = S \cdot \text{BMI}(x, y_j) + B,$9 on Qwen2.5-VL 7B, continues to benefit up to 10K frames, and AdaptToken-Lite reduces inference time by about half with comparable performance (Qi et al., 30 Mar 2026). In this setting the dictionary is the final query-conditioned subset of visual tokens that the LLM actually sees.

5. Communication, scheduling, and edge systems

In semantic communication, the adaptive token dictionary is shaped jointly by semantic importance, interference, and channel state. ATS-ToDMA first selects semantic tokens with importance score $y \mapsto w_y$0, then schedules them into token-domain slots with a transformer-based scheduler, and finally assigns power through a semantic-aware SSINR criterion: $y \mapsto w_y$1 The framework derives analytical bounds on interference and feasible occupancy and reports, against Greedy ATS, semantic throughput $y \mapsto w_y$2, semantic accuracy $y \mapsto w_y$3, aggregate interference $y \mapsto w_y$4, average SSINR $y \mapsto w_y$5, and average transmit power $y \mapsto w_y$6 (Kadam et al., 3 Jul 2026). The dictionary here is “effective” rather than explicit: it is the set of selected, scheduled, and powered semantic tokens that remain active over the air.

A related edge-inference formulation uses semantic token selection inside a transformer-powered DJSCC pipeline. A budget token interpolates between learned low-budget and high-budget embeddings, gate and threshold networks produce masks, and a Lyapunov-based resource-allocation rule selects both token fraction $y \mapsto w_y$7 and compression ratio $y \mapsto w_y$8 under dynamic channel conditions (Devoto et al., 23 May 2025). This combines dictionary-size control with adaptive per-token dimensionality.

At the serving-systems level, OTAS treats token configuration itself as the deployment-time control variable. A unified transformer can either add prompt tokens when $y \mapsto w_y$9 or merge tokens when $(x,y_j)\mapsto w_j$0, and the serving system assigns a batch-level $(x,y_j)\mapsto w_j$1 by combining application-aware selective batching with an online optimization over profiled utility, latency, and memory (Chen et al., 2024). On the reported workloads, OTAS improves system utility by at least $(x,y_j)\mapsto w_j$2, and substantially more on some traces (Chen et al., 2024). This use is distinct from model-internal token adaptation: the dictionary is an operational token configuration selected to satisfy service-level objectives.

6. Tokenizer adaptation, vocabulary expansion, and long-term memory

A further branch concerns adaptation of the token inventory itself. TokenAdapt transplants a new tokenizer into a pretrained LLM by copying shared embeddings and synthesizing embeddings for new tokens through a hybrid of a local estimate from old-token decomposition and a global estimate from semantically similar old tokens: $(x,y_j)\mapsto w_j$3 In zero-shot perplexity-ratio evaluations, the hybrid initialization consistently outperforms ReTok and TransTokenizer baselines, often by about a $(x,y_j)\mapsto w_j$4 margin in aggregate scores, while Supertokens provide additional compression gains through multi-word vocabulary items (Sharthak et al., 14 May 2025). In this setting, the adaptive token dictionary is literal: the vocabulary is redesigned, and the model is adapted to it without full retraining.

AdaptBPE performs a lighter-weight post-training adaptation. It fixes a merge budget $(x,y_j)\mapsto w_j$5, identifies low-utility active merges and high-frequency beneficial merges in an adaptation corpus, and replaces the former with the latter while preserving compatibility with model weights (Liyanage et al., 29 Jan 2026). The result is a domain- or language-specialized tokenizer with the same vocabulary size but different effective inventory. The paper reports better compression utility than several fixed baselines and, on corpora such as PubMed and EMEA, sometimes lower perplexity than the full original vocabulary at the same time that it reduces the active vocabulary (Liyanage et al., 29 Jan 2026).

The Markov-matrix formulation pushes the idea to continual knowledge expansion. There, model memory is represented by a Markov transition matrix over the original vocabulary $(x,y_j)\mapsto w_j$6, and a new token $(x,y_j)\mapsto w_j$7 is introduced by a token-to-dictionary mapping

$(x,y_j)\mapsto w_j$8

so the new token embedding lies in the span of the old embedding dictionary (Pethkar et al., 5 May 2026). Theoretical results show that the required number of samples per new token scales linearly with the number of existing tokens it is mapped to, and the practical embedding-tuning algorithm updates only embeddings of new tokens while leaving the original transition structure unchanged, yielding zero forgetting for the old vocabulary (Pethkar et al., 5 May 2026). This is perhaps the clearest formalization of an adaptive token dictionary as a device for knowledge expansion rather than token compression or loss reweighting.

Across these lines of work, several recurring themes are clear. First, adaptivity may be static after corpus statistics are computed, dynamic per sample or per step, or policy-driven under external constraints. Second, the dictionary may be an explicit bank of atoms, a candidate set, a weighted loss table, a token budget, or a mutable vocabulary. Third, the principal unresolved issues recur across domains: heuristic thresholds, dependence on corpus or attention statistics, sensitivity to domain shift or sparsity, and the tension between richer adaptivity and stable optimization. What remains constant is the central idea that token processing need not be uniform: a model can learn or infer which tokens should exist, which tokens should dominate gradients, which tokens deserve computational budget, and which latent or semantic atoms should represent them.