Model-Aware Tokenizer Transfer (MATT)

• The paper introduces MATT as a framework that leverages deep model signals—such as attention patterns and predictive likelihoods—to overcome static embedding limitations.
• MATT employs variants like Adaptive Tokenization, TokAlign, ALM, and AIM to efficiently adapt vocabularies while preserving both surface and internal semantics.
• Experimental results demonstrate that MATT recovers high accuracy with reduced runtime, making it effective for low-resource language adaptation and cross-model distillation.

Model-Aware Tokenizer Transfer (MATT) refers to a set of techniques for efficiently and robustly adapting pretrained LLMs to operate with new, possibly radically different, tokenizers. These methods incorporate deeper model-internal signals—such as conditional likelihood distributions and attention interactions—rather than treating embedding initialization as a purely lexical or static similarity problem. MATT has emerged as a unifying framework to resolve the critical bottleneck imposed by pre-defined tokenization schemes in LLMs, especially for multilingual adaptation and efficient cross-model knowledge transfer.

1. Motivation and Problem Statement

Pretrained LLMs rely on tokenizers—mappings from text to sequences of discrete symbols—to structure their input data. The choice of tokenizer is typically fixed at training and tightly coupled to the model weights. As a consequence, adaptation of pretrained models to new domains, languages, or downstream tools is substantially constrained by the inability to efficiently reuse the backbone when the tokenizer changes. This is a fundamental obstacle for low-resource adaptation, language expansion (e.g., new scripts), and cross-model distillation.

Previously, embedding initialization for new vocabularies depended primarily on surface-level heuristics: nearest-neighbor and mean-matching in static (FastText or GloVe) vector spaces, or sparse mixtures anchored to pre-existing token embeddings (Haltiuk et al., 24 Oct 2025). Such approaches ignore the model’s actual use of token representations across depth—most crucially, the self-attention dynamics. MATT addresses this limitation by leveraging the structure and computations of the entire model during tokenizer transfer, ensuring that new tokens are integrated in a way that preserves both surface and internal semantics.

2. Core Methodologies

Multiple MATT variants exist, tailored for domain adaptation, cross-tokenizer transfer, and multilingual expansion. Prominent exemplars include:

2.1 Adaptive Tokenization (AT)

Adaptive Tokenization identifies candidate subword sequences that are statistically distinctive in a target domain and augments the model’s vocabulary accordingly. Candidate subwords are scored by the pointwise Kullback–Leibler (KL) divergence between their conditional probability distributions in the base and in-domain corpora:

$$R(s) = D_{kl}(P_D(s) \parallel P_S(s)) = P_D(s) \cdot \log \frac{P_D(s)}{P_S(s)}$$

Sequences with high $R(s)$ capture multi-token phrases frequent in the new domain but unusual in the base. The top $N$ (e.g., 10,000) such sequences are appended as new tokens, and their embeddings are initialized either by averaging the contained base-token embeddings or via linear projections from domain-specific static embeddings. Importantly, the Transformer body remains unchanged; only the embedding matrix is enlarged (Sachidananda et al., 2021).

2.2 TokAlign

TokAlign reframes the MATT problem as one of explicit one-to-one token alignment between source and target vocabularies. Using co-occurrence statistics from a shared corpus, it learns a mapping matrix $M \in \{0,1\}^{|V_s| \times |V_t|}$ maximizing token-pair cosine similarity in embedding space. The source model’s embeddings and LM head parameters are then directly transferred to the aligned target tokens, with random initializations for unmatched tokens. A brief two-stage fine-tuning (Language-Adaptation Tuning, LAT) is then performed: first updating only the new rows, then unfreezing the full model (Li et al., 4 Jun 2025).

2.3 Cross-Tokenizer Distillation via Approximate Likelihood Matching (ALM)

ALM addresses tokenization disparities by matching the chunk-level predictive likelihoods of teacher and student models on raw strings:

• Tokenize input $x$ as $T_T(x)$ (teacher) and $T_S(x)$ (student).
• Identify aligned token chunks whose detokenizations yield identical byte-spans.
• The loss matches the probability that the next chunk of text is predicted by both teacher and student, using a binarised $f$-divergence (typically KL), temperature-scaled for stability.

This framework allows pure distillation independent of ground truth and accommodates radically distinct vocabularies (e.g., subword to byte-level). When teacher and student architectures match, MATT enables tokenizer self-distillation, facilitating unified model ensembles and training-free embedding transfer via small hypernetworks (Minixhofer et al., 25 Mar 2025).

2.4 Attention Influence Modeling (AIM)

Attention Influence Modeling targets the propagation of information through self-attention. AIM aligns segment-level attention-weighted value vectors between the source model (with original tokenizer) and the student model (with new tokenizer):

$$L_{AIM} = \frac{2}{m(m+1)} \sum_{i=1}^{m} \sum_{j=1}^{i} \mathcal{L}^*(\mathfrak{s}_{\ell_T(i),j},\,\mathfrak{s}'_{\ell_{T'}(i),j})$$

Here, $\mathfrak{s}_{i,k}$ aggregates attention outputs for each segment, based on alignment of tokenization boundaries. Only the new-token embeddings are updated, with the rest of the network weights frozen. This results in new embeddings that not only share surface statistical properties but also reproduce the teacher’s attention patterns in higher transformer layers (Haltiuk et al., 24 Oct 2025).

3. Experimental Evidence and Comparative Results

MATT and its variants have been rigorously tested across various domains, tokenization schemes, and languages.

• Domain adaptation: In tasks spanning biomedical, computer science, news, and sentiment classification, Adaptive Tokenization recovers $>97\%$ of the accuracy gains of full domain-adaptive pretraining while being 72× faster and requiring only a 6% parameter increase (mostly in the embedding layer) (Sachidananda et al., 2021).
• Vocabulary adaptation: TokAlign achieves a $29.2\%$ reduction in sequence length for multilingual corpora (13 languages), reduces perplexity (e.g., from $3.4\cdot10^{2}$ to $1.2\cdot10^{2}$), and converges in as few as 5,000 steps (Li et al., 4 Jun 2025).
• Tokenizer-agnostic distillation: ALM (MATT) recovers nearly all of the original model’s performance on standard zero-shot NLP benchmarks, surpassing earlier hybrid distillation methods by 4–12 accuracy points depending on the granularity of the transfer. Ensemble models using MATT self-distillation rival the performance of much larger models trained from scratch (Minixhofer et al., 25 Mar 2025).
• Attention-based transfer: AIM recovers approximately 96% of discriminative accuracy and $>60\%$ of generative BLEU scores in cross-script (e.g., English to Ukrainian) transfer, outperforming all static embedding-initialization baselines, including WECHSEL, FOCUS, and TokAlign (Haltiuk et al., 24 Oct 2025).

Table: Representative Performance Recovery of MATT Variants

Method	Recovery Rate (Discriminative)	Recovery Rate (Generative BLEU)	Relative Runtime
Adaptive Token.	>97%	N/A	72× faster than DAPT
TokAlign	≈100% (with short LAT)	N/A	~1.9× faster than baselines
ALM (MATT)	≈100%	Outperforms hybrids by 8–12 pts	+25% TFLOPs over SFT
AIM (MATT)	85–96%	60%+ BLEU	3–5 hrs on H100

4. Theoretical Insights

The effectiveness of MATT arises from its sensitivity to both the statistical properties of the domain and the internal structure of transformer models, rather than focusing solely on word- or subword-level similarities.

• Structural alignment: By aligning higher-layer attention interactions or chunk-level likelihoods, MATT mitigates the mismatch between superficial token identity and their functional role inside the network, allowing new tokens to integrate into learned attention and prediction pathways (Haltiuk et al., 24 Oct 2025, Minixhofer et al., 25 Mar 2025).
• Statistical targeting: The use of conditional divergence (e.g., KL) ensures that selected new tokens represent meaningful, high-frequency structures in the new domain, reducing reliance on deep contextual recomposition in the transformer body (Sachidananda et al., 2021).
• Embedding efficiency: Methods such as hypernetwork-based embedding projection and progressive fine-tuning minimize parameter bloat and initialization inefficiency, supporting rapid adaptation at constant or near-constant model capacity (Minixhofer et al., 25 Mar 2025, Li et al., 4 Jun 2025).

A plausible implication is that methods which propagate model-aware signals during tokenizer adaptation can generalize better in low-resource or non-alphabetic scripts without requiring prohibitively expensive end-to-end retraining.

5. Limitations and Open Directions

Several limitations remain:

• Input-output embedding tying: MATT (AIM) relies on tied embeddings; untied cases recover less performance, and resolving this may require lightweight mapping networks (Haltiuk et al., 24 Oct 2025).
• Hybridization needs: In highly granular conversions (e.g., subword to bytes), adding standard next-token SFT to pure ALM sometimes yields further gains, suggesting loss-weighting schedules may need to adapt dynamically (Minixhofer et al., 25 Mar 2025).
• Layer depth tradeoffs: Applying AIM across all transformer layers increases memory usage and run time, with best performance typically plateauing after covering approximately one-third of depth.
• Model scale: Most published experiments focus on 0.6–4B parameter models; scaling to 10B+ may require further empirical validation.
• Encoder-only and seq2seq: While theoretically extensible, adaptations beyond decoder-only architectures lack comprehensive evaluation.

6. Practical Recommendations and Applications

• Tokenizer upgrades and expansion: MATT is effective for injecting new scripts or tokens to support low-resource languages, specialized domains, or modified preprocessing without full retraining.
• Cross-model distillation: By unifying vocabularies, MATT enables model ensembling, token-level distillation, and functional comparison across architectures.
• Computational efficiency: The typical adaptation phase is completed in several hours on modern hardware, far below the cost of from-scratch pretraining or hypernetwork tuning across multiple tokenizers.

MATT constitutes a paradigm shift from purely static, embedding-centric approaches to model-aware, structure-preserving adaptation, aligning both token statistics and internal computation for robust and efficient transfer across tokenization boundaries (Sachidananda et al., 2021, Li et al., 4 Jun 2025, Minixhofer et al., 25 Mar 2025, Haltiuk et al., 24 Oct 2025).