Fast Vocabulary Transfer

• Fast Vocabulary Transfer is the process of adapting pretrained models by redesigning tokenization and embedding inheritance to suit corpus-specific requirements.
• It employs an intermediary masked language modeling step to realign token embeddings and improve performance on diverse downstream tasks.
• This method enhances model compression, inference speed, and cross-lingual retrieval by optimizing vocabulary design beyond mere token overlap.

Fast vocabulary transfer is a branch of transfer learning in which a pretrained model is adapted to a new tokenizer or vocabulary rather than forced to retain the vocabulary learned during pretraining. In the terminology introduced by "Fine-Tuning Transformers: Vocabulary Transfer" (Mosin et al., 2021), the central claim is that corpus-specific tokenization, combined with an initialization and fine-tuning strategy for the new vocabulary tokens, can speed up transfer and improve downstream performance. Subsequent work generalizes this idea across multilingual neural machine translation, low-resource cross-lingual transfer, domain adaptation, model compression, and sparse retrieval, while also showing that vocabulary overlap, although beneficial, is neither the only mechanism nor always the dominant one in transfer (Itkonen et al., 5 May 2026).

1. Definition and problem formulation

In standard transfer learning, model parameters are updated while the pretrained vocabulary remains unchanged. Multiple papers identify this as a structural limitation when the downstream corpus differs substantially in language, domain, or token statistics. "AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain" states that the pretrained vocabulary is suboptimal for downstream data when domain discrepancy exists, and proposes treating the vocabulary itself as an optimizable parameter (Hong et al., 2021). "Fine-Tuning Transformers: Vocabulary Transfer" makes the same point from the perspective of corpus-specific tokenization, while "Fast Vocabulary Transfer for LLM Compression" extends it to deployment settings where vocabulary size, embedding size, and sequence length directly affect model footprint and inference speed (Mosin et al., 2021, Gee et al., 2024).

A recurring technical question is what exactly is transferred when vocabularies change. The multilingual machine translation study "The Impact of Vocabulary Overlaps on Knowledge Transfer in Multilingual Machine Translation" separates the effects of a joint vocabulary from the effects of shared hidden layers by comparing joint and disjoint vocabularies. Its experiments show that models with a joint vocabulary generally outperform those with disjoint vocabularies, but that knowledge transfer persists even when vocabulary overlap is reduced to near-zero, confirming that transfer is not reducible to shared token embeddings alone (Itkonen et al., 5 May 2026).

This distinction also appears in cross-lingual retrieval. "Instance-based Transfer Learning for Multilingual Deep Retrieval" reports target-language improvement after adding data from an auxiliary language even with no vocabulary in common with the target, and attributes this to transitive vocabulary overlaps through pivot languages (Arnold et al., 2019). Taken together, these results establish fast vocabulary transfer as a problem of jointly redesigning tokenization, embedding initialization, and parameter reuse, rather than a simple question of maximizing lexical overlap.

2. Core transfer mechanisms

The canonical mechanism in transformer fine-tuning is to learn a new tokenizer on downstream data, rebuild the embedding layer, initialize new token embeddings from the old vocabulary wherever possible, and run an intermediary masked language modelling step before task fine-tuning. The most explicit formulation is VIPI, Vocabulary Initialization with Partial Inheritance, introduced in (Mosin et al., 2021). For a new token $\widetilde{t}_k$, VIPI copies the old embedding if there is a direct match, otherwise averages embeddings over minimal preferred partitions into old tokens, and otherwise uses random initialization:

$$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$

The same paper reports that the intermediary MLM adaptation step is crucial for achieving the best performance with the new vocabulary (Mosin et al., 2021).

A simpler initialization rule appears in "Fast Vocabulary Transfer for LLM Compression". Its Fast Vocabulary Transfer method keeps overlapping embeddings unchanged and initializes unseen tokens by re-tokenizing each new token with the original tokenizer and averaging the embeddings of the resulting old tokens:

$$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$

After this transfer step, the model is continued with MLM training for one epoch on in-domain data and then fine-tuned on downstream tasks (Gee et al., 2024).

Vocabulary expansion is the central operator in AVocaDo. There, domain-specific tokens are added to the pretrained vocabulary according to a tokenization statistic, and the added embeddings are regularized by contrasting representations produced under the original and adapted tokenizations. The regularization term aligns the two representations in the same encoder and is combined with the task cross-entropy objective as $$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$, with $\lambda$ set to $1.0$ in the reported experiments (Hong et al., 2021).

A distinct but related line of work removes trial training from vocabulary design itself. "Vocabulary Learning via Optimal Transport for Neural Machine Translation" formulates vocabularization as an optimal transport problem and proposes VOLT, which selects vocabularies by maximizing the Marginal Utility of Vocabularization without downstream trial training. On WMT-14 English-German, VOLT reports almost $70\%$ vocabulary size reduction and a $0.5$ BLEU gain, while reducing search time from $384$ GPU hours to $30$ GPU hours (Xu et al., 2020).

3. Vocabulary overlap and multilingual machine translation

Vocabulary overlap is most thoroughly analyzed in multilingual neural machine translation. In (Itkonen et al., 5 May 2026), joint vocabularies were created with a single multilingual SentencePiece tokenizer, whereas disjoint vocabularies were enforced by prefixing auxiliary-language tokens so that the shared vocabulary approached zero apart from special tokens. For German–Swedish, the joint vocabulary overlap was $$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$0 tokens ($$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$1); for German–Finnish, it was $$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$2 tokens ($$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$3). The overlap statistic was defined as

$$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$4

The experiments show both the benefit and the limits of overlap. With $$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$5M training lines, adding Swedish to German with a joint vocabulary yielded $$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$6 BLEU and $$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$7 ChrF over the bilingual baseline; adding Finnish yielded $$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$8 BLEU and $$\widetilde{\Theta}[\widetilde{t}_k] = \begin{cases} \Theta[\widetilde{t}_k] & \text{if } \widetilde{t}_k \in V \ \frac{1}{|\mathcal{P}|}\sum_{p \in \mathcal{P}} \frac{1}{|p|}\sum_{t \in p}\Theta[t] & \text{if } \mathcal{P} \neq \emptyset \ \text{random} & \text{otherwise.} \end{cases}$$9 ChrF. Joint-vocabulary models were on average $$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$0 BLEU and $$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$1 ChrF better than disjoint-vocabulary models, and the drop was larger for the related pair German–Swedish ($$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$2 BLEU) than for German–Finnish ($$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$3 BLEU). At the same time, even with disjoint vocabularies, transfer persisted, and the paper concludes that domain-match and language relatedness are more important than a joint vocabulary alone (Itkonen et al., 5 May 2026).

Other multilingual studies push the overlap argument further. "Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages" proposes Overlap BPE (OBPE), which modifies BPE to optimize both corpus compression and inter-language overlap. In low-resource related-language settings, the paper reports that synthetically reducing the overlap to zero can cause as much as a four-fold drop in zero-shot transfer accuracy (Patil et al., 2022). By contrast, "A Universal Parent Model for Low-Resource Neural Machine Translation Transfer" argues for a fixed universal vocabulary, learned jointly over romanized corpora from $$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$4 parent languages plus English, precisely to avoid rebuilding vocabularies for each new child pair. That model uses a constant $$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$5k BPE vocabulary and nearly $$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$6 million sentences of parent training data, and is designed so that new low-resource source-to-English systems can be built more quickly than competing methods (Gheini et al., 2019).

These findings are not contradictory. They indicate that overlap is especially valuable when languages are related and when zero-shot or low-shot lexical anchoring is required, but they also show that shared internal parameters and broad-coverage segmentation can sustain transfer in the absence of direct overlap.

4. Transfer without shared vocabularies

Several influential methods were developed precisely because shared vocabularies are often unavailable or impractical. "Trivial Transfer Learning for Low-Resource Neural Machine Translation" uses a shared subword vocabulary learned on the concatenation of parent and child corpora, then trains a high-resource parent model and continues training on the child corpus without resetting parameters or optimizer state. The paper reports gains even for unrelated languages with different alphabets, including up to $$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$7 BLEU for Finnish-English to Estonian-English transfer and non-negligible gains when parent and child share no language at all (Kocmi et al., 2018).

"Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies" removes the need for shared vocabularies entirely. It aligns child-language embeddings to the parent source embedding space with a linear mapping, injects artificial noise during parent training to make the encoder more language-agnostic, and generates synthetic training data from parent parallel data without back-translation. Across five low-resource tasks, the full method improves plain NMT transfer by up to $$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$8 BLEU and outperforms multilingual joint training by a large margin (Kim et al., 2019).

Dynamic and dictionary-based approaches extend this line. "Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary" updates the vocabulary when new languages arrive, preserving embeddings for overlapping tokens and randomly initializing new items, and reports BLEU gains ranging from $$E_{in}(t_i) = \frac{1}{|T(t_i)|}\sum_{t_j \in T(t_i)} E_{gen}(t_j).$$9 up to $$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$0, while reaching higher performance after training up to $$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$1 of the total training steps required from scratch (Lakew et al., 2018). "Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries" uses bilingual dictionaries, byte-level BPE, fast_align, and iterative removal of mapped subwords from the tokenizer so that longer mapped units fall back to shorter subwords. The paper reports mapped subword coverage of $$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$2–$$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$3 of subword types in most cases and strong gains on low-resource languages such as Uyghur, Khmer, and Manchu (Sakajo et al., 2 Jun 2025).

A more structural redesign appears in "Parallel Tokenizers: Rethinking Vocabulary Design for Cross-Lingual Transfer". There, monolingual tokenizers are trained separately and then aligned so that semantically equivalent words receive the same vocabulary index across languages. On thirteen low-resource languages, the resulting Parallel-13L model attains the best reported fertility score ($$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$4), the best parity score ($$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$5), and the highest task performance across sentiment analysis, hate speech detection, emotion classification, and sentence-embedding similarity among the compared multilingual baselines (Kautsar et al., 7 Oct 2025).

5. Domain adaptation, compression, and retrieval

Domain adaptation provides some of the clearest evidence for fast vocabulary transfer. AVocaDo expands pretrained vocabularies using only downstream data and reports consistent improvements across biomedical, computer science, news, and reviews. Reported gains include BERT on ChemProt from $$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$6 to $$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$7, BERT on ACL-ARC from $$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$8 to $$\mathcal{L} = \mathcal{L}_{CE} + \lambda \mathcal{L}_{reg}$$9, and BERT on Amazon from $\lambda$0 to $\lambda$1. The paper also reports that removing the regularization loss degrades performance by $\lambda$2–$\lambda$3 points depending on the task (Hong et al., 2021).

In biomedical NLP, "Vocabulary Transfer for Biomedical Texts: Add Tokens if You Can Not Add Data" studies vocabulary extension during fine-tuning. On the Kaggle medical text classification task, increasing vocabulary size from $\lambda$4k to $\lambda$5k or $\lambda$6k improves accuracy only when intermediate MLM is used: without MLM, transfer can hurt, but with MLM the reported gains are $\lambda$7 for $\lambda$8k $\lambda$9k and $1.0$0 for $1.0$1k $1.0$2k. The same study reports that, after MLM adaptation, larger vocabularies can also decrease inference time because biomedical terms are represented with fewer, longer tokens (Singh et al., 2022).

Compression-oriented work makes the efficiency argument explicit. "Fast Vocabulary Transfer for LLM Compression" reports up to $1.0$3 model size reduction from vocabulary transfer alone, and up to $1.0$4 when vocabulary transfer is combined with knowledge distillation. Reported inference speedups reach $1.0$5 for vocabulary transfer alone and up to $1.0$6 when combined with distillation, with the strongest gains in specialized domains such as medical and legal text (Gee et al., 2024).

Sparse retrieval shows that vocabulary transfer can alter not only efficiency but also the effective inductive bias of a model. "Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps" identifies a vocabulary gap created by raw, case-sensitive tokenizers and proposes Vocabulary Transfer with Semantic Initialization and Activation Potential Calibration. The paper reports that VT enables ModernBERT to reach $1.0$7 nDCG on BEIR, a $1.0$8 improvement, and argues that the lag of modern encoders in learned sparse retrieval is not an architectural deficiency but a vocabulary mismatch (Geng et al., 20 Apr 2026).

6. Misconceptions, evaluation, and open directions

A persistent misconception is that fast vocabulary transfer is equivalent to maximizing shared subwords. The evidence does not support that simplification. Joint vocabularies improve transfer in multilingual NMT, especially for related languages, but transfer also survives with disjoint vocabularies (Itkonen et al., 5 May 2026). Direct overlap explains only part of multilingual deep retrieval gains, and transitive overlap can support transfer even when two languages share no vocabulary (Arnold et al., 2019). Conversely, in related-language low-resource settings, zero-shot performance can collapse when overlap is synthetically removed (Patil et al., 2022).

A second misconception is that vocabulary modification is useful only for low-resource multilingual settings. The cited literature shows otherwise. It appears in downstream-domain adaptation with fixed pretrained encoders (Hong et al., 2021), in biomedical text classification under data scarcity (Singh et al., 2022), in model compression pipelines (Gee et al., 2024), in multilingual vocabulary allocation for pretraining (Zheng et al., 2021), and in sparse retrieval systems where normalization of the vocabulary changes generalization behavior (Geng et al., 20 Apr 2026).

Evaluation practices therefore vary by application. Machine translation studies report BLEU and ChrF (Itkonen et al., 5 May 2026); transfer in classification settings is usually measured with F1 or accuracy (Hong et al., 2021, Singh et al., 2022); compression work adds model size and inference-time measurements (Gee et al., 2024); sparse retrieval uses nDCG on BEIR (Geng et al., 20 Apr 2026). This suggests that fast vocabulary transfer is best understood not as a single algorithm but as a design space whose objectives differ across tasks: lexical sharing, tokenization compactness, representation compatibility, parameter reuse, and runtime efficiency.

Two research directions stand out. First, automated vocabulary design is increasingly replacing heuristic tokenizer selection. VOLT uses optimal transport to learn vocabularies without trial training (Xu et al., 2020), while VoCap allocates vocabulary capacity per language and combines it with k-NN-based target sampling to make large-vocabulary multilingual pretraining faster; the reported VoCap$1.0$9 + kNN model reaches an average score of $70\%$0 with $70\%$1 pre-training speed relative to the reported baseline (Zheng et al., 2021). Second, low-resource cross-lingual transfer is moving from corpus-heavy methods toward lighter resources such as bilingual dictionaries and aligned token indices (Sakajo et al., 2 Jun 2025, Kautsar et al., 7 Oct 2025).

Within this literature, fast vocabulary transfer denotes a shift in what is considered transferable. Instead of treating the tokenizer as fixed infrastructure, these methods treat vocabulary design, embedding inheritance, and tokenization statistics as active components of transfer learning.