Vocabulary Transfer in NLP

• Vocabulary Transfer (VT) is a transfer-learning approach that adjusts a model's lexical interface by modifying its tokenizer, embeddings, or output vocabulary.
• It covers techniques such as domain adaptation, multilingual continual learning, compression, and inference-time safety, often improving performance when coupled with masked language modeling.
• Empirical results demonstrate VT's practical benefits, with reported gains like up to +13.63 BLEU in multilingual setups and improved accuracy in domain-specific adaptations.

Searching arXiv for recent and foundational papers on Vocabulary Transfer across NLP settings. Vocabulary Transfer (VT) is a family of transfer-learning procedures in which a pretrained model is adapted not only through parameter updates but also through an explicit change to its lexical interface: the tokenizer vocabulary, the embedding table tied to that vocabulary, or the output vocabulary used at inference. In this literature, VT includes vocabulary extension for domain adaptation, vocabulary substitution for multilingual continual learning, vocabulary matching across languages, vocabulary trimming for compression, and cross-vocabulary bridging for inference-time control. The common premise is that a fixed pretrained vocabulary is often a bottleneck when the downstream corpus, language, task objective, or deployment regime differs materially from the setting in which the model was originally trained (Mosin et al., 2021, Lakew et al., 2018, Ushio et al., 2023, Chawla et al., 10 Jun 2026).

1. Scope and major formulations

VT emerged in several adjacent subfields rather than as a single canonical algorithm. In multilingual neural machine translation, dynamic-vocabulary transfer was introduced as a way to extend a trained system to new language pairs without retraining from scratch, while preserving embeddings for overlapping subwords and inserting new items when required (Lakew et al., 2018). In Transformer fine-tuning, VT was later formalized as the problem of finding a corpus-specific downstream tokenization, initializing its embeddings from the pretrained vocabulary, and adapting the model so that the new tokenization becomes usable for classification or masked language modeling (Mosin et al., 2021). Domain-adaptive formulations then treated the vocabulary itself as an optimizable parameter, notably in AVocaDo and in biomedical vocabulary extension work, both of which argued that domain shift is expressed partly as a tokenization mismatch rather than solely as a parameter mismatch (Hong et al., 2021, Singh et al., 2022).

Cross-lingual work broadened the concept further. Some methods transfer embeddings through bilingual dictionaries or vocabulary matching when target-language corpora are scarce (Rybak, 2024, Sakajo et al., 2 Jun 2025). Others investigate how much transfer in multilingual models actually derives from shared tokens, disjoint vocabularies, or transitive overlap through pivot languages (Itkonen et al., 5 May 2026, Arnold et al., 2019). A separate line of work uses vocabulary trimming to specialize multilingual models into smaller language-specific models or to accelerate inference by restricting the active vocabulary (Ushio et al., 2023, Bogoychev et al., 2023). More recent work extends VT to cross-vocabulary safety transfer at decoding time and to sparse retrieval, where vocabulary normalization and coarse-graining are treated as first-class adaptation problems (Chawla et al., 10 Jun 2026, Geng et al., 20 Apr 2026).

Setting	VT operation	Representative papers
Domain fine-tuning	Extend or rebuild downstream vocabulary; initialize new embeddings from old tokens	(Mosin et al., 2021, Hong et al., 2021, Singh et al., 2022)
Multilingual MT	Dynamically update or substitute vocabularies as languages arrive	(Lakew et al., 2018, Garcia et al., 2021, Itkonen et al., 5 May 2026)
Low-resource cross-lingual transfer	Match target tokens to source tokens through dictionaries or decomposition	(Rybak, 2024, Sakajo et al., 2 Jun 2025, Kim et al., 2019)
Compression and specialization	Trim multilingual vocabularies to target-language subsets	(Ushio et al., 2023, Bogoychev et al., 2023, Gee et al., 2024)
Retrieval and safety	Transfer behavior across incompatible vocabularies at training or inference time	(Geng et al., 20 Apr 2026, Chawla et al., 10 Jun 2026)

This dispersion of formulations means that “Vocabulary Transfer” is best understood as a unifying abstraction: it denotes transfer at the level of lexical parameterization, not merely at the level of encoder or decoder weights.

2. Core mechanisms and formal operations

A standard formalization defines an original vocabulary

$V = \{t_k, v_k\}_{k=1}^{M}$

and a downstream or target vocabulary

$$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$

VT then specifies how $\widetilde{V}$ is constructed and how the new embeddings $\widetilde{v}_k$ are initialized from the pretrained vocabulary. In the fine-tuning literature, two basic heuristics recur. Under matched transfer, an embedding is copied when a new token exactly matches an old token. Under averaged transfer, a new token is decomposed into original-vocabulary tokens and its embedding is initialized by averaging the embeddings of the chosen decomposition. In the biomedical formulation, the decomposition is chosen by preferring the fewest tokens and, if tied, the longest token (Singh et al., 2022). The broader Transformer fine-tuning paper packages this strategy as “Vocabulary Initialization with Partial Inheritance” (VIPI), which averages embeddings over surviving partitions after filtering by minimum partition length and longest-token criteria (Mosin et al., 2021).

A second family of methods changes not just initialization but vocabulary selection itself. AVocaDo constructs a domain-specific vocabulary $V_{\mathcal D}$ and forms an adapted vocabulary

$$V_{\mathcal A} \leftarrow V_{\mathcal P} \cup \{V_{\mathcal D_i}\}_{i=0}^{\alpha},$$

then keeps expanding it until a tokenization statistic, the fragment score,

$$f_{\bold C}(V) = \frac{\text{the number of subwords tokenized by }V}{\text{the number of words in }\bold C},$$

falls below a threshold. This makes vocabulary growth conditional on reduced over-fragmentation rather than on raw frequency alone. AVocaDo also adds a contrastive regularization term that aligns representations of the same input under the pretrained and adapted tokenizations, with total objective $$\mathcal L = \mathcal L_{CE} + \lambda \mathcal L_{reg}$$ and $\lambda = 1.0$ in all experiments (Hong et al., 2021).

Cross-lingual VT introduces additional operators. In the dictionary-based approach for low-resource languages, target dictionary entries and English definitions are treated like a parallel corpus; target and source subwords are aligned with fast_align, mapped iteratively, and then removed from the tokenizer so that BPE falls back to shorter segments in the next round. The target embedding is initialized as

$$\boldsymbol{e}_t^T = \sum_{s \in \mathcal{M}_t} c(s|t) \cdot \boldsymbol{e}_s^S,$$

where the weights are relative alignment counts (Sakajo et al., 2 Jun 2025). In vocabulary matching for Silesian and Kashubian, a target token first consults a bilingual dictionary; if its Polish translation exists in HerBERT’s vocabulary, the Polish embedding is copied, otherwise the target token is decomposed by the source tokenizer and averaged (Rybak, 2024). A more radical alternative is to abandon shared discrete vocabularies entirely and map child-language monolingual embeddings into the pretrained parent source space through a learned linear transform

$$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$0

thereby performing VT in continuous embedding space rather than through token identity (Kim et al., 2019).

Across these variants, a recurring empirical lesson is that initialization alone is usually insufficient. Many papers insert an intermediate masked language modeling stage, or a functionally equivalent adaptation stage, between vocabulary remapping and downstream fine-tuning (Mosin et al., 2021, Singh et al., 2022, Gee et al., 2024).

3. Domain adaptation and specialized corpora

Domain-adaptive VT is motivated by the observation that pretrained vocabularies often tokenize specialized corpora inefficiently. Biomedical text is the clearest case in the available literature: it contains abbreviations, acronyms, digits, capitalization, special characters, codes, timestamps, and complex compound terms, and biomedical datasets are often small because of privacy constraints and restricted access to patient records. The biomedical VT study therefore frames vocabulary extension as a way to improve tokenization without requiring more labeled data, starting from a BERT model with a 16,000-token baseline vocabulary and transferring to 8,000, 32,000, or 64,000 tokens (Singh et al., 2022).

Its central empirical result is that vocabulary extension improves classification only when paired with intermediate MLM. On the Kaggle Medical dataset, relative accuracy changes against standard 16k-token fine-tuning were negative for “Transfer only” at all tested sizes, but positive for “Transfer + MLM” at larger sizes: 16,000 $$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$1 32,000 yielded $$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$2, and 16,000 $$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$3 64,000 yielded $$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$4, the best reported result. The same study reports an inference-time trade-off: without MLM, larger vocabularies tend to increase inference time, whereas with MLM, larger vocabularies can decrease inference time because the model adapts to the new tokenization and processes more compact representations (Singh et al., 2022). This directly challenges the assumption that vocabulary growth necessarily slows deployment.

AVocaDo generalizes the same intuition beyond biomedicine. It evaluates on ChemProt, ACL-ARC, HyperPartisan, and Amazon, and reports consistent improvements from vocabulary adaptation plus regularization: on ChemProt, BERT improves from $$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$5 to $$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$6; on ACL-ARC, BERT improves from $$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$7 to $$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$8; on HyperPartisan, BERT improves from $$\widetilde{V} = \{\widetilde{t}_k, \widetilde{v}_k\}_{k=1}^{N}.$$9 to $\widetilde{V}$0; and on Amazon, BERT improves from $\widetilde{V}$1 to $\widetilde{V}$2 (Hong et al., 2021). The paper’s qualitative examples emphasize semantically misleading segmentations such as “bluetooth” being split into “blue” and “tooth,” and its regularization ablation shows that vocabulary adaptation alone helps, but removing $\widetilde{V}$3 hurts performance across all datasets, especially Amazon (Hong et al., 2021).

The original VT fine-tuning paper reaches a similar conclusion on general-domain classification data. Using corpus-specific tokenization with VIPI and one epoch of MLM before classification, it reports gains over fixed-tokenization baselines on Quora, Sentiment140, and Hyperpartisan News, and shows especially strong domain-specific effects on medical diagnosis prediction. On MeDAL with vocabulary size 16,000, standard fine-tuning gives classifier accuracy $\widetilde{V}$4, MLM plus classifier gives $\widetilde{V}$5, VIPI gives $\widetilde{V}$6, and VIPI plus MLM gives $\widetilde{V}$7 (Mosin et al., 2021). This suggests that the benefit of VT increases with vocabulary richness, domain specificity, and the mismatch between pretraining and downstream terminology.

4. Cross-lingual and multilingual transfer

In multilingual settings, VT addresses both an engineering problem and a scientific question. The engineering problem is how to extend a model to new languages without restarting training. The scientific question is how much multilingual transfer actually depends on vocabulary overlap.

Dynamic vocabulary transfer in NMT addresses the incremental engineering problem directly. In “Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary,” the initial German-English model is extended to new language pairs by keeping the intersection between previous and current vocabularies, randomly initializing non-overlapping entries, and reusing the rest of the model parameters. Across five languages and low-resource settings of 5k and 50k parallel sentences, the reported gains range from $\widetilde{V}$8 up to $\widetilde{V}$9 BLEU, and transferred models reach higher performance after training up to $\widetilde{v}_k$0 of the total training steps compared with training from scratch (Lakew et al., 2018). A later continual-learning formulation rebuilds the vocabulary as if new languages had been present from the beginning and then substitutes it into the multilingual model while reusing shared embeddings. In that setting, overlap between old and rebuilt vocabularies is often more than $\widetilde{v}_k$1, and the method incurs only minor degradation on previously learned directions while remaining competitive even when only monolingual data is available for the new languages (Garcia et al., 2021).

The question of overlap has produced a more nuanced empirical record than early shared-vocabulary narratives suggested. In out-of-domain multilingual MT with German$\widetilde{v}_k$2English as the main task and Swedish or Finnish as auxiliary languages, joint vocabularies outperform disjoint ones, but the difference is limited. German+Swedish joint vocabulary yields 30.3 BLEU / 51.1 ChrF, German+Finnish joint yields 28.4 / 49.3, and disjoint models still beat the baseline, with Swedish disjoint at 28.6 / 49.9 and Finnish disjoint at 27.3 / 48.7. The average effect of switching from joint to disjoint vocabularies is a drop of 1.4 BLEU and 0.9 ChrF, leading the authors to conclude that vocabulary overlap helps, but domain match and language relatedness matter more (Itkonen et al., 5 May 2026). In multilingual deep retrieval, the same theme appears in a different form: positive transfer is observed on all 35 target languages and two tasks, and the authors show that auxiliary-language gains can persist even when target and auxiliary languages share no direct vocabulary, because transitive overlap through pivot languages still propagates information in the shared space (Arnold et al., 2019).

At the same time, some studies show that overlap can be decisive in specific regimes. OBPE modifies BPE so that vocabulary generation explicitly rewards overlap between related high-resource and low-resource languages. In related, low-resource settings, synthetically reducing overlap to zero can cause as much as a four-fold drop in zero-shot transfer accuracy, with Hindi–Marathi low-resource NER dropping by $\widetilde{v}_k$3 and text classification by $\widetilde{v}_k$4 in the reported experiments (Patil et al., 2022). A complementary “beyond shared vocabulary” line argues that token identity is too crude a proxy for lexical transfer and instead builds graph-based word equivalence classes from alignment data. Its GraphMerge model improves average BLEU from 26.1 to 28.4 on IWSLT14 and from 28.0 to 29.1 on EC30, while keeping inference latency exactly the same as baseline (Wu et al., 2023).

Low-resource language adaptation gives VT a more lexical form. Vocabulary matching from Polish HerBERT to Silesian and Kashubian shows that simply copying the encoder is not enough: the “Not matched, HerBERT init” model scores 48.71 on Silesian and 46.08 on Kashubian for masked word prediction, whereas “Matched HerBERT” reaches 60.27 and 59.13. On Silesian passage retrieval, matched HerBERT improves over fine-tuned HerBERT from 92.00 to 96.00 Accuracy@10 and from 79.72 to 81.79 NDCG@10 (Rybak, 2024). Dictionary-based VT extends this logic to script-distant and under-documented languages: across German, Japanese, Old English, Uyghur, Sanskrit, Khmer, and Manchu, the reported mapped-coverage rates range from 77.35% to 97.27%, and the method often outperforms overlap-based baselines for Uyghur, Khmer, Sanskrit, and Manchu (Sakajo et al., 2 Jun 2025).

A plausible implication is that multilingual VT is not a single mechanism. The literature supports at least four partially independent transfer channels: direct token overlap, continuous embedding alignment, domain-aligned auxiliary data, and transitive connectivity through other languages.

5. Compression, trimming, and deployment-time specialization

A different branch of VT treats vocabulary not as a source of downstream generalization but as a source of model size and latency. Vocabulary trimming reduces a multilingual model to a target-language-specific sub-vocabulary by deleting tokens irrelevant to the target language. The main multilingual-LM compression paper proposes two schedules—Pre-FT VT and Post-FT VT—and shows that VT can retain original performance while using much less of the vocabulary. It states that in general around 50% of the original vocabulary size is enough, and that with less than 40% of full model parameters, original performance on QA/QG can generally be maintained (Ushio et al., 2023). Specific examples include mT5 on Korean, where vocabulary is reduced to 29.2% of original and parameters to 39.7%, and XLM-V on Arabic sentiment, where vocabulary is reduced to 11.8% and parameters to 20.2% (Ushio et al., 2023).

Inference-time trimming for autoregressive LLMs is more fragile. The study of Unicode-based and corpus-based heuristics finds that VT reduces the memory usage of small models by nearly 50% and has an upper bound of 25% improvement in generation speed, but does not perform consistently well for each language and yields diminishing returns in larger models (Bogoychev et al., 2023). On GPU, none of the tested selection methods improve inference speed in a meaningful way; on CPU, favorable cases on BLOOM-560M and BLOOM-1B7 show roughly 20% time improvement, while larger models see weaker gains (Bogoychev et al., 2023). This indicates that vocabulary restriction can help when the embedding/output layer is a large fraction of total computation, but not when other bottlenecks dominate.

Fast Vocabulary Transfer for model compression takes a middle position between retraining and heuristic trimming. It trains an in-domain tokenizer, copies embeddings for shared tokens,

$\widetilde{v}_k$5

and initializes unseen tokens by averaging the embeddings of their tokenized pieces,

$\widetilde{v}_k$6

This “Fast Vocabulary Transfer” (FVT) reduces model size by about 5.14%, 10.28%, and 15.42% for the 75%, 50%, and 25% vocabulary settings, respectively, and yields speedups up to 1.40× on ADE. When combined with knowledge distillation, the reported reduction reaches up to 54.98% with speedup up to 2.76× (Gee et al., 2024). The paper presents VT as complementary to pruning, quantization, and distillation because it compresses the vocabulary-dependent embedding interface rather than the Transformer stack itself.

These compression results complicate a common assumption in the VT literature. Adding tokens can improve modeling in domain adaptation, yet removing tokens can also improve deployment efficiency. The two directions are not contradictory: they optimize different objectives under different constraints.

6. Retrieval, safety, and other late-stage extensions

Recent work has extended VT beyond conventional fine-tuning. In learned sparse retrieval, the problem is not domain coverage but the “Vocabulary Gap”: modern encoders such as ModernBERT and RoBERTa-large use raw, case-sensitive vocabularies optimized for reconstruction, whereas sparse retrieval benefits from more normalized vocabularies such as bert-base-uncased. The proposed VT framework migrates a pretrained encoder from source vocabulary $\widetilde{v}_k$7 to target vocabulary $\widetilde{v}_k$8 through overlap preservation, semantic initialization, prior-aware bias transfer, overlap-aware masking, and Activation Potential Calibration (APC). Empirically, ModernBERT-VT reaches 52.4 average BEIR nDCG@10, a +4.7 improvement over naive ModernBERT sparse fine-tuning, while RoBERTa-large is restored from 1.4 BEIR under naive sparse fine-tuning to 51.3 with VT (Geng et al., 20 Apr 2026). The paper’s theoretical account uses a many-to-one normalizer $\widetilde{v}_k$9 and a row-stochastic aggregation matrix $V_{\mathcal D}$0 to argue that coarse-graining can tighten generalization bounds if representation-compatibility is preserved.

Inference-time safety transfer introduces a different cross-vocabulary problem: mixing logits from a safe anchor model with a specialist draft model normally requires a shared tokenizer. ALIGNBEAM removes that assumption by decoding anchor top-$V_{\mathcal D}$1 tokens to text, re-encoding them with the draft tokenizer, and then mixing probabilities token-by-token in the draft space. The mixing rule is

$V_{\mathcal D}$2

applied when an anchor token re-encodes to a single draft token (Chawla et al., 10 Jun 2026). Across cross-vocabulary and same-vocabulary evaluation pairs, the reported macro average refusal rate on HarmBench plus AdvBench rises from 35.1% to 87.8%, while utility loss is about 0.4 percentage points on GSM8K and MedQA (Chawla et al., 10 Jun 2026). This is VT without weight updates: the transfer occurs entirely at decoding time.

A broader inference is that VT has ceased to be merely a tokenizer-engineering technique. It is now used to mediate incompatibilities between pretrained lexical spaces, deployment constraints, and behavioral control objectives.

7. Recurring findings, misconceptions, and open problems

Several claims recur across otherwise disjoint VT literatures. First, changing the vocabulary alone is rarely enough. In biomedical classification, “Transfer only” is often negative, whereas “Transfer + MLM” produces the gains (Singh et al., 2022). In sparse retrieval, semantic initialization without APC is insufficient, and the removal of APC can make the adapted model worse than the direct baseline on BEIR (Geng et al., 20 Apr 2026). In model compression, FVT substantially outperforms Partial Vocabulary Transfer, showing that random initialization of unseen tokens is usually inadequate (Gee et al., 2024).

Second, shared vocabulary is neither necessary nor sufficient for transfer. It helps in related-language and low-resource settings, as OBPE demonstrates (Patil et al., 2022), but disjoint-vocabulary multilingual MT still transfers through shared hidden layers (Itkonen et al., 5 May 2026), and multilingual retrieval can improve through transitive vocabulary overlaps even when there is no direct lexical overlap between auxiliary and target languages (Arnold et al., 2019). Conversely, merely lowercasing inputs or sharing surface forms can fail when semantic integrity is disrupted, as shown in sparse retrieval and in cross-script settings (Geng et al., 20 Apr 2026, Sakajo et al., 2 Jun 2025).

Third, the optimal vocabulary transformation is objective-dependent. Domain adaptation often prefers vocabulary extension or reconstruction of domain-specific units (Hong et al., 2021, Singh et al., 2022). Compression often prefers trimming (Ushio et al., 2023, Bogoychev et al., 2023). Multilingual continual learning benefits from substitution with high-overlap reuse (Garcia et al., 2021). Sparse retrieval benefits from coarse-graining and normalization (Geng et al., 20 Apr 2026). This suggests that “better vocabulary” is task-relative rather than absolute.

The open problems are correspondingly diverse. Several papers note that their results are limited to text classification, base-size BERT models, or a small set of datasets (Singh et al., 2022). Low-resource cross-lingual methods may depend on the availability and quality of bilingual dictionaries or on the segmentation behavior of the source tokenizer (Rybak, 2024, Sakajo et al., 2 Jun 2025). Graph-based multilingual methods remain sensitive to alignment quality and may become more computationally noticeable with very large vocabularies (Wu et al., 2023). In sparse retrieval, representation-compatibility is a theoretical condition rather than a directly measurable guarantee, and optimal normalization remains task- and domain-dependent (Geng et al., 20 Apr 2026). A related vocabulary-optimization literature, such as VOLT, addresses the choice of vocabulary size through optimal transport rather than direct transfer; this suggests a continuing convergence between vocabularization and VT, but the two are not identical (Xu et al., 2020).

Overall, VT has developed from a niche concern about embedding remapping into a general perspective on lexical adaptation. The literature supports a stable core thesis: pretrained models do not only transfer through hidden layers and attention weights; they also transfer, or fail to transfer, through the discrete vocabulary that mediates every input and output symbol.