Vocabulary Gap: A Multidisciplinary Overview

• Vocabulary gap is a concept describing mismatches between available lexical resources and the specific communicative, modeling, or retrieval requirements in various domains.
• The term encompasses challenges in adaptive tokenization, dynamic vocabulary expansion, and alignment between differing systems or languages.
• Insights from research highlight practical remedies ranging from entropy-guided vocabulary curricula to social scaffolding in clinical interventions and multilingual NLP.

“Vocabulary gap” is a polysemous technical term denoting a mismatch between lexical resources and communicative, representational, or retrieval requirements. In developmental and clinical work, it refers to deficits or atypical trajectories in word knowledge and use, such as the individualized, socially grounded gap in word knowledge and usage targeted by cooperative vocabulary-learning systems for children with Autism (Chodkiewicz et al., 13 May 2025). In language modeling, it refers to the disconnect between fixed vocabularies and evolving model uncertainty, or to tokenization mismatch across models, domains, or training stages (Yu, 25 Feb 2025). In lexical typology and multilingual NLP, it denotes a lexical gap: a concept that lacks a single-word lexicalization in a given language (Khishigsuren et al., 2022). In information retrieval, it denotes the failure of exact lexical matching when queries and relevant documents use different words for the same concept (Gysel, 2017). The term therefore names a family of structurally related but non-equivalent problems.

1. Core meanings and formalizations

The literature uses “vocabulary gap” in several distinct senses. These senses are linked by a common pattern—lexical inventories are misaligned with the task—but they differ in ontology, metrics, and remedies.

Domain	Formalization or criterion	Representative work
Lexical typology	$G=\{(c,l)\in C\times L \mid |W(c,l)|=0\}$	(Khishigsuren et al., 2022)
LLM pretraining	Static vocabulary fails to adapt to evolving “easy” versus “hard” sequences	(Yu, 25 Feb 2025)
Teacher–student LM	$\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$	(Shin et al., 24 Mar 2025)
Pretrain–finetune NLG	Token-set mismatch $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$	(Liu et al., 2021)
Retrieval	Relevant documents and queries use different words for the same concepts; or query tokens $t\notin T_i$	(Gysel, 2017, Li et al., 2024)
Sparse retrieval	Redundant surface forms inflate the discrete keyspace $V$	(Geng et al., 20 Apr 2026)

A central misconception is to treat these usages as interchangeable. In lexical typology, a gap is about the absence of a single-word lexicalization for a concept–language pair. In LLM training, the gap may instead arise because a tokenizer is static, because two models tokenize the same string differently, or because surface-form proliferation wastes capacity. In retrieval, the gap is neither lexical absence nor tokenizer mismatch, but failure of exact matching under bag-of-words or sparse token overlap.

This distinction matters because the remedy depends on the formulation. Cross-lingual lexical gaps may require paraphrase or explicit gap annotation rather than forced one-word translation (Khishigsuren et al., 2022). Pretraining inefficiency may call for entropy-guided vocabulary expansion rather than a larger fixed vocabulary (Yu, 25 Feb 2025). Query–document mismatch may call for document expansion or latent semantic matching rather than vocabulary redesign (Gysel, 2017).

2. Clinical and cognitive manifestations

In developmental and clinical research, vocabulary gaps are documented as delays and atypical patterns in receptive and expressive vocabulary growth. CoVoL operationalizes this problem for children with Autism Spectrum Disorder through a cooperative two-player “tacting” scenario derived from Applied Behavior Analysis’ Verbal Behavior protocols. In two-player mode, a pictogram appears, one player names the object aloud, the system recognizes the spoken word, and a correct response triggers a brief reward screen before the turn passes. The design emphasizes predictable repetition and individualized pacing, with prompts configurable from one every three seconds up to 55 per minute; multisensory reinforcement through planned audio prompts and feedback; configurable “human-like” reward delays via a minimum inter-stimulus interval $\Delta t_{\text{reward}}$; and attribute-focused scaffolding for color, shape, and category prompts. Recognition performance is instrumented through recognition accuracy $A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$ and mean response latency $L=\frac{1}{N}\sum_{i=1}^N t_{\text{resp},i}$. In Prototype I, two English Vosk models were tested on $N=118$ audio samples: the 128 MB model achieved $A>85\%$ but with latencies up to $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$0 on a mid-2013 MacBook Pro, whereas the 40 MB “small” model achieved only $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$1. Planned evaluation uses three study arms—child–therapist, child–parent, and peer-to-peer pairings—with video coding of turn-taking success rates, spontaneous tactical labeling outside prompts, and emergent social behaviors (Chodkiewicz et al., 13 May 2025).

This clinical use of the term foregrounds social grounding. CoVoL was explicitly motivated by the observation that existing technology-assisted vocabulary learning had not incorporated turn-taking in a way similar to real-world social contexts. The gap is therefore not only lexical coverage, but word use under cooperative interaction. The paper also identifies limitations that complicate closure of the gap, including stuttering, single-word rather than sentence-level recognition, and the fact that initial speech-model validation was based on a single adult speaker rather than dialectal or child-voice variation (Chodkiewicz et al., 13 May 2025).

A related cognitive perspective studies language production under hard vocabulary constraints. One formulation defines an allowed vocabulary $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$2 consisting of the $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$3 most frequent English word lemmas plus their inflected forms, with $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$4, and a hard constraint $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$5 iff every token in a generated sentence belongs to $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$6. In a human experiment with 192 questions and 144 native English speakers, responses under restricted vocabularies tracked greedy sampling more closely than globally optimal Sequential Monte Carlo sampling, though high-skill participants increased deletion and revision behavior at the tight vocabularies $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$7 and $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$8. Both humans and models also increased the relative frequency of semantically light words such as “thing,” “do,” and “people” as $\lvert V_s \cap V_t\rvert/\lvert V_s\rvert \ll 1$9 decreased (Clark et al., 14 May 2026). This suggests that vocabulary gaps can also be studied as constrained production problems in which communicative adequacy depends on planning under lexical scarcity.

3. Static vocabularies, curriculum, and scaling laws in LLM pretraining

One influential formulation states that modern LLMs rely on static vocabularies fixed before pretraining, unlike human language acquisition, which proceeds from smaller units to larger chunks. The resulting “vocabulary gap” is the disconnect between a static, one-shot vocabulary and the model’s evolving notions of “easy” versus “hard” sequences. “Scaling LLM Pre-training with Vocabulary Curriculum” addresses this by alternating model optimization with entropy-guided vocabulary expansion or contraction. The merge criterion requires monotonically decreasing next-token entropy along a candidate span and an entropy threshold $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$0. Empirically, bits-per-character follows an approximately log-linear dependence on vocabulary size, $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$1, with $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$2 under compute-matched static training and $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$3 under vocabulary curriculum. On enwiki8 with a small GPT model of about 10 M parameters, five incremental iterations expanded the vocabulary from 92 to 18,276 tokens, and the curriculum method improved BPC from 1.5303 to 1.5131 at 4,359 tokens and from 1.4637 to 1.3764 at 18,276 tokens relative to the compute-matching baseline (Yu, 25 Feb 2025).

This line of work argues that vocabulary growth should be adaptive rather than fixed. The reported “optimal computation allocation pattern” is that predictable multi-character sequences are merged into long tokens, while hard-to-predict spans remain short. In effect, the tokenizer becomes a learned allocator of modeling granularity rather than a static preprocessing artifact (Yu, 25 Feb 2025).

A complementary line studies vocabulary size as a scaling-law variable. “Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies” derives compute-optimal allocations in which non-vocabulary parameters and training data scale as $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$4, while vocabulary parameters scale as $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$5. The paper reports $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$6, $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$7, and $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$8, and predicts that a 70 B Llama-style model should have an optimal vocabulary of about 216 K tokens, roughly seven times larger than 32 K. In a 3 B-parameter setting at $$\lvert V_{\text{now}} \setminus V_{\text{prey}}\rvert \gg 0$$9 FLOPs, increasing vocabulary size from 32 K to 43 K improved ARC-Challenge from 29.1 to 32.0 under the same compute budget (Tao et al., 2024).

Taken together, these results suggest that the “vocabulary gap” in pretraining is not reducible to a single recommendation. One approach advocates dynamic vocabulary adaptation during training; another shows that even in static regimes, conventional vocabularies are often too small for larger models. The two perspectives are compatible: both treat tokenization as a scaling variable rather than a fixed design choice.

4. Alignment across models, tasks, and tokenizers

When two models assign different token boundaries to the same text, the vocabulary gap becomes an alignment problem. “Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling” formalizes this with a student vocabulary $t\notin T_i$0, a teacher vocabulary $t\notin T_i$1, and a gap when $t\notin T_i$2. The method aligns student and teacher tokens through character-offset spans, computing a one-to-many mapping in overall $t\notin T_i$3 time for a pair of token sequences. Instead of matching logits across incompatible vocabularies, it aggregates teacher causal-language-modeling loss over mapped spans and reweights student losses, keeping only the top-$t\notin T_i$4 largest loss gaps plus unmapped tokens. With a 1.1 B TinyLlama student and several $t\notin T_i$5 B teachers on OpenWebMath, a Qwen2.5-Math-Instruct teacher sharing only about $t\notin T_i$6 of its vocabulary with the student raised the average score across nine math-reasoning benchmarks from about 21.5 under naive continual pretraining to about 29.3, a 46% relative improvement, and outperformed universal logit distillation by 33% (Shin et al., 24 Mar 2025).

A separate pretrain–finetune formulation treats the vocabulary gap as subword mismatch between an upstream vocabulary $t\notin T_i$7 and a downstream vocabulary $t\notin T_i$8. “Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation” introduces an embedding generator that initializes unseen downstream tokens from morphologically similar upstream tokens, using subwords, hyperwords, and one of three generators: AVG-EG, ATT-EG, or PATT-EG. Training combines the original pretraining loss with a knowledge-distillation term weighted by $t\notin T_i$9. In En$V$0Zh domain adaptation, the best configuration, PATT-EG+KD, improved BLEU from 30.2 to 37.9 on the Thesis domain and from 52.1 to 54.3 on the Laws domain relative to vanilla pretrain–finetune. In transfer experiments, it improved WMT14 En$V$1De from 28.2 to 29.8 and SQuAD question generation from ROUGE-L 25.9 to 26.8, BLEU 3.31 to 3.55, and METEOR 9.27 to 9.86. The generator stabilized in about 20 K training steps and reduced average subword-sequence length, speeding decoding by about 50% (Liu et al., 2021).

Vocabulary replacement can also be framed as a token-alignment problem at model scale. TokAlign defines a one-to-one mapping matrix between source and target token IDs using token–token co-occurrence statistics from a mixed corpus of roughly 1 billion tokens, then rearranges embeddings and LM-head rows accordingly, followed by two-stage Language Adaptation Tuning. The method reduced normalized perplexity after initialization from about $V$2 under strong baselines to $V$3, restored vanilla performance in as few as 5k steps, and after vocabulary unification enabled token-level distillation to improve average accuracy on six QA and reasoning tasks from 50.6% under sentence-level distillation to 55.0%, using only 235M tokens (Li et al., 4 Jun 2025).

These results show that vocabulary gaps in model transfer are not merely about missing embeddings. They involve sequence divergence, incompatible loss spaces, and altered compression rates. The recurring strategy is alignment: by constructing mappings between vocabularies, one can preserve pretrained manifolds while still changing the discrete lexicon.

5. Cross-lingual lexical gaps and multilingual lexical diversity

In lexical typology, a lexical gap is a concept–language pair for which no single-word lexicalization exists. One formalization defines, for domain concepts $V$4 and languages $V$5, the set $V$6, where $V$7 is the set of lexical items in language $V$8 whose primary meaning corresponds to concept $V$9. “Using Linguistic Typology to Enrich Multilingual Lexicons: the Case of Lexical Gaps in Kinship” combines seed lexicalizations, candidate-gap identification, cross-lingual confirmation, and implicational-universal closure to build a kinship resource with 198 concepts, 699 languages, 1,911 word forms, and 37,370 confirmed gaps. In downstream machine translation evaluation on 1,000 test sentences containing kinship terms, systems achieved kinship-term $\Delta t_{\text{reward}}$0 where a target-language lexicalization existed, but only $\Delta t_{\text{reward}}$1 when the concept–language pair was a gap; gap-aware post-editing raised the gap-case score from 0.42 to 0.61 (Khishigsuren et al., 2022).

A key point in this literature is that a lexical gap does not entail conceptual absence. “Discovering Lexical Gaps Using Embeddings from Multilingual LLMs” explicitly defines lexical gaps, or untranslatabilities, as meaning–form pairings in language A for which language B has no atomic word, even though speakers of B can often paraphrase the concept via multiword expressions or periphrasis. The method extracts contextualized embeddings from Korean–English bilingual LLMs, constructs 4,000 embedding spaces across model, layer, dimensionality, and alignment choices, and measures nearest-neighbor similarity or CSLS between source and target vocabularies. In 94% of Korean-to-English spaces and 97% of English-to-Korean spaces, gap words were less aligned than non-gap words; logistic classifiers on unaligned embedding spaces achieved AUC 0.81 and 0.76 and retrieved 18/19 Korean and 26/27 English gap words (Jung et al., 23 May 2026).

Crowdsourcing has been proposed as another mechanism for making lexical gaps explicit. LingoGap models three relations—equivalent term, language-specific term, and lexical gap—and operationalizes them through a three-step microtask: ask whether the target language has an equivalent meaning, select equivalent terms if present, and otherwise enter missing terms and glosses. Quality control uses Krippendorff’s $\Delta t_{\text{reward}}$2 with a threshold $\Delta t_{\text{reward}}$3, attention checks, qualification tests, and expert validation. Across English–Arabic and Indonesian–Banjarese food terminology, the method identified 2,140 gaps in English↔Arabic and 951 in Indonesian↔Banjarese, with average $\Delta t_{\text{reward}}$4 around 0.83–0.85 in the reported experiments (Khalilia et al., 2024).

Multilingual LLM adaptation exposes a related but distinct issue: vocabulary sharing. A large-scale study over 101 languages found that the extent of overlap with English subwords strongly structures transfer behavior, yielding four quadrants—Reciprocal, Selfish, Altruistic, and Idle—based on whether bilingual and multilingual performance both improve, only one improves, or neither improves. For severely over-tokenized “Idle” languages such as Khmer, Lao, Kannada, and Telugu, neither vocabulary expansion nor LoRA produced more than 1 spBLEU gain, whereas a post-tokenization shared-prefix stripping trick raised average spBLEU by +2.5 on 80 k-sample fine-tuning (Yuan et al., 2023). This suggests that cross-lingual vocabulary gaps may emerge either as true lexical non-equivalence or as tokenizer-induced over-segmentation.

6. Retrieval, document expansion, and sparse lexical matching

In information retrieval, the vocabulary gap refers to the fact that relevant documents and user queries often use different words to express the same concepts. This undermines term-based ranking because exact lexical overlap is treated as a proxy for relevance. One response is semantic matching in latent vector spaces. “Remedies against the Vocabulary Gap in Information Retrieval” combines query formulation from complex text structures with unsupervised latent representations such as a log-linear expert-finding model, Latent Semantic Entities for product search, and the Neural Vector Space Model for document retrieval. Across several benchmarks, NVSM outperformed latent baselines by +10–70%, and fusion with query-likelihood models yielded consistent improvements of $\Delta t_{\text{reward}}$5–40% (Gysel, 2017).

In e-commerce search, the gap often takes the more specific form of missing product tokens. Doc2Token defines, for product $\Delta t_{\text{reward}}$6, indexed metadata $\Delta t_{\text{reward}}$7, its token set $\Delta t_{\text{reward}}$8, and novel target tokens $\Delta t_{\text{reward}}$9, where $A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$0 are satisfactory user queries. A T5-base encoder–decoder is trained to generate single novel tokens with a frequency-weighted loss $A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$1, $A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$2, and beam search of size 10 discards predictions already present in $A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$3. The paper introduces novel ROUGE and reports that Doc2Token improves $A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$4-nROUGE from 0.481 for Doc2Query to 0.500, while producing 100% novel tokens versus 21% for Doc2Query. Training on 3.1 M examples took about 108 minutes per epoch on 8×A100 GPUs; inference on 100K products took 76 minutes on one K80 GPU. In online deployment at Walmart.com, human-annotated NDCG@10 increased by +0.49% ($A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$5) and revenue per session by +0.28% ($A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$6) (Li et al., 2024).

Sparse retrieval exposes yet another version of the problem. “Why Advanced Encoders Lag on Sparse Retrieval?” attributes the underperformance of encoders such as ModernBERT in learned sparse retrieval to a vocabulary gap created by raw, case-sensitive vocabularies designed for lossless reconstruction. Redundant surface forms split a single semantic unit into multiple entries, wasting capacity and harming exact lexical overlap. The proposed Vocabulary Transfer framework migrates encoders to a normalized target vocabulary using Semantic Initialization and Activation Potential Calibration. On BEIR, ModernBERT-VT reached 52.4 nDCG@10, improving over naïve ModernBERT by +4.7; the same procedure also raised RoBERTa-large from 1.4 to 51.3 and BERT-cased from 10.6 to 50.2 in the reported sparse-retrieval setting (Geng et al., 20 Apr 2026).

These retrieval-oriented studies converge on a narrow but important point: exact token overlap is often an inadequate proxy for semantic match, but the correction mechanism depends on failure mode. If documents omit user terminology, token prediction and indexing help. If tokenizers fragment semantic units into redundant surface forms, coarse-graining or vocabulary transfer is more appropriate. If lexical overlap is structurally sparse, latent semantic models remain complementary.

7. Benchmarking, diagnostics, and recurring misconceptions

Vocabulary gaps are also measurable as benchmark deficits. One proposal defines a vocabulary test $A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$7 with item set $A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$8 and model-specific coverage $A=\frac{1}{N}\sum_{i=1}^N I(\text{correct}_i)$9, with vocabulary gap $L=\frac{1}{N}\sum_{i=1}^N t_{\text{resp},i}$0. Using multiple-choice synonym tests and Yes/No lexical-recognition tests in English and Spanish, seven LLMs were evaluated under deterministic prompting. The results show that strong English performance does not guarantee robust lexical knowledge across languages or item types. On multiple-choice tests, GPT-4 achieved 100.0 on StuVoc-Eng and 95.0 on StuVoc-Esp, while Mistral-7B achieved 99.3 on StuVoc-Eng but 68.8 on StuVoc-Esp. On Spanish Yes/No recognition, pseudoword rejection was especially weak: Mistral-7B achieved 0.0 correct rejection on the non-word subset, Llama-7B 7.1, and GPT-4 82.1, meaning GPT-4 still hallucinated 18% of Spanish non-words (Martínez et al., 2023).

These findings complicate a common assumption that larger or more capable general-purpose LLMs have largely solved lexical knowledge. The benchmark literature instead indicates non-negligible lexical gaps, especially in lower-resource languages and non-word rejection (Martínez et al., 2023). A plausible implication is that vocabulary-gap diagnostics are most informative when they separate lexical coverage, pseudoword rejection, translation equivalence, tokenization efficiency, and retrieval matching rather than conflating them.

Across domains, the literature suggests four recurring mechanisms behind the label “vocabulary gap”: missing lexicalization, static or suboptimal tokenization, mismatch between vocabularies used by different systems or modalities, and failure of exact lexical overlap in search or sparse matching. The remedies therefore range from paraphrase and gap annotation, to entropy-guided vocabulary curricula, token-alignment and embedding-transfer methods, document expansion, normalized sparse vocabularies, and socially grounded learning environments. The term’s breadth is a source of ambiguity, but also a useful reminder that lexical representation is often the bottleneck between data, models, and communication.