VocabTailor: Custom Vocabulary Techniques

• VocabTailor is a framework that dynamically tailors vocabularies at token, word, and class levels to optimize memory usage and task-specific performance.
• It integrates techniques such as dynamic vocabulary selection, cross-lingual mnemonic generation, curriculum personalization, and token alignment to boost efficiency in NLP and vision applications.
• Empirical findings demonstrate significant memory reductions (up to 99%), improved prediction accuracy, and enhanced personalization in educational and deployment contexts.

VocabTailor is a family of techniques and systems for tailoring vocabularies—at the level of tokens, words, object classes, or learning targets—to downstream models, tasks, users, and educational objectives. Methods grouped under the VocabTailor paradigm address the challenges of memory efficiency in LLMs, personalized vocabulary instruction, dynamic and context-sensitive adaptation in object detection, and robust open-vocabulary token alignment and transfer. This article presents core concepts, theoretical formulations, representative architectures, empirical findings, and the principal limitations and open research problems identified in the literature.

1. Dynamic and Memory-Efficient Vocabulary Management in LLMs

VocabTailor, as formalized in "VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small LLMs" (Zhang et al., 21 Aug 2025), addresses the prohibitive memory footprint of vocabulary components in SLMs. Vocabularies of 128 K tokens can require ~2 GB for embedding and LM head matrices. Standard static vocabulary pruning causes segmentation shifts and information loss.

The VocabTailor approach exploits lexical locality—the observation that only a small token subset is needed for any single inference—and computational asymmetry between the embedding (lookup, bandwidth-bound) and LM head (large matrix multiply, compute-bound). The architecture decouples the tokenizer, embedding, and LM head components:

• Embeddings: The full embedding matrix remains intact and is offloaded to CPU RAM, avoiding tokenization drift.
• LM Head: A hybrid static–dynamic vocabulary selection scheme is implemented.
  • Offline: Build a static task vocabulary $\mathcal{T}$ using profiling, filtering on input-appearing tokens, Unicode block restrictions, and tolerance-based document-frequency cutoffs.
  • Inference: For input $i$ with unique tokens $\mathcal{I}_i$, form an active vocabulary $\mathcal{V}_i = \mathcal{I}_i \cup \mathcal{T}$ and fetch only corresponding LM head rows from CPU to GPU.

Reductions of up to 99% are achieved in vocabulary memory usage, with minimal or no performance degradation across machine translation, summarization, code completion, and extraction tasks. VocabTailor consistently outperforms static pruning by avoiding segmentation shifts and preserving input/task locality (Zhang et al., 21 Aug 2025).

2. Cross-Lingual Mnemonic and Keyword Adaptation

PhoniTale, recast for VocabTailor (Kang et al., 7 Jul 2025), delivers a two-phase pipeline for generating phonologically grounded mnemonics for typologically distant language pairs, particularly for L2→L1 vocabulary learning (e.g., English→Korean):

• Phase 1: Keyword Sequence Retrieval
  • L2 input word ($w_{\rm L2}$) is transliterated to IPA and processed by a Bi-LSTM encoder–decoder, with total loss $$\mathcal{L}_{\rm total} = \mathcal{L}_{\rm CE} + \lambda\,\mathcal{L}_{\rm cont}$$.
  • Syllable boundaries are detected; segments are matched to L1 keywords via a phonological similarity $\phi(S_i,w)$ that combines a PanPhon-based cosine metric and structural adjustments for syllable overlap and phone alignment.
• Phase 2: LLM-Based Mnemonic Generation
  • L1 keyword sequence ($\mathcal{W}_{\rm L1}$) and target meaning are input to an LLM, with prompt engineering ensuring required keyword inclusion. Multiple candidates are ranked by mask-and-predict context completeness.

Empirical results demonstrate phonetic similarity 0.95 and zero keyword omission for PHT (PhoniTale), with short-term recall matching or exceeding human-crafted mnemonics, though user preference still favors handcrafted cues (Kang et al., 7 Jul 2025). Current limitations include monolingual focus and the need for broader L1 keyword sets.

3. Curriculum Adaptation and Personalization in Vocabulary Learning

VocabTailor’s pedagogical adaptation includes RL-based curriculum sequencing (Zaidi et al., 2017) and graph-based learner models (Kokku et al., 2018, Ponnusamy et al., 2020):

• Curriculum Q-Learning: A tutor agent models vocabulary acquisition as a two-tier Markov Decision Process (MDP), managing CEFR-level state and per-word activity. The agent’s reward function penalizes revisit of mastered items and encourages focus on items at the learner’s edge of the ZPD. The Q-values for state–action pairs are updated via standard Q-learning, and action selection is ε-greedy.
  • Virtual-student simulations reveal stable personalized curriculum trajectory convergence in <80 steps, with learning gains predicted by Δ-probability-of-success (based on a negated Gompertz item response model) (Zaidi et al., 2017).
• Graph-Based Knowledge and Phased Learner Models: The knowledge base is a directed graph $G = (V, E, R)$, with nodes as words/concepts and edges denoting relations (synonym, part-of, etc.), optionally weighted by confidence. Each learner tracks per-word proficiency $l(v)$, updated by exponentially weighted moving averages; staged transitions (Parked, Learning, Assessment-Only, Learned) depend on fixed thresholds. Adaptive content selection blends sampling from learning and assessment pools (Kokku et al., 2018).
• Network-Driven Centrality Selection: Morphologically clustered “word families” form nodes in an undirected weighted graph, where family-level centrality determines selection order for practice. Post-response updates propagate experience across neighbors; multi-gap exercises select maximally informative sentences and distractors using combinatorial and graph-theoretic constraints (Ponnusamy et al., 2020).

4. Token Alignment, Vocabulary Unification, and Knowledge Transfer

The TokAlign VocabTailor module (Li et al., 4 Jun 2025) enables efficient vocabulary replacement and transfer in LLMs:

• One-to-one token mapping is established via row-wise cosine alignment between GloVe embeddings for source ($i$0) and target ($i$1) vocabularies, inducing mapping matrix $i$2.
• Embedding and LM head rearrangement: For each target token $i$3, embeddings and head weights are initialized by weighted sums over mapped source tokens.
• Two-stage fine-tuning: Freeze everything but new embed/LM head for early steps, unfreeze all for the rest.
• Yields a reduction in perplexity at initialization (to ≈120 from 340 for strong baselines), >29% compression in tokenized length across 13 languages, and enables logit-level distillation by unifying token IDs between teacher and student models. Downstream, as few as 5k steps suffice for complete recovery of task performance (Li et al., 4 Jun 2025).

5. Personalization, OOV Expansion, and Contextual Adaptation

VocabTailor encompasses models for reader- and instance-specific vocabulary adaptation:

• Personalized Complexity Models: Individualized CWI models fit logistic regression classifiers using 23 psycholinguistic and lexical features, updated via entropy-based active learning loops with cluster-based label propagation. Only ~23 queries per user are required for F₁ ≈ 0.75, outperforming group and frequency thresholds (Gooding et al., 2022). This confirms the critical role of reader-dependence in lexical complexity.
• On-Device OOV Expansion: Closed-vocabulary federated learning models are augmented with per-user adapters that learn embeddings for the user’s most frequent OOV words, placed after the CharCNN encoder. Only the adapter MLP is updated locally, keeping global parameters fixed. The system achieves >97% reduction in unknown-word rates and 2–6% absolute increases in EMR₃ (top-3 prediction accuracy), with a minimal parameter/memory footprint (Wang et al., 2023).
• Visual Domain Test-Time Vocabulary Adaptation: For open-vocabulary object detection, plug-and-play adapters (VocAda) refine the class list for each image using captioning (LLaVA-Next-7B), noun-phrase extraction (spaCy), and selection via CLIP or LLMs. This consistently yields +2–3 mAP points over baselines, with the trade-off of increased inference time (5–10 s/image for LLM-based selection) (Liu et al., 31 May 2025).

6. Generative and Contextual Exercise Generation

Systems such as Storyfier (VocabTailor instantiation) leverage controlled seq2seq models (T5-Large) for context-rich, learner-specific content:

• Input is an unordered set of target words; output is a beam-decoded story penalized for omission or excessive repetition of required words.
• Exercise pipelines combine reading, cloze-testing, and turn-based, AI-assisted writing, with adaptive batch size based on retention ($i$4), coherence-enforcing re-ranking, and scaffolded output.
• Empirical results indicate that, while generated stories enhance engagement and enjoyment, they do not guarantee increased word retention, especially without semantic coherence among targets. Overreliance on AI reduces productive struggle and learning gains (Peng et al., 2023).

7. Open Challenges and Future Directions

Current limitations across the VocabTailor spectrum include domain specificity (e.g., English–Korean focus in phonological mnemonics (Kang et al., 7 Jul 2025)), context-agnostic complexity labeling (Gooding et al., 2022), memory and transfer constraints in alignment and offloading (Li et al., 4 Jun 2025, Zhang et al., 21 Aug 2025), and the question of balancing automation with productive learner effort (Peng et al., 2023).

Promising research directions involve adaptive re-profiling for domain shift, integration with quantization/pruning, structured extension to multimodal and retrieval-augmented systems, longitudinal evaluation for retention, and hybrid workflows blending human/LLM curation or personalization modules. Privacy-preserving OOV expansion, robust user modeling, and evaluation at scale remain active challenges in educational and deployed machine learning contexts.