Mixed-Vocabulary Training

• Mixed-vocabulary training is a technique that integrates diverse token sequences from different vocabularies to enable effective cross-modal and multilingual adaptation.
• It employs curriculum schedules and mapping methods, such as scheduled interleaving and embedding space alignment, to resolve vocabulary mismatches and out-of-vocabulary challenges.
• Empirical results reveal significant gains in low-resource and multimodal tasks, demonstrating its potential for scalable model compression and improved transfer learning.

Mixed-vocabulary training encompasses a family of techniques in which neural sequence models are exposed to input/output token sequences drawn from heterogeneous vocabularies—potentially differing in granularity, modality, or origin. Unlike conventional pretraining with a static vocabulary, mixed-vocabulary schemes are instrumental for distillation, cross-lingual adaptation, multimodal modeling, curriculum learning, and vocabulary expansion. Such approaches critically address empirical and computational challenges arising from vocabulary mismatch, out-of-vocabulary (OOV) phenomena, representation transfer, and data scarcity.

1. Definitions and Taxonomic Scope

Mixed-vocabulary training is characterized by the use of token sequences from more than one vocabulary source, which may vary in:

• Granularity: Coexisting vocabularies may encode tokens as characters, subwords (e.g., BPE, WordPiece), or higher-order linguistic/semantic units (acoustic frames, speech units).
• Modality: Tokens may arise from distinct modalities (text, speech, vision, etc.), appearing together or on a curriculum.
• Origin: Some schemes mix vocabularies to bridge teacher–student architectures, expand language coverage, or accommodate multitask objectives.

A canonical setting involves LLM pretraining or fine-tuning with sequences in which tokens correspond to both source and target vocabularies. Examples include scheduled interleaved speech-text tokens for S2ST (Futami et al., 12 Jun 2025), wordpiece mixtures for BERT compression (Zhao et al., 2019), cross-lingual subword expansions (Zheng et al., 2021, Wang et al., 2019), multi-task union of closed/open lexica (Sun et al., 15 Sep 2024), dynamic curricula (Yu, 25 Feb 2025), and explicit teacher–student mapping under mismatched vocabularies (Shin et al., 24 Mar 2025).

2. Curriculum and Scheduling Strategies

A powerful paradigm within mixed-vocabulary training leverages curriculum schedules that gradually vary the proportion of tokens from each vocabulary source. This facilitates progressive adaptation—either across modalities or granularities, or from teacher to student.

Scheduled Interleaved Training: In S2ST, a pretrained LLM is exposed to sequences where a fraction $p(k)$ of words are represented by text tokens (BPE) and the remainder by discrete speech units; $p(k)$ decays linearly at fixed intervals until only speech units remain as inputs. Mathematically, the schedule is piecewise-constant:

$$p(k) = \max\left\{0, p_0 - \Delta p \left\lfloor \frac{k}{\Delta k} \right\rfloor \right\}$$

Curriculum schedules have been shown to outperform unscheduled and static mixing approaches, with largest gains observed for low-resource language pairs and under severe data constraints (Futami et al., 12 Jun 2025).

Vocabulary Curriculum Learning: Dynamic expansion of the vocabulary based on entropy-guided selection induces a mixed-vocabulary regime across training stages. Here, longer, predictable sequences are promoted to new token types as the model improves, while shorter, high-entropy tokens remain for difficult, diverse contexts (Yu, 25 Feb 2025). Direct empirical evidence shows log-linear scaling gains in bits-per-character that surpass any fixed-vocabulary baseline.

3. Cross-vocabulary Alignment and Mapping Algorithms

Handling the representational mismatch between heterogeneous vocabularies is central to mixed-vocabulary training, particularly in teacher–student and multilingual setups. Key methodologies include:

Lexical Alignment (VocAgnoLM): For teacher–student distillation under vocabulary mismatch, each student token is mapped to the minimal contiguous span of teacher tokens covering the same character offsets. This token-level mapping enables loss reweighting and aggregation, bypassing the breakdowns of KL divergence when vocabularies do not overlap. Empirically, token-level alignment achieves far higher transfer than coarse sequence chunking (Shin et al., 24 Mar 2025).

Embedding Space Mapping: In multilingual BERT vocabulary expansion, new subwords are mapped into the model's embedding space by either joint Procrustes alignment (stepwise, language → English → BERT) or mixture mapping (convex combination of nearest English subword embeddings w.r.t. CSLS distance). Mixture mapping is robust to low-resource and typologically distant languages, with consistently higher gains on token-level tasks (Wang et al., 2019).

Mixed-Tokenizer Sampling: In BERT distillation, tokenization at each word alternates randomly between teacher and student models, yielding mixed-vocab inputs. Alignment between embedding spaces is ensured only implicitly by masked language modeling (MLM) loss, with no explicit similarity constraints (Zhao et al., 2019).

4. Objective Functions and Loss Formulations

Mixed-vocabulary regimes demand careful consideration of loss design:

• Cross-entropy Loss over Mixed Vocabularies: Training is conducted via standard cross-entropy over the union of active vocabularies at each step, with masking strategies ensuring balanced exposure to each origin (Zhao et al., 2019, Futami et al., 12 Jun 2025).
• Weighted or Aggregated Teacher-guided Losses: In vocabulary-agnostic teacher distillation, the student’s negative log-probabilities are selectively reweighted using teacher-derived loss aggregates, enabling supervision even when token/embedding spaces strongly diverge (Shin et al., 24 Mar 2025).
• Multi-task Objectives: In multi-task learning (MTL) with fixed and open lexica, losses are summed over primary Seq2Seq pronunciation prediction and auxiliary acoustic-feature regression, with $\lambda$ weighting selected for best generalization to extra-exclusive words (Sun et al., 15 Sep 2024).
• Entropy-guided Token Addition: In vocabulary curriculum models, vocabulary-expansion stages are informed by per-token conditional entropy, ensuring merges are only made for highly predictable (monotonic, low-entropy) sequences (Yu, 25 Feb 2025).

5. Empirical Benchmarks and Impact

Mixed-vocabulary training consistently yields gains over single-vocabulary baselines on classification, translation, tagging, and LM tasks, particularly for low-resource languages, OOV tokens, and compressed models. Selected results:

Setting	Baseline	Mixed-Vocab Approach	Gain (Metric)
S2ST (It-En)	12.8/4.23	19.5/4.23 (ILT sched)	+6.7 BLEU, no UTMOS loss (Futami et al., 12 Jun 2025)
BERT GLUE (6-L)	81.7%	84.3% (mixed-vocab KD)	+2.6% average (Zhao et al., 2019)
XTREME Avg	60.7	63.7 (VoCap+kNN)	+3.0 points (Zheng et al., 2021)
MRC (zh–en)	38.1%	39.3% (mixture map)	+1.2% accuracy (Wang et al., 2019)
TTS PER (excl.)	2.5%	1.6% (MTL-mixed-vocab)	–36% error, +4 pts Acc. (Sun et al., 15 Sep 2024)
TinyLlama math	21.5	29.3 (VocAgnoLM)	+46% task accuracy (Shin et al., 24 Mar 2025)
LLM BPC (enwiki8, V=7931)	1.5103	1.4385 (cur.)	–4.75% vs. static (Yu, 25 Feb 2025)

Low-resource languages and OOV tokens benefit most, as mixture approaches (curricula, mapping, or distillation) alleviate data scarcity and representational coverage gaps.

6. Practical Applications and Generalization

Mixed-vocabulary methodologies underlie numerous practical and research advances:

• End-to-end Speech Translation: Curriculum mixing of speech and text tokens streamlines adaptation of text-trained LLMs to discrete speech units, removing the need for speech-only adapters and enabling translation in low-data domains (Futami et al., 12 Jun 2025).
• Efficient Model Compression: Mixed-vocab distillation provably aligns model representations when compressing large LMs to small-vocabulary students for on-device use (Zhao et al., 2019).
• Multilingual Expansion without Re-Pretraining: Subword mappings enrich the coverage of pre-trained models on new languages or dialects, with minimal architecture changes (Wang et al., 2019).
• Dynamic, Adaptive Language Modeling: Entropy-guided vocabulary curriculum enables simultaneous learning and expansion, reflecting the adaptive nature of human language acquisition (Yu, 25 Feb 2025).
• Pronunciation Modeling: Multi-task mixed-vocabulary approaches efficiently bootstrap knowledge for excluded words using transcribed speech (Sun et al., 15 Sep 2024).
• Teacher–Student Distillation with Arbitrary Vocabularies: Vocabulary-agnostic alignment removes the bottleneck of vocabulary overlap, allowing high-performance transfer regardless of tokenizer divergence (Shin et al., 24 Mar 2025).

Generalization principles motivate extension to vision-text (patch-token curricula), multi-dialect MT, dialog agents, and additional modalities. Mixed-vocabulary schedules provide a pathway for fine-grained, curriculum-driven representation expansion across domains and modalities without sacrificing embedding compatibility.

7. Limitations and Future Directions

Key challenges in mixed-vocabulary training include:

• Alignment Complexity: Word-level or token-level alignment is computationally intensive (e.g., O(N log M) per batch for character span mapping) (Shin et al., 24 Mar 2025).
• Embedding Overhead: Merging or expanding vocabularies increases parameter count; memory budget must be balanced against potential accuracy gains (Zheng et al., 2021, Zhao et al., 2019).
• Domain Transfer: Schemes trained on synthetic or narrow-domain data (e.g., S2ST vocoder and semantic units) may require additional adaptation for real-world deployment (Futami et al., 12 Jun 2025).
• Hyperparameter Sensitivity: Curriculum schedules, λ balances in MTL, top-k thresholds for reweighting, and aggregation choices can substantially affect empirical outcomes.
• Limited Theoretical Analysis: While observed log-linear scaling laws and information-theoretic rationale exist (Yu, 25 Feb 2025), broader theoretical guarantees regarding optimal mixing patterns remain undeveloped.

Future work points toward applying mixed-vocabulary principles to continual, task-adaptive distillation, multimodal generalization, scalable curriculum schedules, and more sophisticated mapping algorithms that handle non-contiguous spans, insertions, or deletions. Extension to batch-efficient or compressed inference remains an open research frontier.

---

Mixed-vocabulary training constitutes a versatile and empirically validated class of algorithms for neural sequence modeling. By explicitly mixing token sources—via curriculum schedules, mapping, multitask, or alignment strategies—these techniques afford substantial improvements in transfer, efficiency, and flexibility across a wide range of NLP, speech, and multimodal tasks.