---
title: Online Vocabulary Update Algorithm
url: https://www.emergentmind.com/topics/online-vocabulary-update-algorithm
type: topic
---

# Online Vocabulary Update Algorithm

An online vocabulary update algorithm incrementally refines, expands, or adapts the set of word, subword, or categorical feature representations used by a model as new data arrives in a stream or dynamic environment. Unlike batch-based learning, which processes large static corpora to produce fixed vocabularies, online methods address nonstationarity, domain drift, resource constraints, and interactive use cases where new words or categories appear continuously or input distributions shift. This paradigm spans applications from domain adaptation in NLP, input methods, federated discovery, categorical feature encoding, and dynamic tokenizer extension for LLMs, and encompasses a diversity of mathematical and algorithmic frameworks.

## 1. Mathematical Frameworks and Update Objectives

Online vocabulary update algorithms formalize vocabulary refinement as a streaming or sequential process under various learning or optimization objectives. Broad settings include:

- **Incremental unsupervised domain adaptation**: Updating word-context co-occurrence statistics without retraining the supervised classifier, so representations for words gradually drift from source toward target distribution as new sentences arrive [1604.00502].
- **Dynamic word likelihood estimation**: Maintaining a continuous word-likelihood score (IME Word Likelihood, IWL) for each character sequence, which directly informs language model probabilities used in input decoding [1712.04158].
- **Probabilistic embedding inference**: Employing Bayesian online inference to update hash-based categorical feature embeddings in bounded memory while tracking parameter uncertainty [2511.20893].
- **Tokenization expansion through statistics-driven selection**: Dynamically expanding the subword vocabulary of a language model using fragment scores and BPE statistics to minimize over-fragmentation on streaming domain data [2110.13434].
- **Privacy-preserving OOV discovery**: Secure aggregation and LDP-based randomization in federated settings for online mining of new words without leaking identifiable user input [2404.11607].
- **Transformation-based composition**: Reshaping vocabulary to maximize coverage via composition rules applied to base forms and transformation offsets, freeing slots for OOVs and reducing redundancy [2510.17001].

Many algorithms operate at the level of word, subword, or category-specific tables, e.g., $freq_L[w][i]$, $IWL(w)$, posterior parameters $\lambda_{b,j}$, or embedding matrices, and update these in response to immediate or recent data, rather than making global updates.

## 2. Algorithmic Strategies and Pseudocode

Representative online vocabulary update routines instantiate the following canonical steps:

| Application               | Step 1: Data Handling          | Step 2: Update Rule                | Step 3: Culling or Selection       |
|---------------------------|-------------------------------|------------------------------------|------------------------------------|
| DA for POS tagging [1604.00502] | Sentence stream; left/right context | Increment context counts; update features | N/A, all words accumulate stats     |
| IME wordhood [1712.04158]        | User confirmed input sequence      | Add substrings; segment; IWL increments | Periodic pruning by IWL/capacity    |
| Hash embedding [2511.20893]      | Streaming categorical features     | Bayesian VI update for hashes/Buckets | Fixed parameter table; no culling   |
| Domain tokenizer [2509.26124]    | Batch or incremental domain corpus | Merge appending to BPE; embedding averaging | Budget-limited selection           |
| Federated OOV [2404.11607]       | User device OOV prefixes           | LDP randomization; secure aggregation | Top-k by noisy count per prefix/round |
| Vocab reshaping [2510.17001]     | Ongoing corpus statistics          | Surface form removal; base+transform insertion | Lowest-frequency forms removed      |

This diversity illustrates the alignment between the nature of the data stream, type of vocabulary manifold (lexicon, subword, categorical), and resource or privacy constraints, but all share the fundamental property of incremental, stateful evolution with update rules local to each step.

## 3. Theoretical Properties: Memory, Convergence, and Robustness

- **Memory efficiency** is a recurring constraint: probabilistic hash embedding maintains a bounded $B \times d \times 2$ parameter table irrespective of vocabulary size [2511.20893], IME vocabularies are capped and pruned [1712.04158], while federated OOV discovery enforces client and per-layer capacity strictness [2404.11607]. Categorical streaming settings preclude per-item tables, favoring compact shared representations.
- **Convergence**: Domain adaptation by online counting converges to batch statistics over the full test corpus; empirically, accuracy matches batch DA within a few dozen occurrences per word [1604.00502]. Online IME adaptation stabilizes top-1 accuracy within a few thousand MIUs and shows diminishing fluctuations [1712.04158]. Bayesian hash embedding is formally invariant to item arrival order in exact updates [2511.20893], and periodic vocabulary expansion maintains stable fragment scores [2110.13434].
- **Robustness to order and drift**: Probabilistic methods guarantee permutation invariance of the posterior, mitigating catastrophic forgetting and arrival-order sensitivity [2511.20893]. Online segmentation-based wordhood treats unknown words no differently from known, promoting resilient adaptation to shifting user input [1712.04158].

## 4. Application Domains and Empirical Performance

- **Domain-adaptive NLP**: Online DA for POS tagging delivers accuracy within $0.03\%$ of batch DA, with up to $6\%$ better OOV tagging than static representations [1604.00502].
- **IME input likelihood**: Online IWL adaptation for Chinese IME achieves top-1 scores of $55.3\%$ on People’s Daily and $51.4\%$ on Touchpal domain, far surpassing static trigram [1712.04158].
- **LLM tokenizer customization**: Incrementally extending tokenizers reduces domain fertility by up to $20\%$ (sequence length), with a net $8$–$30\%$ increase in inference throughput and negligible impact on NLU/accuracy [2509.26124].
- **Vocabulary reshaping**: Removing up to $10\%$ of surface forms and recomposing OOVs expands coverage to tens of thousands of word forms, with performance differential within $1$–$3\%$ on benchmarks [2510.17001].
- **Federated OOV mining**: Gboard’s privacy-preserving pipeline provides $92.1\%$ coverage of rare OOVs after two passes, under $(\epsilon', \delta) = (0.315, 10^{-10})$ central DP [2404.11607].

These results consistently demonstrate that online vocabulary updates, via a range of mechanisms, match or outperform batch and static approaches in coverage and adaptation, especially for uncommon or new terms.

## 5. Practical Hyperparameters and Engineering Considerations

- **Update batch size and frequency**: IME IWL updates use $\alpha=1.0$, $\beta=5.0$, $\gamma=1.0$; periodic pruning every $per$ steps up to cap [1712.04158]. Online BPE expansion via AVocaDo controls initial and incremental merge count $\alpha$, $\beta$, and fragment score threshold $\gamma$ [2110.13434]. LLM domain vocab extension sets merge budget $N$, typically $K$ to several tens of thousands [2509.26124].
- **Initialization**: New token embedding rows are initialized by weighted averages of splits (e.g., $e_t = \frac{1}{2}(e_{t_1} + e_{t_2})$) [2509.26124], or averages over constituent subwords [2110.13434]; transformation offsets for compositional vocab are induced from the difference between surface and base embeddings [2510.17001].
- **Pruning/culling**: IME and federated methods prune by minimal IWL or top noisy count; compositional vocab reshaping drops lowest-utility (lowest frequency) forms [2510.17001].
- **Privacy/DP mechanics**: Federated OOV applies $\epsilon$-LDP per user-layer, secure aggregation on server, and central $(\epsilon', \delta)$ DP on final release [2404.11607]. Communication and computation strictly bounded.
- **Resource adaptation**: PHE recommends $B \approx 5–10 \times$ expected vocab size and $K = 2–4$ hash functions [2511.20893].

## 6. Limitations, Challenges, and Future Directions

- **Retraining burden**: Most algorithms avoid classifier retraining, updating only representation or likelihood tables [1604.00502, 1712.04158, 2110.13434], making them suitable for real-time or low-latency scenarios.
- **Backward compatibility**: Frequent vocabulary changes risk disrupting downstream services (e.g., position embeddings, old-to-new token mapping); stability of the mapping and model infrastructure is necessary [2110.13434, 2509.26124].
- **Culling trade-offs**: Aggressive capacity-based pruning may remove rare but valuable terms; criteria are often frequency-based to prioritize utility.
- **Complexity management**: Model architectures must accommodate embedding-table growth or compositional inference logic, ideally minimizing code changes and runtime impact [2510.17001].
- **Privacy constraints**: In consumer-facing deployments, privacy guarantees via LDP and DP aggregation raise communication and protocol complexity [2404.11607].

A plausible implication is that ongoing work will further integrate online vocabulary updates with robust continual learning, richer compositionality, and strict privacy constraints, enabling adaptive, scalable, and fair NLP and ML systems in live environments.

## 7. Comparative Summary Table

| Paper/Algorithm        | Vocabulary Type      | Update Mechanism                 | Memory Bound | Empirical Gains                          |
|-----------------------|---------------------|----------------------------------|-------------|------------------------------------------|
| FLORS Online DA [1604.00502]    | Lexical/word          | Context bigram count increment     | $O(|V|)$    | $<0.03\%$ from batch; $+6\%$ OOV accuracy |
| OMWA IME [1712.04158]           | Word (Chinese IME)    | Additive IWL/segmentation          | Capacity-capped | $+30$ pts top-1 over static trigram       |
| PHE hash [2511.20893]           | Categorical features  | Bayesian VI, per-hash updates      | $O(Bd)$     | Outperforms deterministic, permutation-invariant |
| AVocaDo [2110.13434]            | Subword (BPE)         | Fragment score-driven BPE merging  | Variable    | $+1\sim13$ F1 pts vs. base BERT/SciBERT   |
| Domain Tokenizer [2509.26124]   | Subword (LLM)         | Frequency-sorted merge appending   | Budgeted    | $20\%$ shorter seq, $30\%$ speedup        |
| Federated OOV [2404.11607]      | Lexical/word (OOV)    | LDP randomized, secure aggregation | Trie-layer cap| $92.1\%$ coverage, $(0.315,10^{-10})$ DP |
| Vocab Diet [2510.17001]         | Surface/composit.     | Morphological decompositon, offset | $-10\%$ slots| $70$–$95\%$ new forms composable, $<3\%$ perf drop |

This cross-section delineates both the technical strategies and measurable advances, confirming the centrality of online vocabulary update algorithms in modern adaptive NLP and ML.

Source: https://www.emergentmind.com/topics/online-vocabulary-update-algorithm