Efficient Subword Bigram Model

• The subword bigram model is a statistical and neural approach that represents words via adjacent length-2 substrings to capture local sequential dependencies.
• It employs training objectives like regression to pre-trained embeddings and skip-gram with negative sampling to achieve fast inference and significant memory efficiency.
• Empirical evaluations demonstrate robust syntactic performance and efficient segmentation, making it a lightweight, scalable alternative for large-scale NLP applications.

An efficient subword bigram model is a statistical or neural approach to language modeling and representation that leverages bigram (length-2) substrings of words or subword units as the atomic elements for probability estimation, segmentation, or embedding functions. Such models aim to capture local sequential dependencies or morphological structure while maintaining computational tractability for large-scale language processing or model training. They have been utilized for word vector generalization, lexically grounded tokenization, and as minimal sufficient subnetworks in neural LLMs.

1. Mathematical Foundations of Subword Bigram Models

Character- and subword-based bigram models instantiate the core idea of representing a string $w$ through its sequence of adjacent, length-2 components. Let $\Sigma$ denote the character or subword alphabet, and let $w$ be augmented by special boundary symbols, e.g., $<$, $>$, so $⟨w⟩ = \langle$ $w$ $\rangle$.

For a character bigram embedding model, the set of bigrams is: $$G_w = \{ t : t \text{ is a substring of } ⟨w⟩,~|t| = 2 \}$$ Each $g \in G_w$ is associated with a learned vector $z_g \in \mathbb{R}^d$. The word embedding or representation is then: $v_w = \frac{1}{|G_w|} \sum_{g \in G_w} z_g$ For unsupervised segmentation, a subword bigram model uses a fixed subword vocabulary $\mathcal{S}$. Given a segmentation $w \mapsto s_1 s_2 \ldots s_n$ with $s_i \in \mathcal{S}$, the probability of a segmentation is modeled as a bigram LLM: $$P(s_1 s_2 \ldots s_n) = P(s_1) \prod_{i=2}^n P(s_i \mid s_{i-1})$$ Parameter estimation proceeds via count statistics smoothed (e.g., Laplace) from a training corpus, enabling log-probability calculations for arbitrary segmentations (Libovický et al., 2024, Zhao et al., 2018, Bojanowski et al., 2016).

2. Learning and Inference Algorithms

Training Objectives

Two principal learning objectives are prevalent:

• Regression to Pre-trained Embeddings: Minimize mean-squared error between the subword bigram-derived embedding and a fixed "target" word vector:

$$\frac{1}{|W|} \sum_{w \in W} \frac{1}{2} \| v_w - u_w \|^2_2$$

where $u_w$ is a pre-trained word embedding and $W$ is the vocabulary (Zhao et al., 2018).

• Skip-Gram with Negative Sampling: Use the sum or average of bigram vectors as input, and maximize the likelihood of context words using the negative-sampling objective:

$$L = - \sum_{t=1}^T \sum_{c \in C_t} \left[ \log \sigma(x_{w_t}^\top v_{w_c}) + \sum_{n \in N_{t,c}} \log \sigma(-x_{w_t}^\top v_n) \right]$$

with $x_{w_t}$ the summed bigram embedding (Bojanowski et al., 2016).

For probabilistic segmentation, bigram tables $P(s)$ and $P(s'|s)$ are estimated via counts (with smoothing) and inference proceeds via:

• Exact dynamic programming (DP): $O(L\times M\times|\mathcal{S}|)$ time for string length $L$ and maximum subword length $M$
• Beam search (pruned DP): $O(L\times M\times k)$ time for beam width $k$ (Libovický et al., 2024).

Test-time Inference

At inference, embeddings or segmentation are computed as an average or a DP/beam search through the segmentation lattice, with only a hash table or trie of bigrams and the compact probability tables required.

3. Computational Efficiency and Resource Requirements

Bigram models are designed for high efficiency:

• Memory: Embedding-based models only require $O(|\Sigma|^2 \cdot d)$ memory for bigram parameters. With hashing, this can be further bounded, e.g., $K \simeq 2 \times 10^6$ hashed entries for character bigrams (Bojanowski et al., 2016). In segmentation, storing bigram tables for $\mathcal{S} = 32,000$ yields $\sim 50$MB after quantization versus $\sim$1GB for embedding-based models (Libovický et al., 2024).
• Speed: Training and inference involve $O(L \cdot d)$ operations per word for embeddings, or linear time DP/beam search for segmentation. Representative run times are $~0.1$ ms per word for bigram segmentation versus $~5$ ms for embedding-based methods (Libovický et al., 2024); embedding models process $> 100,000$ words/s/thread (Bojanowski et al., 2016).
• Scalability: Bigram subnetworks isolated in large transformer LMs cover next-token prediction circuits with $<1\%$ of non-embedding parameters, leading to $>200\times$ speedups and $>500\times$ reduction in inference memory (Chang et al., 21 Apr 2025).

4. Empirical Performance

Intrinsic and Extrinsic Evaluation

• Word similarity (English, Spearman $\rho$):
  • Rare Words: Word-only skipgram $43.4$, bigram subword $41.0$, full subword ($n=3$–6) $48.0$ (Bojanowski et al., 2016)
  • German similarity (Gur350): word-only $61.0$, bigram $57.0$, full subword $70.0$ (Bojanowski et al., 2016)
• Syntactic analogy (English): bigram $71.0\%$, full subword $74.9\%$; bigrams perform robustly on purely syntactic tasks, but less so on similarity (Bojanowski et al., 2016).
• POS and morphosyntactic tagging (23 languages): Bigram-based segmentation, distilled from embedding-driven segmentation, raises POS accuracy by up to $+0.712$ over baselines (Libovický et al., 2024).
• Morphological segmentation (SIGMORPHON 2018): Morpheme-boundary precision for distilled bigram $86.8\%$, embedding $87.0\%$, unigram $84.3\%$ (Libovický et al., 2024).
• Machine translation (IWSLT’17): Bigram-based segmentation yields average chrF gains of $+0.3$–$+0.4$ normalized points over BPE or unigram baselines (Libovický et al., 2024).

Degradation and Trade-offs

Restricting to bigrams typically yields decreased performance in intrinsic similarity and morphologically-rich language tasks versus using longer $n$-grams, but achieves substantial runtime and memory savings. Bigram subnetworks, even when constituting $<1\%$ of parameters, achieve surprisal correlation $r > 0.95$ with the empirical bigram model (Chang et al., 21 Apr 2025, Zhao et al., 2018).

5. Applications and Model Variants

Lexical Segmentation and Tokenization

Efficient subword bigram models are used for segmentation algorithms that replace expensive embedding-based inference with fast segmentation, suited for large-scale pre-processing:

• After initial distillation from a morphologically-aware segmenter (e.g., using Morfessor and embedding-based segmentation), bigram counts enable low-resource, lexically informed segmentation at inference (Libovický et al., 2024).

Neural Language Modeling

Isolated bigram subnetworks in transformer LMs show that a sparse subset of the model suffices to implement $P(w_t | w_{t-1})$ mappings. Mechanistically, these subnetworks concentrate in the embedding, first MLP, and output projection matrices, with negligible involvement of attention or deeper MLP layers (Chang et al., 21 Apr 2025).

Embedding Generalization

BOS models and fastText-like architectures use subword bigram embeddings to construct representations for OOV and rare words, crucial for morphologically rich languages (Zhao et al., 2018, Bojanowski et al., 2016).

6. Efficiency Optimizations and Data Structures

• Storing bigram tables using dense or sparse arrays indexed by subword IDs, option to quantize to int8 for further compression (Libovický et al., 2024)
• Reversed tries or hash tables for $\mathcal{S}$ to permit efficient subword lookup
• Precomputing likely successor subwords for pruned beam search
• Substring hashing (e.g., FNV-1a) for character bigram mapping to fixed-size tables (Bojanowski et al., 2016)
• Hogwild (lock-free) parallel SGD for rapid updates during embedding learning (Bojanowski et al., 2016)

These strategies ensure practical speed and scalability for both training and application.

7. Context, Limitations, and Interpretive Remarks

Bigram-only subword models represent a tractable compromise, offering lightweight computation, fast inference, and reasonable coverage of local patterns. However, they systematically miss longer-range morphological structure, resulting in degraded performance for fine-grained semantic tasks. For morphologically complex languages and semantic similarity objectives, higher-order $n$-grams ($n = 3$–6) consistently outperform pure bigrams (Zhao et al., 2018, Bojanowski et al., 2016). Nevertheless, with appropriate distillation or subnet selection, bigram models remain a robust, efficient baseline or component within larger segmentation and language modeling frameworks.

---

References:

• "Generalizing Word Embeddings using Bag of Subwords" (Zhao et al., 2018)
• "Enriching Word Vectors with Subword Information" (Bojanowski et al., 2016)
• "Lexically Grounded Subword Segmentation" (Libovický et al., 2024)
• "Bigram Subnetworks: Mapping to Next Tokens in Transformer LLMs" (Chang et al., 21 Apr 2025)