---
title: N-gram HD Encoders & Transformer Fusion
url: https://www.emergentmind.com/topics/n-gram-hd-encoders
type: topic
---

# N-gram HD Encoders & Transformer Fusion

N-gram HD (High-Dimensional) Encoders are computational frameworks for representing n-gram statistics and contextual span information from text as fixed-length, high-dimensional vectors. These approaches synthesize ideas from hyperdimensional computing (HDC) and neural encoding architectures to produce distributed, resource-efficient representations suitable for both classic classifiers and modern Transformer-based models. Two principal lines—hyperdimensional binding/bundling schemes [2003.01821] and n-gram Transformer fusion architectures [2105.01279]—define current methodologies.

## 1. Hyperdimensional Computing Principles for N-gram Encoding

Hyperdimensional (HD) or Vector-Symbolic Architectures encode symbols, sequences, and sets by associating each with a high-dimensional vector, typically with dimension $D \in [10^3, 10^5]$. Each base vector $v_S \in \{\pm1\}^D$ is drawn i.i.d. and nearly orthogonal. This property supports distributed operations:

- **Binding (element-wise multiplication, $\odot$):** Used to encode sequence or order, crucial for n-grams.
- **Bundling (element-wise addition, $+$):** Aggregates multiple hypervectors (e.g., n-gram counts) in the same $D$-dimensional space.

For character n-gram encoding: each character is mapped to a random bipolar vector, permuted by a fixed operator $\rho$ to encode positional information, and bound together to form n-gram representations. These are aggregated across a text to form a single $D$-dimensional summary vector. The process is defined as:

\[
v_w = \bigodot_{j=1}^n \rho^j(v_{S_j}),
\qquad
V = \sum_{w} c(w)\,v_w,
\qquad
\widehat{V} = \frac{V}{\|V\|_2}
\]
where $c(w)$ is the n-gram count in a document, and $\rho$ denotes a fixed cyclic permutation of vector coordinates.

## 2. Algorithmic Workflow and Pseudocode

A standard workflow for N-gram HD encoding [2003.01821] consists of the following steps:

1. **Initialization:** Assign each symbol $S$ in the alphabet $\Sigma$ a random bipolar base vector $v_S \in \{\pm1\}^D$.
2. **N-gram Extraction:** Use a sliding window over the text $D$ to enumerate all overlapping character n-grams.
3. **Binding:** For each n-gram $w$, bind the permuted symbol hypervectors as above.
4. **Bundling:** Accumulate the bound n-gram hypervectors into the sum $V$.
5. **Normalization:** Obtain $\widehat{V}$ by $\ell_2$ normalization.
6. **Classifier Input:** The normalized HD vector is input to standard classifiers.

Pseudocode for the encoder (as per [2003.01821]):

```plaintext
Initialize item memory: for each S∈Σ, v_S ← random bipolar vector in {±1}^D
V ← 0 ∈ ℝ^D
for each n-gram w in D:
    h ← v_{D[i]}
    for j = 2 to n:
        h ← h ⊙ ρ^{j}(v_{D[i + j - 1]})
    V ← V + h
normalize: ẊV ← V / ||V||₂
return ẊV
```

The computational complexity is $O(|D| \cdot n \cdot D)$.

## 3. Trade-offs: Dimensionality, Context Size, and Resource Efficiency

A key advantage of N-gram HD encoders is the decoupling of output dimensionality $D$ from the combinatorial explosion in $n$ (traditional n-gram models require $a^n$ counters; HD encoding always yields a $D$-dimensional vector). Selection of $D$ governs fidelity and resource consumption:

- **Larger $D$:** Higher fidelity to true n-gram histograms, increased classification accuracy, with increased memory and compute cost.
- **Smaller $D$:** Substantial memory and throughput savings, potential loss in accuracy if under-parameterized.

Empirical results indicate that F$_1$ scores increase rapidly for small $D$ (e.g., $D = 32$ offers marked improvement), saturating at dataset-dependent $D^*$. For instance, $D^* \approx 512$ for small corpora, $D^* \approx 4096$ for large [2003.01821]. Resource improvements scale proportionally: train/test speedups and memory reductions of $10\times$ to $100\times$ over dense n-gram representations are common at minimal accuracy loss.

## 4. Integration into Transformer Architectures: N-gram Fusion Techniques

In neural text encoders such as ZEN 2.0 [2105.01279], n-gram high-dimensional embeddings are constructed via unsupervised statistical extraction and Transformer-based contextualization:

- **N-gram Extraction:** Spans of length $2$–$8$ are selected by pointwise mutual information (PMI) and frequency thresholds; for Chinese, PMI$\ge3$, $c(w)\ge15$ yields $|V_{ng}|\approx261,000$; for Arabic, PMI$\ge10$, $c(w)\ge20$, $|V_{ng}|\approx194,000$.
- **HD Embedding & Encoding:** A learnable lookup table $E_{ng}\in\mathbb{R}^{|V_{ng}|\times d}$ (with $d=768$ or $1024$), followed by a six-layer Transformer encoder for contextualization.

Fusion with character/token-level Transformer states is executed layer-wise. Let $\mathbf{v}_i^{(l)}$ denote the token state at layer $l$; overlapping n-grams $\{g_{i,1},...,g_{i,K_i}\}$ contribute weighted contextual vectors. Fusion is done by:
\[
\mathbf{v}_i^{(l)*} = \mathbf{v}_i^{(l)} + \sum_{k=1}^{K_i} p_{i,k} \boldsymbol\mu_{i,k}^{(l)}
\]
where $p_{i,k} = \frac{c(g_{i,k})}{\sum_{k'} c(g_{i,k'})}$; no gating or concatenation is used. N-gram signals augment token-level representations directly. The n-gram encoder operates as a parallel Transformer.

## 5. Empirical Evaluation and Performance Metrics

HyperEmbed [2003.01821] evaluated N-gram HD encoders on three small (Chatbot, AskUbuntu, WebApplication) and one large (20NewsGroups) corpus:

- **Baselines:** Conventional character n-gram counts ($\sim$200,000 dimensions).
- **HD Embeddings:** $n=2$–4, $D=2^5$–$2^{14}$.
- **Classifiers:** Ridge, KNN, MLP, PA, RF, LSVC, SGD, NC, BNB.

Key results:

- For AskUbuntu (MLP, $n=3$, $D=512$): F$_1$ = 0.91 vs. baseline 0.92, $4.62\times$ faster training, $3.84\times$ faster test, $6.18\times$ memory reduction.
- For 20NewsGroups ($D=2048$, $n=2$–3): Most classifiers maintained $\sim$90% of baseline F$_1$ with $50$–$200\times$ speedup and $100\times$ memory reduction.

Linear classifiers and shallow MLPs generally outperformed local or tree-based models, which lost accuracy due to the distributed representation's smoothing effects.

ZEN 2.0 [2105.01279] demonstrates consistent state-of-the-art improvements across a battery of Chinese and Arabic NLP tasks (e.g., MSR-CWS F$_1$ = 98.66, CMRC2018 F$_1$ = 89.92), outperforming prior benchmarks typically by 0.1–2.0 points in absolute metric terms.

## 6. Practical Guidelines and Adaptation Considerations

Recommendations for practitioners [2003.01821, 2105.01279]:

- Select $n=2$–4 in small corpora; $n=2$–3 in large.
- Sweep $D$ from $2^5$ to $2^{14}$; select minimal $D$ achieving $95$–98% of baseline accuracy.
- Prefer linear and shallow neural classifiers for HD vectors.
- For resource-constrained environments, binarize encodings and classifier weights.
- ZEN 2.0 architecture adapts to multiple languages (Chinese, Arabic) and domains via threshold tuning and separate n-gram vocabularies, without structural changes.

## 7. Significance and Applications

N-gram HD encoders address scaling and efficiency challenges in embedding n-gram statistics for NLP tasks. By leveraging high-dimensionality and distributed encoding, they provide concise, accurate representations with dramatic resource savings. They integrate seamlessly into classic ML pipelines and Transformer-based neural architectures, enabling robust, end-to-end modeling of contextual spans. This framework offers practical trade-offs between memory, speed, and accuracy, and demonstrates superiority in multilingual, multi-domain applications, verified experimentally on several production-scale corpora [2003.01821, 2105.01279].

Source: https://www.emergentmind.com/topics/n-gram-hd-encoders