---
title: Hashed N-Gram Feature Space
url: https://www.emergentmind.com/topics/hashed-n-gram-feature-space
type: topic
---

# Hashed N-Gram Feature Space

A hashed n-gram feature space is a high-dimensional representation in which all (possibly overlapping) n-grams extracted from a sequence (text, DNA, or other) are deterministically mapped into a fixed-size feature vector using a hash function. This mapping collapses the combinatorially large space of distinct n-grams into a manageable dimensionality, supports both dense and sparse usage, incurs only minor information loss when properly parameterized, and allows for efficient large-scale learning, retrieval, and data selection without explicit enumeration of the n-gram vocabulary.

## 1. Mathematical Construction of Hashed N-Gram Feature Spaces

Let $T$ denote a sequence (of characters, bytes, or tokens) over an alphabet $\Sigma$. The process for constructing a hashed n-gram feature space comprises:

- **N-Gram Extraction:** For each $n$ in a prescribed set (e.g., $n\in\{3,4,5\}$), enumerate all substrings $g$ of length $n$ in $T$. Optionally, boundary markers ensure context fidelity (as in NUMEN, with “$\sqcup$” and “$”$) [2601.15205].
- **Hashing:** Each n-gram $g$ is mapped to an integer bucket $h(g) \in \{0,1,\dots,D-1\}$ using a deterministic function; e.g., $h(g) = \mathrm{CRC32}(g)\bmod D$ (character-level, NUMEN), $h(g) = (H(g)\bmod D) + 1$, with $H$ being FNV or CityHash (byteSteady) [2106.13302], or any appropriate hash function.
- **Aggregation:** For each bucket $j$, a feature count is computed:
$$
x_j = \sum_{g \in \mathcal{G}(T)} w(g)\,\mathbf{1}[h(g)=j]
$$
where $w(g)$ weights n-grams (e.g., longer n-grams receive larger weight) [2601.15205].
- **Post-Processing:** Optionally, counts are log-saturated to mimic BM25-style diminishing returns, and/or L2-normalized to obtain a unit vector suitable for maximum inner-product search (MIPS) or cosine similarity retrieval [2601.15205]. In classification contexts, the average of embedding lookups indexed by $h(g)$ is used [2106.13302].
- **Dimensionality:** The user sets $D \ll |\Sigma|^n$ arbitrarily large for collision mitigation and separability, trading off memory for accuracy.

## 2. Hash Function Choice, Independence, and Computational Considerations

The choice and analysis of hash functions for n-grams have profound implications:

- **Pairwise Independence Optimality:** Recursive (rolling) hash families, regardless of how constructed, can be at most pairwise independent; full k-wise independence ($k>2$) is impossible for sliding n-gram windows [0705.4676].
- **Irreducible vs. Cyclic Hashes:** Hashes built on irreducible polynomials over $\mathrm{GF}(2)[x]$ are formally pairwise independent but require $O(nw)$ operations per update; cyclic polynomial schemes, which correspond to bitwise rotations and XORs, are $O(1)$ per update and, after dropping $n-1$ bits, are also pairwise independent—this yields practical throughputs exceeding 100 million n-grams/sec per thread [0705.4676].
- **Uniformity and Collisions:** CRC32, FNV, and CityHash offer fast, hardware-accelerated, and empirically uniform distributions [2601.15205][2106.13302], with negligible bias and collision probability provided that $D$ is chosen commensurately with the expected number of n-grams per instance (see Section 3 below).

## 3. Collision Probability and Dimensionality Selection

Collisions—two distinct n-grams mapping to the same bucket—are a core property of the hashing trick:

- **Birthday Paradox Analysis:** Given $N$ n-grams and hash space size $D$, the collision probability is approximately
$$
P(\mathrm{collision}) \approx 1-\exp\left(-\frac{N^2}{2D}\right)
$$
For example, with $N=50$ and $D=32,\!768$, $P(\mathrm{collision})\approx 3.8\%$ [2601.15205].
- **Empirical Plateaus:** Retrieval or classification quality plateaus—recall ceases to increase—when collisions become prevalent. In NUMEN, increasing $D$ from $2^{12}$ to $2^{15}$ yields monotonic recall improvements, with saturation occurring as collisions surpass a few percent [2601.15205]; byteSteady observes similar diminishing returns above $D=2^{24}$ for byte-level tasks [2106.13302].
- **Theoretical Guidance:** The tradeoff between $\|\mathbf{x}\|_\infty/\|\mathbf{x}\|_2$, $m$ (hash dimension), $\epsilon$ (relative distortion tolerance), and $\delta$ (failure probability) is now fully characterized: for  $\rho = \|\mathbf{x}\|_\infty/\|\mathbf{x}\|_2$,
    - $m \geq 2/\epsilon^2\delta$ suffices for all $\mathbf{x}$ [1805.08539];
    - For less "peaky" vectors, smaller $m$ is possible, with the hashing trick achieving Johnson-Lindenstrauss-type bounds up to mild constant and log-log factors [1805.08539].

## 4. Applications: Retrieval, Classification, and Data Selection

Hashed n-gram feature spaces underpin a variety of large-scale machine learning and information retrieval systems:

- **Dense Retrieval (NUMEN):** Documents and queries are encoded into L2-normalized hashed n-gram vectors. Retrieval is performed via maximum inner product search, using standard tools (e.g., FAISS), and at sufficient $D$ matches or exceeds classic BM25 recall, achieving 93.90% Recall@100 at $D=32,768$ on the LIMIT benchmark [2601.15205]. The principal advantage is elimination of embedding bottlenecks: the geometry scales as $D$ increases, directly targeting the "sign-rank" bottleneck of low-dimensional learned representational spaces [2601.15205].
- **Classification (byteSteady):** Each byte-level n-gram is mapped via a hash function to a compact embedding table, and the per-instance representation is the average of selected embeddings. byteSteady achieves state-of-the-art or near-SOTA text and gene classification, robust to n-gram collisions and with modest model footprint (e.g., 256 MB at $d=16$, $D=2^{24}$, $n=\{4,8,12,16\}$), while allowing optional compression-based speedups [2106.13302].
- **Data Selection (DSIR):** Hashed n-gram histograms are used as tractable proxies for true n-gram distributions in multi-billion document corpora, enabling efficient calculation of document importance weights and associated data curation for language model training. KL-reduction evaluated in the hashed n-gram space correlates $r=0.82$ with downstream accuracy [2302.03169].

### Empirical Results: Recall@K vs. Hash Dimension (NUMEN, LIMIT Benchmark)

| Dimension | Recall@2 | Recall@10 | Recall@100 |
|-----------|----------|-----------|------------|
| 512       | 2.70%    | 7.15%     | 21.30%     |
| 1024      | 13.20%   | 23.85%    | 45.10%     |
| 2048      | 33.05%   | 49.45%    | 68.80%     |
| 4096      | 56.50%   | 70.10%    | 83.20%     |
| 8192      | 70.65%   | 81.60%    | 89.85%     |
| 16384     | 79.45%   | 86.65%    | 93.05%     |
| 32768     | 81.45%   | 88.00%    | 93.90%     |

For comparison, BM25’s Recall@100 is 93.6% [2601.15205].

## 5. Algorithmic and Practical Design Patterns

- **N-gram Order Selection:** Classification and retrieval benefit from multi-scale n-gram inclusion; e.g., byteSteady reports best results with $n\in\{4,8,12,16\}$ (text), $n\in 2[1..8]=\{2,4,6,8,10,12,14,16\}$ (genes) [2106.13302]. NUMEN fixes $n\in\{3,4,5\}$ [2601.15205].
- **Weighting and Saturation:** Assigning larger weights to longer n-grams and log-saturating repeated grams improve robustness to redundancy and match traditional IR heuristics (e.g., BM25) [2601.15205].
- **Learning vs. Determinism:** NUMEN is fully training-free; byteSteady learns embedding vectors but uses fixed hashing; DSIR operates entirely with hashed count vectors and generative mixture models [2601.15205][2106.13302][2302.03169].
- **Compression Extensions:** In byte-level settings, n-gram extraction can be performed on Huffman-compressed inputs. Lossless compression reduces runtime per document nearly linearly, with minimal impact on classification error for light compression (e.g., 0.2–0.5% increase), but more aggressive compression creates an explicit accuracy-speed trade-off frontier [2106.13302].
- **Streaming and Scalability:** Hashed n-gram feature maps can be constructed in $O(L)$ time per document (with $L$ its length), require modest per-document memory (scaling with $d$ or $D$), and support streaming aggregation or distributed computation [2302.03169].

## 6. Theoretical Guarantees and Limitations

- **Norm Preservation:** When $\|\mathbf{x}\|_\infty/\|\mathbf{x}\|_2$ is small (typical for n-gram histograms), hashed feature spaces behave like sparse Johnson-Lindenstrauss transforms. The central bound is: for $\varepsilon, \delta$,
$$
\Pr\left[|\,\|A\mathbf{x}\|_2^2 - \|\mathbf{x}\|_2^2\,| < \varepsilon \|\mathbf{x}\|_2^2\right] \geq 1-\delta
$$
for $m \geq 2/(\varepsilon^2\delta)$, otherwise as a precise function of $\|\mathbf{x}\|_\infty/\|\mathbf{x}\|_2$ [1805.08539].
- **Hash Family Result:** Recursive n-gram hashing cannot exceed pairwise independence due to window overlap constraints; no rolling hash attains full 3-wise independence [0705.4676].
- **Collision Robustness:** Empirically, collisions at the few-percent level have little effect; both learning-based (byteSteady) and unsupervised (DSIR) pipelines are robust by virtue of averaging embeddings or focusing on histogram statistics [2106.13302][2302.03169].
- **Trade-offs:** There is a direct, quantifiable trade-off among feature space dimensionality, collision rate, empirical accuracy (retrieval/classification), and system memory requirements [2601.15205][2106.13302][1805.08539].

## 7. Extensions, Impact, and Open Challenges

Hashed n-gram feature spaces enable scalable learning and retrieval over unlimited vocabularies, support flexible dimensionality allocation per task, and admit practical hardware-accelerated implementations. They are now deployed across dense retrieval, fast multiclass text and DNA classification, and billions-scale document resampling for language model pretraining [2601.15205][2106.13302][2302.03169]. Open directions include architectural integration with transformer language models (specifically for retrieval augmentation), dynamic or adaptive dimensionality for low-resource tasks, and rigorous analysis of the expressivity–collision trade-off in broader mixture-of-expert and continual learning settings.

Source: https://www.emergentmind.com/topics/hashed-n-gram-feature-space