---
title: Hashed N-Gram Feature Spaces
url: https://www.emergentmind.com/topics/hashed-n-gram-feature-spaces
type: topic
---

# Hashed N-Gram Feature Spaces

Hashed n-gram feature spaces are a class of representations in which all $n$-grams from a document or sequence are mapped via a hash function into a fixed-dimensional vector space. This technique enables efficient handling of the vast and sparse combinatorial space of $n$-grams common in natural language and other sequential domains, circumventing the need to store or operate directly on exponentially large $n$-gram vocabularies. Key instantiations range from simple hashing of bag-of-$n$-gram counts to randomized embeddings with theoretical norm-preservation guarantees, as well as advanced hybridization mechanisms for memory-efficient model scaling. Applications span data selection for language model pretraining, fast classification, dense and sparse retrieval, and scalable embedding table management.

## 1. Formal Definitions and Construction

The canonical hashed $n$-gram mapping extracts all $n$-grams (unigrams, bigrams, character or byte $n$-grams, depending on the task) from an input $x$ to form a multiset $N(x)$. Each element $g \in N(x)$ is mapped via a hash function $h$ into a discrete bucket $j \in \{1,\ldots,m\}$, and the counts or statistics for each bucket are accumulated into a vector $z \in \mathbb{N}^m$ or $x \in \mathbb{R}^m$, e.g.
$$
z[j] = \sum_{g \in N(x)} 1[h(g)=j], \quad x[j] = \text{frequency of bucket } j.
$$
No further normalization is required unless specified. In byteSteady [2106.13302], $n$-grams are aggregated across multiple lengths, hashed into $M$ buckets, and then embedded into $\mathbb{R}^d$ via bucket-wise embedding lookup. NUMEN [2601.15205] constructs high-dimensional count vectors with log-saturation and $L_2$ normalization, weighting longer $n$-grams more heavily. In X-GRAM [2604.21724], a hybrid routing scheme combines deterministic assignments for head (frequent) $n$-grams and randomized (alias mixed) hashes for the long tail.

## 2. Hash Function Properties and Theoretical Guarantees

Hashing $n$-grams requires balancing computational efficiency against desirable statistical properties such as uniformity, collision probability, and (pairwise) independence. Recursive hash families (including rolling hashes used for efficient substring processing) have been proven to possess at most pairwise independence [0705.4676]. For pairwise independent hashing, irreducible polynomials over $\mathbb{F}_2$ are utilized, at higher compute cost, while cyclic polynomial hashing accelerates updates at the cost of only quasi-pairwise guarantees (recovering full pairwise independence after discarding $n-1$ bits).

Feature hashing is a randomized linear embedding mapping $A\!: \mathbb{R}^n \to \mathbb{R}^m$ where each column has a single randomly signed nonzero, with the aggregate projection
$$
y_{h(j)} \mathrel{+}= \sigma_j x_j,
$$
implementing the hashing trick [1805.08539]. For a document vector $x$ with $r = \|x\|_\infty / \|x\|_2$, the tightness of norm preservation under this embedding is governed by $m$, $r$, a distortion $\varepsilon$, and failure probability $\delta$. The mapping preserves $\|x\|_2^2$ up to $\pm\varepsilon$ with probability $1-\delta$ if
$$
m \gtrsim \varepsilon^{-2} \log(1/\delta)
$$
for all $x$, but for well-spread $x$ (i.e., many distinct $n$-grams), significantly smaller $m$ suffices. High-dimensional hashing (e.g., $d \sim 10^4$–$10^5$) is therefore sufficient to support both sub-linear collision probability and norm conservation for realistic $n$-gram feature distributions [2601.15205, 1805.08539].

## 3. Algorithmic Implementations and Complexity

A prototypical pipeline for hashed $n$-gram featurization—e.g., DSIR [2302.03169]—operates as follows:
1. Compute global hashed $n$-gram distributions, e.g., $q$ for a raw corpus and $p$ for a target;
2. For each candidate document, extract all unigrams and bigrams, hash and tally into a length-$m$ count vector $z$;
3. Estimate an importance weight based on the generative likelihood ratio $\prod_j (p[j]/q[j])^{z[j]}$;
4. Select via weighted resampling (e.g., GumbelTopK).

Time complexity is $O(NL)$ for $N$ documents of length $L$, with storage requirements scaling as $O(m)$ for the feature distributions [2302.03169]. In byteSteady [2106.13302], feature extraction and embedding lookup are linear in $|\mathcal{N}| \cdot L$, while NUMEN [2601.15205] maintains fixed per-document memory and compute cost proportional to $d$.

X-GRAM [2604.21724] introduces a hybrid hashing/alias mixing approach, with bucket allocation reflecting empirical token frequency and memory-centric scaling of parameter tables decoupled from FLOPs per forward pass. The pipeline supports integration into value or residual streams in a Transformer, with $O(\rho |\mathcal{V}|)$ table parameters for a compression ratio $\rho$.

## 4. Empirical and Comparative Analysis

Hashed $n$-gram features enable large-scale, rapid data selection, classification, and retrieval across modalities:
- DSIR with hashed unigrams and bigrams enables selection of $10^8$ documents from The Pile in under 5 hours, with KL-reduction in hash space correlating $r=0.82$ to downstream accuracy [2302.03169]. Compared to discriminative FastText or unigrams alone, joint $n$-gram hashing improves F1 by 0.7% and outperforms expert/manual curation baselines.
- byteSteady, using hashed byte $n$-grams with embedding, matches or slightly outperforms large CNN and FastText baselines on diverse multilingual corpora and gene classification tasks, scaling to $n=16$ and $M = 2^{24}$ [2106.13302].
- NUMEN, employing deterministic CRC32-based hashing into up to $d=32,768$ dimensions, achieves Recall@100 of 93.90% on LIMIT—exceeding BM25 and all learned dense retrievers—while colliding with probability only $3.8\%$ at average $N=50$ $n$-grams per document [2601.15205].
- X-GRAM demonstrates that hybrid frequency-aware hashing, combined with alias mixing, can efficiently compress long-tail token sets while preserving or surpassing baseline accuracy (improvements up to $4.4$ points) even with 50% compression of the vocabulary table [2604.21724].

Empirical studies confirm that performance saturates beyond a moderate number of hash buckets or embedding dimensions, and that log-saturation and normalization significantly mitigate the dominance of highly frequent $n$-grams [2106.13302, 2601.15205]. HyperEmbed [2003.01821] demonstrates that hyperdimensional randomized embeddings lose little performance (often $<1$–$2\%$ F1) while achieving up to $100\times$ memory and speed improvements over full $n$-gram statistics.

## 5. Collision Behavior, Interpretability, and Trade-offs

Hashing entails a fundamental trade-off: reduced feature space dimensionality at the cost of collisions—distinct $n$-grams mapping to the same bucket. This introduces noise, but generative models operating directly on raw counts are robust to moderate collision rates [2302.03169]. The probability of collision for $N$ $n$-grams and $m$ buckets is $1-\exp(-N^2/2m)$ [2601.15205]. Empirically, even with collision rates up to a few percent, distributional properties needed for tasks such as data selection or retrieval are maintained.

Reduced interpretability is an inherent consequence: bucket indices lose direct correspondence to human-readable $n$-grams. While inspection of specific hash buckets is not possible, distributional matching on the hashed space remains predictive of downstream performance. Increasing $m$ decreases collisions but increases storage and computational cost. Specialized hash functions with pairwise independence (e.g., cyclic polynomial, irreducible polynomial) are preferred in applications requiring rigorous statistical guarantees or provable sketching error bounds [0705.4676]. Hybrid schemes, as in X-GRAM, offer mitigation by isolating frequent (head) items into deterministic slots while compressing tail items via carefully managed hashing and alias mixing [2604.21724].

## 6. Extensions: Hyperdimensional and Hybrid Approaches

Distributed representations such as HyperEmbed project $n$-gram statistics into a fixed high-dimensional bipolar space using binding, bundling, and permutation [2003.01821]. These representations decouple feature dimensionality from $n$-gram order, allowing the designer to set the feature-discriminability trade-off via the embedding dimension $d$. Memory and compute improvements of $10$–$100\times$ are observed with negligible classification loss, especially for global classifiers (MLP, Ridge).

Recent advances (X-GRAM) propose data-aware routing, alias mixing, and integration with attention/residual pathways to further optimize memory-centric scaling and catastrophic slot collapse, enabling scaling to billion-parameter systems [2604.21724].

## 7. Applications and Impact Across Domains

Hashed $n$-gram feature spaces have demonstrated impact across:
- Language model pretraining data selection (DSIR) [2302.03169]
- Text and DNA fast classification (byteSteady) [2106.13302]
- Dense and sparse document retrieval (NUMEN, BM25+hashing) [2601.15205]
- Memory-efficient token embedding and parameter scaling in transformers (X-GRAM) [2604.21724]
- Large-scale sketching and approximate statistics for NLP (hashing trick, HyperEmbed) [2003.01821, 1805.08539]

The universality, hardware-efficiency, ease of high-dimensional scaling, and alignment with theoretical random projection/statistical physics models make hashed $n$-gram spaces foundational in modern large-scale NLP, IR, and bioinformatics systems.

Source: https://www.emergentmind.com/topics/hashed-n-gram-feature-spaces