Hashed N-Gram Feature Spaces

• Hashed n-gram feature spaces are representations where n-grams are hashed into fixed-dimensional vectors, drastically reducing memory requirements and handling sparse data.
• They employ diverse hashing methods—from simple count-based mappings to advanced hybrid techniques—to balance computational efficiency, collision management, and theoretical guarantees.
• Applications span language model pretraining, fast classification, and dense/sparse retrieval, demonstrating significant speed and accuracy gains in NLP, IR, and bioinformatics.

Hashed n-gram feature spaces are a class of representations in which all $n$-grams from a document or sequence are mapped via a hash function into a fixed-dimensional vector space. This technique enables efficient handling of the vast and sparse combinatorial space of $n$-grams common in natural language and other sequential domains, circumventing the need to store or operate directly on exponentially large $n$-gram vocabularies. Key instantiations range from simple hashing of bag-of-$n$-gram counts to randomized embeddings with theoretical norm-preservation guarantees, as well as advanced hybridization mechanisms for memory-efficient model scaling. Applications span data selection for LLM pretraining, fast classification, dense and sparse retrieval, and scalable embedding table management.

1. Formal Definitions and Construction

The canonical hashed $n$-gram mapping extracts all $n$-grams (unigrams, bigrams, character or byte $n$-grams, depending on the task) from an input $x$ to form a multiset $N(x)$. Each element $g \in N(x)$ is mapped via a hash function $n$0 into a discrete bucket $n$1, and the counts or statistics for each bucket are accumulated into a vector $n$2 or $n$3, e.g.

$n$4

No further normalization is required unless specified. In byteSteady (Zhang et al., 2021), $n$5-grams are aggregated across multiple lengths, hashed into $n$6 buckets, and then embedded into $n$7 via bucket-wise embedding lookup. NUMEN (Sharma, 21 Jan 2026) constructs high-dimensional count vectors with log-saturation and $n$8 normalization, weighting longer $n$9-grams more heavily. In X-GRAM (Chen et al., 23 Apr 2026), a hybrid routing scheme combines deterministic assignments for head (frequent) $n$0-grams and randomized (alias mixed) hashes for the long tail.

2. Hash Function Properties and Theoretical Guarantees

Hashing $n$1-grams requires balancing computational efficiency against desirable statistical properties such as uniformity, collision probability, and (pairwise) independence. Recursive hash families (including rolling hashes used for efficient substring processing) have been proven to possess at most pairwise independence (0705.4676). For pairwise independent hashing, irreducible polynomials over $n$2 are utilized, at higher compute cost, while cyclic polynomial hashing accelerates updates at the cost of only quasi-pairwise guarantees (recovering full pairwise independence after discarding $n$3 bits).

Feature hashing is a randomized linear embedding mapping $n$4 where each column has a single randomly signed nonzero, with the aggregate projection

$n$5

implementing the hashing trick (Freksen et al., 2018). For a document vector $n$6 with $n$7, the tightness of norm preservation under this embedding is governed by $n$8, $n$9, a distortion $n$0, and failure probability $n$1. The mapping preserves $n$2 up to $n$3 with probability $n$4 if

$n$5

for all $n$6, but for well-spread $n$7 (i.e., many distinct $n$8-grams), significantly smaller $n$9 suffices. High-dimensional hashing (e.g., $n$0–$n$1) is therefore sufficient to support both sub-linear collision probability and norm conservation for realistic $n$2-gram feature distributions (Sharma, 21 Jan 2026, Freksen et al., 2018).

3. Algorithmic Implementations and Complexity

A prototypical pipeline for hashed $n$3-gram featurization—e.g., DSIR (Xie et al., 2023)—operates as follows:

1. Compute global hashed $n$4-gram distributions, e.g., $n$5 for a raw corpus and $n$6 for a target;
2. For each candidate document, extract all unigrams and bigrams, hash and tally into a length-$n$7 count vector $n$8;
3. Estimate an importance weight based on the generative likelihood ratio $n$9;
4. Select via weighted resampling (e.g., GumbelTopK).

Time complexity is $n$0 for $n$1 documents of length $n$2, with storage requirements scaling as $n$3 for the feature distributions (Xie et al., 2023). In byteSteady (Zhang et al., 2021), feature extraction and embedding lookup are linear in $n$4, while NUMEN (Sharma, 21 Jan 2026) maintains fixed per-document memory and compute cost proportional to $n$5.

X-GRAM (Chen et al., 23 Apr 2026) introduces a hybrid hashing/alias mixing approach, with bucket allocation reflecting empirical token frequency and memory-centric scaling of parameter tables decoupled from FLOPs per forward pass. The pipeline supports integration into value or residual streams in a Transformer, with $n$6 table parameters for a compression ratio $n$7.

4. Empirical and Comparative Analysis

Hashed $n$8-gram features enable large-scale, rapid data selection, classification, and retrieval across modalities:

• DSIR with hashed unigrams and bigrams enables selection of $n$9 documents from The Pile in under 5 hours, with KL-reduction in hash space correlating $n$0 to downstream accuracy (Xie et al., 2023). Compared to discriminative FastText or unigrams alone, joint $n$1-gram hashing improves F1 by 0.7% and outperforms expert/manual curation baselines.
• byteSteady, using hashed byte $n$2-grams with embedding, matches or slightly outperforms large CNN and FastText baselines on diverse multilingual corpora and gene classification tasks, scaling to $n$3 and $n$4 (Zhang et al., 2021).
• NUMEN, employing deterministic CRC32-based hashing into up to $n$5 dimensions, achieves Recall@100 of 93.90% on LIMIT—exceeding BM25 and all learned dense retrievers—while colliding with probability only $n$6 at average $n$7 $n$8-grams per document (Sharma, 21 Jan 2026).
• X-GRAM demonstrates that hybrid frequency-aware hashing, combined with alias mixing, can efficiently compress long-tail token sets while preserving or surpassing baseline accuracy (improvements up to $n$9 points) even with 50% compression of the vocabulary table (Chen et al., 23 Apr 2026).

Empirical studies confirm that performance saturates beyond a moderate number of hash buckets or embedding dimensions, and that log-saturation and normalization significantly mitigate the dominance of highly frequent $x$0-grams (Zhang et al., 2021, Sharma, 21 Jan 2026). HyperEmbed (Alonso et al., 2020) demonstrates that hyperdimensional randomized embeddings lose little performance (often $x$1–$x$2 F1) while achieving up to $x$3 memory and speed improvements over full $x$4-gram statistics.

5. Collision Behavior, Interpretability, and Trade-offs

Hashing entails a fundamental trade-off: reduced feature space dimensionality at the cost of collisions—distinct $x$5-grams mapping to the same bucket. This introduces noise, but generative models operating directly on raw counts are robust to moderate collision rates (Xie et al., 2023). The probability of collision for $x$6 $x$7-grams and $x$8 buckets is $x$9 (Sharma, 21 Jan 2026). Empirically, even with collision rates up to a few percent, distributional properties needed for tasks such as data selection or retrieval are maintained.

Reduced interpretability is an inherent consequence: bucket indices lose direct correspondence to human-readable $N(x)$0-grams. While inspection of specific hash buckets is not possible, distributional matching on the hashed space remains predictive of downstream performance. Increasing $N(x)$1 decreases collisions but increases storage and computational cost. Specialized hash functions with pairwise independence (e.g., cyclic polynomial, irreducible polynomial) are preferred in applications requiring rigorous statistical guarantees or provable sketching error bounds (0705.4676). Hybrid schemes, as in X-GRAM, offer mitigation by isolating frequent (head) items into deterministic slots while compressing tail items via carefully managed hashing and alias mixing (Chen et al., 23 Apr 2026).

6. Extensions: Hyperdimensional and Hybrid Approaches

Distributed representations such as HyperEmbed project $N(x)$2-gram statistics into a fixed high-dimensional bipolar space using binding, bundling, and permutation (Alonso et al., 2020). These representations decouple feature dimensionality from $N(x)$3-gram order, allowing the designer to set the feature-discriminability trade-off via the embedding dimension $N(x)$4. Memory and compute improvements of $N(x)$5–$N(x)$6 are observed with negligible classification loss, especially for global classifiers (MLP, Ridge).

Recent advances (X-GRAM) propose data-aware routing, alias mixing, and integration with attention/residual pathways to further optimize memory-centric scaling and catastrophic slot collapse, enabling scaling to billion-parameter systems (Chen et al., 23 Apr 2026).

7. Applications and Impact Across Domains

Hashed $N(x)$7-gram feature spaces have demonstrated impact across:

• LLM pretraining data selection (DSIR) (Xie et al., 2023)
• Text and DNA fast classification (byteSteady) (Zhang et al., 2021)
• Dense and sparse document retrieval (NUMEN, BM25+hashing) (Sharma, 21 Jan 2026)
• Memory-efficient token embedding and parameter scaling in transformers (X-GRAM) (Chen et al., 23 Apr 2026)
• Large-scale sketching and approximate statistics for NLP (hashing trick, HyperEmbed) (Alonso et al., 2020, Freksen et al., 2018)

The universality, hardware-efficiency, ease of high-dimensional scaling, and alignment with theoretical random projection/statistical physics models make hashed $N(x)$8-gram spaces foundational in modern large-scale NLP, IR, and bioinformatics systems.