Additive Feature Hashing

• Additive Feature Hashing is a method that transforms high-dimensional sparse features into fixed-length numerical vectors via additive accumulation and hash functions.
• It preserves inner products and norms in expectation, supporting streaming updates and incremental, one-pass computation without explicit vocabulary storage.
• Both sparse and dense variants trade off computational efficiency and noise reduction, with applications in word embeddings, classification, and similarity search.

Additive feature hashing is a family of hash-based embeddings that map high-dimensional categorical or sparse feature representations into fixed-length numerical vectors by additive accumulation. In the classical construction, often identified with the hashing trick, each feature is mapped to a bucket by a hash function and optionally multiplied by a random sign before being added to that bucket; in a later dense formulation, each feature is mapped deterministically to a high-dimensional random vector and instance representations are formed by summing those vectors. The unifying objective is to eliminate explicit vocabulary storage while preserving inner products, norms, or cosine similarities in expectation, thereby supporting streaming updates, dimensionality reduction, and large-vocabulary learning (Freksen et al., 2018, Andrecut, 2021, Argerich et al., 2016).

1. Formal constructions

The canonical additive feature-hashing map sends a sparse vector $x$ into a lower-dimensional vector $\Phi(x) \in \mathbb{R}^k$ by means of two hash functions: an index hash $h:\text{feature}\rightarrow\{0,\dots,k-1\}$ and a sign hash $\xi:\text{feature}\rightarrow\{-1,+1\}$. The embedding is

$\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$

All features that collide in the same bucket contribute additively to the same coordinate. Because the mapping is linear, $\Phi(x+y)=\Phi(x)+\Phi(y)$, which is the basic reason the method supports one-pass construction, incremental updates, and exact merging by summation (Argerich et al., 2016).

A matrix formulation makes the same idea explicit. For the classical scheme, one defines a sparse projection matrix $A\in\mathbb{R}^{m\times n}$ with entries

$A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$

so every column has exactly one nonzero entry, equal to either $-1$ or $1$. The hashed vector is then $\Phi(x) \in \mathbb{R}^k$0, with coordinate form $\Phi(x) \in \mathbb{R}^k$1. This is the sparsity-$\Phi(x) \in \mathbb{R}^k$2 case of sparse Johnson–Lindenstrauss embeddings; more general sparse JL constructions assign each original coordinate to $\Phi(x) \in \mathbb{R}^k$3 rows, with entries $\Phi(x) \in \mathbb{R}^k$4, so that each column has exactly $\Phi(x) \in \mathbb{R}^k$5 nonzeros (Jagadeesan, 2019).

The dense variant called Additive Feature Hashing in 2021 replaces bucket updates by vector summation. A categorical feature $\Phi(x) \in \mathbb{R}^k$6 is mapped to a $\Phi(x) \in \mathbb{R}^k$7-dimensional random vector $\Phi(x) \in \mathbb{R}^k$8, typically either Rademacher or Gaussian, and an instance with feature set $\Phi(x) \in \mathbb{R}^k$9 is embedded as

$h:\text{feature}\rightarrow\{0,\dots,k-1\}$0

To avoid storing all $h:\text{feature}\rightarrow\{0,\dots,k-1\}$1, the vector is generated deterministically from the feature identity, for example by using a cryptographic hash such as SHAKE-256 to produce bits and interpreting them as entries in $h:\text{feature}\rightarrow\{0,\dots,k-1\}$2, optionally scaled by $h:\text{feature}\rightarrow\{0,\dots,k-1\}$3 (Andrecut, 2021).

2. Preservation guarantees and JL connections

In the classical signed scheme, additive feature hashing preserves inner products and norms in expectation. For any $h:\text{feature}\rightarrow\{0,\dots,k-1\}$4,

$h:\text{feature}\rightarrow\{0,\dots,k-1\}$5

For unit vectors, the fluctuation of the squared norm satisfies the second-moment bound $h:\text{feature}\rightarrow\{0,\dots,k-1\}$6, where $h:\text{feature}\rightarrow\{0,\dots,k-1\}$7. The sharp high-probability behavior depends on the data-dependent ratio $h:\text{feature}\rightarrow\{0,\dots,k-1\}$8. Defining $h:\text{feature}\rightarrow\{0,\dots,k-1\}$9 as the largest admissible ratio such that feature hashing preserves norms within relative error $\xi:\text{feature}\rightarrow\{-1,+1\}$0 with probability at least $\xi:\text{feature}\rightarrow\{-1,+1\}$1, the tight asymptotic characterization is

$\xi:\text{feature}\rightarrow\{-1,+1\}$2

whenever $\xi:\text{feature}\rightarrow\{-1,+1\}$3; moreover, $\xi:\text{feature}\rightarrow\{-1,+1\}$4 if $\xi:\text{feature}\rightarrow\{-1,+1\}$5 and $\xi:\text{feature}\rightarrow\{-1,+1\}$6 if $\xi:\text{feature}\rightarrow\{-1,+1\}$7 (Freksen et al., 2018).

The sparse-JL generalization with $\xi:\text{feature}\rightarrow\{-1,+1\}$8 nonzeros per column improves this picture. Let $\xi:\text{feature}\rightarrow\{-1,+1\}$9 denote the largest $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$0 such that the JL guarantee holds uniformly for all vectors in $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$1. The main result of the 2019 analysis is a tight tradeoff showing that increasing sparsity from $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$2 to $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$3 improves norm preservation approximately by a factor of $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$4 in the intermediate regimes. Empirically, on TF–IDF datasets with $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$5 and fixed $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$6, moving from $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$7 to $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$8 reduced the failure probability by approximately $\Phi(x)_j = \sum_{i:h(i)=j} \xi(i)x_i.$9, and moving to $\Phi(x+y)=\Phi(x)+\Phi(y)$0 or $\Phi(x+y)=\Phi(x)+\Phi(y)$1 reduced it by approximately $\Phi(x+y)=\Phi(x)+\Phi(y)$2 (Jagadeesan, 2019).

The dense AFH construction has an analogous expectation-preservation property, but its noise mechanism is different. For two instances $\Phi(x+y)=\Phi(x)+\Phi(y)$3 and $\Phi(x+y)=\Phi(x)+\Phi(y)$4, the inner product decomposes into the weighted overlap term plus a cross-noise term $\Phi(x+y)=\Phi(x)+\Phi(y)$5 with mean zero. With normalized Rademacher vectors, $\Phi(x+y)=\Phi(x)+\Phi(y)$6 for distinct tokens, and Hoeffding’s inequality gives

$\Phi(x+y)=\Phi(x)+\Phi(y)$7

More generally, the variance proxy of the cross-noise term is bounded by

$\Phi(x+y)=\Phi(x)+\Phi(y)$8

so inner products concentrate as $\Phi(x+y)=\Phi(x)+\Phi(y)$9 grows. This places dense AFH in the same broad similarity-preserving tradition as Johnson–Lindenstrauss projections, even though the representation is obtained by summing feature-specific random directions rather than by index collisions (Andrecut, 2021).

3. Streaming algorithms and implementation

The additive property gives the classical method a particularly simple computational profile. For a sparse input, one maintains an array of length $A\in\mathbb{R}^{m\times n}$0, and for each nonzero feature $A\in\mathbb{R}^{m\times n}$1 performs the update $A\in\mathbb{R}^{m\times n}$2. Because there is exactly one addition per nonzero in the classical scheme, embedding time is $A\in\mathbb{R}^{m\times n}$3. The same linearity also makes the method naturally streaming: if a coordinate changes by $A\in\mathbb{R}^{m\times n}$4, one updates only the corresponding hashed bucket; if partial results are computed on separate shards, they can be merged exactly by vector addition (Freksen et al., 2018).

Hash2Vec exploits these same properties for corpus-scale word embeddings. The algorithm processes text once, keeps a sliding buffer of the last $A\in\mathbb{R}^{m\times n}$5 tokens, and updates word vectors by hashed accumulation of local context signals. For a corpus of $A\in\mathbb{R}^{m\times n}$6 tokens and window size $A\in\mathbb{R}^{m\times n}$7, the time complexity is $A\in\mathbb{R}^{m\times n}$8, the vector store is $A\in\mathbb{R}^{m\times n}$9, and the streaming buffer is $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$0. Because updates are purely additive, shards can be processed independently and merged by summation, and the resulting model is identical to single-pass processing. With fixed corpus and fixed $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$1, $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$2, $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$3, and $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$4, the construction is deterministic and does not require stochastic training (Argerich et al., 2016).

Dense AFH exchanges sparse index updates for dense vector addition. The reference implementation constructs $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$5 on the fly by SHAKE-256, converts the resulting bits into $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$6, scales by $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$7, and accumulates $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$8 into the instance vector. This makes the per-instance cost $A_{i,j} = \sigma(j)\,\mathbf{1}\{h(j)=i\},$9 rather than $-1$0, and the dense vectors can increase memory bandwidth relative to sparse hashing. The benefit is that there is no index sharing among features and no need to store a vocabulary or a per-token vector table (Andrecut, 2021).

4. Hash2Vec and additive hashing for word embeddings

Hash2Vec applies additive feature hashing directly to word co-occurrence features. Instead of materializing the full $-1$1 co-occurrence matrix, it maintains a vector $-1$2 for each vocabulary item $-1$3 and streams through the corpus, adding hashed, signed context contributions. With a symmetric window of size $-1$4, the update for target word $-1$5 and context token $-1$6 at distance $-1$7 is

$-1$8

where the paper reports best results with a Gaussian decay $-1$9. If $1$0, the method reduces to hashing raw co-occurrence counts (Argerich et al., 2016).

The construction is an explicitly distributional one. The reduced vector for a word is the sum of hashed, signed, weighted indicator vectors of its contexts, so if a pair $1$1 appears repeatedly, the contribution accumulates across occurrences. The paper’s aggregated formula is

$1$2

This makes Hash2Vec a compressed surrogate for the co-occurrence matrix: with flat weighting it hashes raw co-occurrence rows, and with distance weighting it hashes distance-weighted rows. The paper explicitly suggests using these hashed rows either directly or as inputs to downstream matrix-factorization methods such as GloVe, thereby providing a memory-efficient substitute for the $1$3 matrix (Argerich et al., 2016).

Empirically, the reported similarity-benchmark correlations increase with dimensionality and approach the use of full co-occurrence vectors. In a comparison to GloVe trained on the same corpus with $1$4 and $1$5, the nearest neighbors returned by Hash2Vec are qualitatively similar. For the query “computer,” Hash2Vec returns [program, computers, hardware, programs, game], while GloVe returns [computers, software, systems, hardware, game]; for “king,” Hash2Vec returns [england, iii, henry, charles, james], while GloVe returns [son, emperor, ii, kings]. The method also solves some analogy patterns, including Paris:France :: Moscow:Russia and Cow:Milk :: Pig:Meat. The paper presents this as evidence that a training-free additive hashing scheme can capture semantic structure in linear time in the size of the data (Argerich et al., 2016).

5. Classification, similarity search, and empirical performance

Dense AFH was evaluated against the standard hashing trick on synthetic string similarity, written language identification, and SMS spam detection. In the synthetic experiment, similarity between original and corrupted strings was measured by cosine similarity over $1$6-gram features at various dimensionalities $1$7, and the AFH and HT curves were described as essentially indistinguishable. On WiLI-2018, which contains 235 languages with 500 training and 500 test paragraphs per language, AFH and HT using $1$8 grams and nearest-neighbor classification by cosine similarity achieved $1$9 accuracy when $\Phi(x) \in \mathbb{R}^k$00. On the SMS spam dataset of 5,574 messages, with 747 spam and 4,827 ham, AFH and HT with 3-gram features and nearest-neighbor classification achieved, at $\Phi(x) \in \mathbb{R}^k$01–$\Phi(x) \in \mathbb{R}^k$02, $\Phi(x) \in \mathbb{R}^k$03 accuracy, $\Phi(x) \in \mathbb{R}^k$04 spam caught, and $\Phi(x) \in \mathbb{R}^k$05 blocked hams (Andrecut, 2021).

Task	Setup	Reported result
WiLI-2018	3-gram features; cosine nearest neighbor; $\Phi(x) \in \mathbb{R}^k$06	$\Phi(x) \in \mathbb{R}^k$07 accuracy
SMS spam filtering	3-gram features; cosine nearest neighbor; $\Phi(x) \in \mathbb{R}^k$08–$\Phi(x) \in \mathbb{R}^k$09	$\Phi(x) \in \mathbb{R}^k$10 accuracy; $\Phi(x) \in \mathbb{R}^k$11 spam caught; $\Phi(x) \in \mathbb{R}^k$12 blocked hams

A separate application line treats feature hashing itself as a valid JL projection for locality-sensitive hashing. Two families were proposed for angular distance: Feature Hashing LSH, which hashes by the $\Phi(x) \in \mathbb{R}^k$13 over projected coordinates, and Directional Feature Hashing LSH, which hashes by the signs of projected coordinates. The sign-based family reduces to Charikar’s Hyperplane LSH when the projection becomes dense, while the sparse feature-hashing realization is faster because each input coordinate contributes only through signed additions. On SIFT 1M, for the target operating point $\Phi(x) \in \mathbb{R}^k$14 at distance $\Phi(x) \in \mathbb{R}^k$15 and $\Phi(x) \in \mathbb{R}^k$16 at distance $\Phi(x) \in \mathbb{R}^k$17, the reported parameter pairs included $\Phi(x) \in \mathbb{R}^k$18 for Feature Hashing LSH with $\Phi(x) \in \mathbb{R}^k$19 and $\Phi(x) \in \mathbb{R}^k$20, and $\Phi(x) \in \mathbb{R}^k$21 for Directional Feature Hashing with $\Phi(x) \in \mathbb{R}^k$22; the paper reports that Feature Hashing LSH is the fastest at the same precision/recall target, with Directional Feature Hashing and Hyperplane LSH next fastest (Argerich et al., 2017).

6. Limitations, trade-offs, and terminology

The principal limitation of classical additive feature hashing is collision noise. When $\Phi(x) \in \mathbb{R}^k$23 or $\Phi(x) \in \mathbb{R}^k$24 is too small, unrelated features share buckets and contaminate one another’s coordinates. The sign hash removes systematic bias by making collisions cancel in expectation, but the residual variance decreases only as the embedding dimension grows. Theoretical guarantees therefore depend not only on $\Phi(x) \in \mathbb{R}^k$25, $\Phi(x) \in \mathbb{R}^k$26, and $\Phi(x) \in \mathbb{R}^k$27, but also on how concentrated the input vector is through the ratio $\Phi(x) \in \mathbb{R}^k$28; heavy coordinates are intrinsically harder to handle in the one-nonzero-per-column regime (Freksen et al., 2018).

Hash2Vec inherits these general issues and adds domain-specific ones. The paper notes sensitivity to the embedding dimension $\Phi(x) \in \mathbb{R}^k$29 and the window size $\Phi(x) \in \mathbb{R}^k$30: too small $\Phi(x) \in \mathbb{R}^k$31 increases collision noise, while too large $\Phi(x) \in \mathbb{R}^k$32 can add topical noise. High-frequency function words can dominate co-occurrence counts, so stopword removal or down-weighting is helpful. Polysemy remains unresolved because the method learns a single vector per word, and the basic construction does not compute PMI or apply GloVe’s weighting function; it is instead presented as a fast compressed surrogate that can, if desired, feed downstream matrix factorization (Argerich et al., 2016).

Dense AFH avoids index sharing but is not noise-free. Its cross-terms are small random inner products rather than explicit bucket collisions, and these terms cancel only in expectation. The trade-off is computational rather than combinatorial: generating and adding dense random vectors costs $\Phi(x) \in \mathbb{R}^k$33 per instance, which can be heavy relative to sparse HT’s $\Phi(x) \in \mathbb{R}^k$34 updates, especially in pipelines optimized for sparse data. The paper also notes that insufficiently independent hash outputs can create dependencies, and recommends cryptographic hashes such as SHAKE-256 to mitigate this (Andrecut, 2021).

The terminology is therefore not entirely uniform. In some literature, “additive feature hashing” denotes the classical hashing trick itself, emphasizing that colliding features are summed with random signs; in the 2021 paper, the same phrase denotes a dense additive alternative based on summing feature-specific random vectors. A common source of confusion is to treat these as identical constructions. They are instead closely related similarity-preserving embeddings that use different mechanisms: sparse signed index collisions in the classical case, and dense random-direction summation in the later case. This suggests that the phrase is best understood as naming a design principle—hash-determined additive superposition—rather than a single fixed algorithm (Jagadeesan, 2019, Andrecut, 2021).