Feature Hashing in Machine Learning

• Feature hashing is a technique that maps high-dimensional data into lower-dimensional space using hash and sign functions, preserving key geometric properties.
• It enables efficient, one-pass computation on sparse data, making it ideal for large-scale NLP, multitask learning, and streaming applications.
• The method delivers theoretical guarantees on norm and inner-product preservation, with practical trade-offs managed by hash function choice and parameter tuning.

Feature hashing, also known as the "hashing trick," is a randomized technique for mapping high-dimensional or categorical feature spaces into lower-dimensional, fixed-size vector spaces through the use of hash functions. This method enables practitioners to construct compact, computationally efficient representations while approximately preserving pairwise distances and inner products. Feature hashing is widely adopted in large-scale machine learning pipelines, natural language processing, multitask learning, and approximate nearest neighbor search, among other applications.

1. Formal Definition and Construction

Feature hashing defines a projection from a high-dimensional feature vector $x \in \mathbb{R}^D$ to a lower-dimensional vector $y \in \mathbb{R}^m$ using two independent hash functions:

• Index hash $h : \{1, ..., D\} \to \{1, ..., m\}$ assigns each original feature to one of $m$ bins.
• Sign hash $\xi : \{1, ..., D\} \to \{\pm 1\}$ assigns a random sign to each feature for unbiasedness.

The embedding is defined by: $$y_i = \sum_{j: h(j) = i} \xi(j) \cdot x_j, \quad \forall i = 1, ..., m.$$ In matrix form, $y = \Phi x$, where $\Phi \in \mathbb{R}^{m \times D}$ has exactly one non-zero entry per column, equal to $\pm 1$ at row $h(j)$ as chosen by the sign hash. This approach preserves sparsity, as $\|\Phi x\|_0 \leq \|x\|_0$ (0902.2206, Freksen et al., 2018).

A variant with sparsity $s > 1$ per column, as in sparse Johnson–Lindenstrauss (JL) transforms, chooses for every feature index $j$ a set of $s$ distinct hash locations and signs, and distributes $x_j$ equally (with scaling $1/\sqrt{s}$) among those bins (Jagadeesan, 2019).

2. Algorithmic Procedures and Streaming Implementation

Feature hashing is highly amenable to streaming and online computation. For a sparse vector $x$ or a sequence of input tokens, the algorithm iterates through non-zero features:

• For each active feature $j$, compute $i = h(j)$ and sign $\sigma = \xi(j)$.
• Increment $y_i$ by $\sigma \cdot x_j$.

The time complexity per input instance is $O(\mathrm{nnz}(x))$, requiring only $O(m)$ memory for the hashed vector. This process eliminates the need for constructing and storing an explicit vocabulary or feature dictionary and is trivially parallelizable (0902.2206, Argerich et al., 2016).

In NLP, the Hash2Vec approach constructs word embeddings by aggregating context features with distance-dependent weights: $$v(w)_i = \sum_{(c_w, d) \in F(w)} \xi(c_w) \cdot f(d) \cdot 1_{[h(c_w) = i]},$$ where $F(w)$ is the multiset of observed word-context pairs and $f(d)$ is a window-dependent weighting function (e.g., Gaussian decay or uniform) (Argerich et al., 2016).

3. Theoretical Guarantees and Statistical Properties

Feature hashing admits rigorous guarantees based on concentration inequalities for random projections:

• Norm and inner-product preservation: For $x$ with bounded $\|x\|_\infty/\|x\|_2$ (i.e., not overly sparse), feature hashing preserves $\|x\|_2$ and inner products up to $(1 \pm \epsilon)$ distortion with exponentially small failure probability, provided $m = O(\epsilon^{-2} \log(1/\delta))$ (0902.2206, Freksen et al., 2018, Dahlgaard et al., 2017).
• Bias and variance: The mapping is unbiased: $$\mathbb{E}_{h, \xi}[\langle \Phi x, \Phi x'\rangle] = \langle x, x'\rangle$$. The variance of $\langle \Phi x, \Phi x' \rangle$ scales as $O(1/m)$ for well-distributed $x$ (0902.2206, Freksen et al., 2018).
• Tightness and tradeoffs: The distortion guarantees are tight up to constant factors. With $m \ge 2/(\epsilon^2 \delta)$, arbitrary input vectors are preserved. For smaller $m$, the maximum tolerated $\|x\|_\infty/\|x\|_2$ decreases, with explicit formulae for the threshold as a function of $m, \epsilon, \delta$ (Freksen et al., 2018, Jagadeesan, 2019).

Increasing the sparsity parameter $s$ above $1$ in the JL matrix improves concentration, tolerating larger ratios $\|x\|_\infty/\|x\|_2$ for the same $m$ (Jagadeesan, 2019).

4. Practical Hash Function Choices and Empirical Effect

Multiple hash function families are deployable in practice, with significant empirical and theoretical consequences:

• Multiply-mod-prime: 2-wise independent, extremely fast, but can exhibit large bias and heavy tails for structured input; not recommended when input is adversarial or highly regular (Dahlgaard et al., 2017).
• MurmurHash3: Fast, widely used, empirically near-random but lacks provable guarantees (Dahlgaard et al., 2017).
• Mixed Tabulation: Provably behaves like a truly random hash for feature hashing, zero bias and optimal tail bounds, and is 40% faster than MurmurHash3 while empirically matching fully random performance (Dahlgaard et al., 2017).

In large-scale benchmarks (News20, MNIST, synthetic data), mixed tabulation and MurmurHash3 provide tight concentration and norm/inner-product preservation, unlike multiply-mod-prime, which may yield severe outliers (Dahlgaard et al., 2017).

5. Applications: NLP, Multitask Learning, and Approximate Nearest Neighbor Search

Feature hashing supports a range of major applications:

• Word Embeddings: Hash2Vec applies feature hashing to derive word vectors directly from context windows in text, with performance (Spearman correlation against human similarity ratings) approaching GloVe when $D$ is in the $500$–$1000$ range (Argerich et al., 2016).
• Multitask Learning: In large-scale multitask learning (e.g., per-user spam filtering with $|U| \sim 4 \times 10^5$ users), feature hashing enables compact model storage via a single aggregate vector, with quantified negligible interference under proper hashing (0902.2206).
• Locality-Sensitive Hashing (LSH): Feature hashing is a sparse Johnson–Lindenstrauss projection and serves as a base for two efficient LSH families for angular and Euclidean distances, matching hyperplane and Voronoi LSH in discrimination power but offering up to $10\times$ faster indexing (Argerich et al., 2017).
• Document and Tokenization Tasks: Feature hashing obviates explicit vocabularies and enables high-throughput processing of categorical features in SMS spam detection and language recognition; both standard and additive feature hashing perform similarly, with the latter providing marginally improved separation at high dimension (Andrecut, 2021).

6. Parameter Selection, Strengths, and Limitations

Key guidelines for effective feature hashing configuration:

• Embedding Dimension ($m$ or $D$): For text and NLP tasks, $m$ in the range $300$–$1000$ is typical for high-quality embeddings; for multitask models, $m \sim 10^6$–$10^7$ supports $\epsilon \sim 0.01$–$0.05$ (0902.2206, Argerich et al., 2016).
• Window Size in NLP ($k$): A context window $k=5$–$15$ balances locality and co-occurrence signal in word embedding tasks (Argerich et al., 2016).
• Downweight Frequent Features: Remove or downweight high-frequency elements to satisfy the key $\|x\|_\infty/\|x\|_2$ assumption and minimize collision-induced distortion (Argerich et al., 2016, 0902.2206).
• Hash Function Choice: Prefer mixed tabulation where tight concentration and adversarial robustness are required (Dahlgaard et al., 2017).

Strengths:

• Deterministic, stateless, one-pass computation.
• Memory and time complexity independent of the original feature dimension.
• Parallelizable and MapReduce/Spark-friendly.
• Adaptable to dynamic feature introduction and streaming data (Argerich et al., 2016, 0902.2206).

Limitations:

• Sensitive to very large feature weights (“heavy hitters”). If $\|x\|_\infty \gg \|x\|_2$, more aggressive dimensionality or preprocessing may be needed.
• Preserves only a single prototype—incapable by itself of modeling polysemy or intra-class variance unless extended with multihashing or phrase-level feature construction.
• Collision behavior is controlled but not eliminated, especially at low $m$; sign-hashing is essential for unbiasedness (Argerich et al., 2016, 0902.2206, Freksen et al., 2018).

7. Extensions and Variants

• Additive Feature Hashing: Embeds tokens as high-dimensional random ±1 vectors (via a bitstring hash), then sums token-vectors for final representation. Marginally better at reducing cross-token interference for large $L$, but computationally costlier per token; empirical accuracy is nearly identical in NLP tasks (Andrecut, 2021).
• Sparse JL with $s > 1$: Using more than one nonzero per column improves norm preservation for spiky vectors, with only modest increase in computational cost ($O(s\,\mathrm{nnz}(x))$) (Jagadeesan, 2019).
• Multi-hashing: Splitting frequent tokens into several independently hashed copies reduces variance from heavy hitters but increases overall variance as a tradeoff (0902.2206).
• Feature Hashing for LSH: Two new LSH families—FH Voronoi and Directional FH—enable efficient sublinear approximate nearest neighbor search, with fast random projections and quantifiable performance trade-offs through parameter $k$ (sparsity) (Argerich et al., 2017).

Feature hashing remains an indispensable tool for dimensionality reduction in machine learning, underpinned by rigorous guarantees and validated by large-scale empirical results. Its flexibility, parallelizability, and provable accuracy continue to underpin efficient model construction and deployment in high-dimensional regimes (0902.2206, Freksen et al., 2018, Dahlgaard et al., 2017, Argerich et al., 2016, Andrecut, 2021, Argerich et al., 2017, Jagadeesan, 2019).