Probabilistic Hash Embedding (PHE)

• PHE is a probabilistic method that combines multiple hash functions with Bayesian online learning to efficiently embed evolving categorical data.
• It maintains fixed memory usage and avoids catastrophic forgetting by using a mean-field variational approximation for posterior inference.
• Empirical evaluations show that PHE outperforms traditional one-hot and deterministic hash methods in online classification, sequence modeling, and recommendation tasks.

Probabilistic Hash Embedding (PHE) is a method for compact, adaptive learning of categorical feature embeddings in online and streaming data settings with unbounded or evolving vocabularies. PHE integrates the bounded-memory efficiency of classical feature hashing with Bayesian approaches, enabling streaming models to adapt to new categories without catastrophic forgetting and with provable invariance to the order of data arrival. Memory usage is fixed and independent of the number of unique categories observed.

1. Motivation and Limitations of Traditional Methods

In online applications such as recommendation systems and anomaly detection, categorical features frequently arise, often with vocabularies that expand over time. Classical strategies for categorical embedding include one-hot encoding—where each category $s\in\mathcal S$ is assigned its own vector row in matrix $W\in\mathbb R^{|\mathcal S|\times d}$—and deterministic feature hashing (the “hashing trick”), where $h:\mathcal S\to\{0,1,\dots,B-1\}$ maps feature values to a fixed-size table $E\in\mathbb R^{B\times d}$, i.e., $\phi(s)=E_{h(s)}$.

Although feature hashing bounds memory usage by $B\,d$, it introduces collisions, where distinct categories $s\ne s'$ are mapped to the same row $h(s)=h(s')$. In streaming data, this leads to parameter interference: updates for one category may irreversibly degrade the embedding for another, and resulting models depend on the arrival order of the data. One-hot encoding, while collision-free, retains unbounded memory growth as $\mathcal S$ grows. Empirically, unless learning schedules for hashed embeddings are hand-tuned, new categories lead to rapid “forgetting” of prior information, and neither classical strategy provides the invariance or resilience required for robust online adaptation (Li et al., 25 Nov 2025).

2. Probabilistic Model Architecture

PHE constructs the embedding table $E$ as a random object, specifically a collection of independent Gaussian random variables:

$$E_{b,j}\sim\mathcal{N}(0,1),\quad b=0,\ldots,B-1,\;j=1,\dots,d.$$

To further reduce collision probability, PHE employs $K$ independent hash functions $\{h^{(1)},\dots,h^{(K)}\}$, each mapping the categorical domain to bucket indices.

Given a feature $s$, bucket indices $z^{(k)}=h^{(k)}(s)$ are deterministically selected. The $K$ row-vectors $E_{z^{(k)},:}$ are pooled (e.g., via weighted summation $E_s = \sum_{k=1}^K w_k E_{z^{(k)},:}$) to yield the embedding for $s$. Weights $w_k$ may themselves be hashed and learned. Observed targets $y$ are generated according to a suitable likelihood $p_\theta(y\mid E_s)$, such as categorical for classification ($\mathrm{Cat}(\mathrm{softmax}(W E_s))$) or Gaussian for regression.

This probabilistic construction enables PHE to marry the statistical regularization of Bayesian models with hash-based memory constraints.

3. Bayesian Online Learning and Posterior Inference

The objective in streaming settings is to maintain a posterior over the embeddings,

$$p(E\mid\mathcal D_t)\propto p(E)\prod_{i=1}^t p_\theta(y_i\mid E_{s_i}),$$

where $\mathcal D_t$ is the observed history. Exact inference is intractable, so PHE uses a mean-field variational approximation:

$$q_\lambda(E)=\prod_{b=0}^{B-1}\prod_{j=1}^d \mathcal{N}(E_{b,j};\,\mu_{b,j},\sigma_{b,j}^2),$$

with $\lambda=\{\mu_{b,j},\sigma_{b,j}\}$.

• Batch (Offline) ELBO:

$$\mathcal{L}(\lambda,\theta)=\mathbb{E}_{q_\lambda(E)}\left[\sum_{i=1}^N\log p_\theta(y_i\mid E_{s_i})\right] - \mathrm{KL}\bigl[q_\lambda(E)\|p(E)\bigr].$$

• Online ELBO Updates:

Upon receiving new data $\mathcal D_{t+1}$, the prior $p(E)$ is replaced by the previous approximate posterior $q_{\lambda_t}(E)$:

$$\mathcal{L}_{t+1}(\lambda)=\mathbb{E}_{q_\lambda(E)}\left[\sum_{(s,y)\in\mathcal D_{t+1}}\log p_\theta(y\mid E_s)\right]-\mathrm{KL}[q_\lambda(E)\Vert q_{\lambda_t}(E)].$$

This architecture keeps the parameter count fixed at $2B\,d$ (means and variances for each entry in $E$), regardless of streaming vocabulary growth.

4. Theoretical Guarantees and Properties

PHE provides several strong theoretical properties:

• Bounded Memory: The model requires $O(B\,d)$ parameters for means and $O(B\,d)$ for variances, totaling $2B\,d$, independent of the number of distinct observed categories. In contrast, one-hot embedding tables require $O(|\mathcal S|\,d)$.
• Arrival-Order Invariance: Bayesian updating ensures that, in principle, the inference posterior is permutation invariant:

$$p(E\mid\{(s_i,y_i)\}_{i=1}^N)=p(E\mid \mathcal D_{\pi(1)},\dots,\mathcal D_{\pi(N)}),$$

for any permutation $\pi$. The PHE variational inference preserves this property in practice.

• Collision Probability: Using $K$ hash functions, the probability of collision is $O(B^{-K})$.
• No Catastrophic Forgetting: By maintaining a posterior over bucket embeddings rather than deterministic point estimates, PHE avoids the destructive parameter overwriting observed in deterministic hash embeddings.
• Learning-Rate Robustness: The model does not require hand-tuned learning-rate schedules for new categories.

5. Empirical Evaluation

PHE has been benchmarked on multiple online learning tasks:

• Online Classification: On UCI tabular datasets (Adult, Bank, Mushroom, Covertype), PHE matches or slightly exceeds the performance of collision-free expandable embeddings (EE, which have unbounded memory), outperforming deterministic Ada-hash baselines by 2–5 percentage points, and using only 9–62% of EE’s parameter count.
• Multi-task Sequence Modeling: On the Retail dataset (4,000 products with Deep Kalman Filters), PHE reduces mean absolute error (MAE) from 3.7 (EE baseline) to 3.0, surpassing Ada-hash variants, while requiring only 2% of EE’s parameters.
• Large-Scale Recommendation: On MovieLens-32M (200K users, 87K movies), PHE achieves 14.7 MAE, matching the Bayesian expandable embedding (P-EE, with unbounded memory), using only 4% of P-EE’s parameters and outperforming Ada-hash variants (15.1–15.3 MAE).

Across all tasks, PHE demonstrates memory efficiency, invariance to data order, and accuracy equivalent to collision-free embeddings, but with strict memory bounds (Li et al., 25 Nov 2025).

Task	Metric	PHE Performance	Baseline(s)	Memory Ratio vs EE
Online classification	Acc (%)	Matches/exceeds EE, +2-5pt vs Ada	EE, Ada-hash	9–62%
Sequence modeling	MAE	3.0 (vs 3.7 for EE)	EE, Ada-hash	2%
Large-scale recommendation	MAE	14.7 (equal to P-EE, < Ada-hash)	P-EE (collision-free), Ada-hash	4%

6. Practical Implications and Use Cases

PHE is suitable for any deployment that must learn with unbounded, dynamically evolving categorical feature sets, especially in streaming and non-stationary data scenarios. Its invariance to input order, robustness to interference, and fixed resource footprint enable usage in online recommendation, anomaly detection, and large-scale sequence modeling where previous approaches are impractical either due to catastrophic forgetting (hash-based) or unbounded memory overhead (one-hot, expandable tables).

A plausible implication is that PHE serves as a simple drop-in replacement for any model requiring categorical embeddings under streaming conditions where vocabulary sizes cannot be predefined or capped (Li et al., 25 Nov 2025).

7. Relationship to Related Methods

PHE generalizes and subsumes prior approaches by combining feature hashing with Bayesian variational inference. It improves upon deterministic hash embeddings by preventing destructive interference and eliminating arrival-order dependence. In comparison with expandable embeddings—both deterministic (EE) and Bayesian (P-EE)—PHE achieves comparable or superior predictive performance in online settings while using a fraction (2–4%) of the memory footprint. These attributes are realized without the need for dynamic model parameter growth or complex hyperparameter tuning (Li et al., 25 Nov 2025).

Empirically, PHE’s ability to retain old knowledge, adapt to sudden vocabularic drift, and remain agnostic to learning-rate schedules marks a substantive advancement in online learning architectures for categorical data.