Semantic ID Prefix-Ngram

• Semantic ID Prefix-ngram is a hierarchical tokenization methodology that transforms raw IDs into multi-level discrete codes derived from clustered content embeddings.
• It unifies memorization and generalization by summing prefix-based embeddings, thereby enhancing performance for tail and cold-start items.
• Empirical evaluations demonstrate improved embedding stability, reduced variance, and efficient scaling in large-scale, cross-domain recommender systems.

Semantic ID prefix-ngram is a hierarchical tokenization methodology for representing discrete entities such as items in large-scale recommendation systems. It replaces random or raw ID-based embeddings with semantically meaningful, multi-level discrete codes derived from clustered content embeddings. By constructing and summing embeddings for prefix sub-sequences of these codes, the approach unifies memorization and generalization, providing strong modeling advantages for tail and cold-start items, improved embedding stability under drift, and efficient scaling to extremely large cardinality domains (Zheng et al., 2 Apr 2025, Hu et al., 11 Nov 2025, Singh et al., 2023).

1. Motivation and Theoretical Foundations

Traditional ID-based models in recommender systems face fundamental challenges: extremely high item cardinality, highly skewed engagement distribution (head-tail), and dynamic churn of IDs over time. Approaches based on random hashing induce unstructured collisions, leading to the corruption of semantic signals, poor knowledge transfer to infrequent or novel items, and instability in learned representations as items appear or disappear.

Semantic ID prefix-ngram addresses these issues by tokenizing each item according to its position in a hierarchical clustering over content embeddings. This forms a path in a tree, with hierarchical granularity at each prefix length: coarse clusters at shallow prefixes, fine-grained distinctions at deeper prefixes. Prefix-ngram tokenization is tightly coupled to the learned semantic hierarchy, such that "collisions" (multiple items mapping to the same embedding) are semantically meaningful, enabling knowledge sharing among similar items and robust generalization (Zheng et al., 2 Apr 2025, Singh et al., 2023).

2. Formal Construction and Tokenization Mechanism

Let $x \in \mathbb{R}^d$ denote an item's content embedding. An RQ-VAE (Residual-Quantized Variational Autoencoder) with $L$ layers and codebook size $K$ per layer is trained:

• For $l = 1, \ldots, L$:
  • $c_l = \arg\min_k \| v_k^{(l)} - r_l \|_2$
  • $r_{l+1} = r_l - v_{c_l}^{(l)}$
• $c(x) = (c_1, c_2, ..., c_L) \in \{0, ..., K-1\}^L$ forms an item's Semantic ID.

For prefix-ngram tokenization with maximum depth $n \leq L$, each item $x$ is mapped to $G=n$ tokens:

$$t_i(x) = \sum_{j=1}^i [K^{i-j} \cdot (c_j + 1)] - 1,\,\,i=1,\ldots,n$$

These tokens act as unique identifiers for each hierarchy prefix. Embedding lookup proceeds by a sum-pool over these tokens:

$e(x) = \sum_{i=1}^n E_{f_i(x)}$

where $E$ is the global embedding table, and $f_i(x)$ hashes $t_i(x)$ into a partitioned table or via modulo-hashing with disjoint offsets (Zheng et al., 2 Apr 2025, Singh et al., 2023).

In the cross-domain recommendation setting (Hu et al., 11 Nov 2025), a similar RQ-VAE quantization is performed with domain-specific and universal codebooks, optionally fused via a gating network per item, then the resulting code sequence is encoded as prefix-ngrams via domain-aware prefix-trees.

3. Embedding Stability, Generalization, and Representation Learning

Semantic ID prefix-ngram is designed to mitigate instability and drift, which are inherent to ID churn and evolving content. Because item tokens are clustered semantically, items retiring or arriving only alter leaves or shallow tree branches, ensuring that embedding re-use between different items retains semantic consistency. Knowledge transfer for long-tail and cold-start items is fostered by coarse prefixes, which aggregate data across semantically similar but infrequent entities.

Empirically, prefix-n-gram aggregation is superior to flat n-gram or random hashing approaches due to hierarchical smoothing: summing over all prefixes leverages both global and local information, balancing the memorization for frequent items (deep prefixes) and generalization for rare items (shallow prefixes). SentencePiece tokenization further compresses the code sequence representation but loses explicit hierarchical structure ("n-grams vs. SPM" in (Singh et al., 2023)):

Method	Table Size	Head/Long Tail Balance
Unigram-SID ($K$)	$K \cdot d$	Only broad, poor memorization
Bigram-SID ($K^2$)	$K^2 \cdot d$	Finer, may overfit, gigantic table
Prefix-n-gram ($n$)	$\sum_{k=1}^n K^k \cdot d$ (hashed)	Hierarchical blend, scalable
SPM (vocab $V$)	$V \cdot d$	Data-driven, non-hierarchical, adaptive

Prefix-n-gram hashing optimally balances representation capacity and parameter size, supporting both tail generalization and head memorization without the high-dimensional sparsity penalty of pure n-grams (Singh et al., 2023, Zheng et al., 2 Apr 2025).

4. Integration with Recommender System Architectures

Semantic ID prefix-ngram integrates directly into DLRM-style and attention-based architectures by replacing each raw item or sequence of items with their corresponding prefix-ngrams:

• Sparse module performs sum-pooling of prefix-ngram embeddings.
• Sequential recommendation models (Transformers, PMA) process user histories encoded by these embeddings.
• The attention mechanism benefits from lower entropy and more informative token patterns, focusing on high-signal user interactions with less reliance on padding or self-loops (Zheng et al., 2 Apr 2025).

In cross-domain generative models, prefix-trees (tries) of valid semantic ID prefixes are constructed per domain, supporting constrained autoregressive semantic generation. During inference, valid continuations are efficiently enumerated, reducing search space and eliminating hallucinated or syntactically invalid code sequences (Hu et al., 11 Nov 2025).

5. Empirical Evaluation and Performance

Large-scale experiments demonstrate that prefix-ngram tokenization delivers substantial gains in recommendation quality, stability, and computational efficiency:

• Offline token-param ablation (Meta Ads Ranking): prefix-6gram achieves a −0.215% Train NE improvement over random hashing, with increasing depth showing monotonic gains (Zheng et al., 2 Apr 2025).
• Segment-level analysis reveals robust gains for tail items (−0.04% NE) and cold-starts (−0.41% NE).
• Long-term retention and data scaling: prefix-n-grams sustain or improve performance as historical data grows.
• User-history modeling: in combination with bypass, Transformer, or PMA modules, prefix-ngrams yield up to a −0.110% Eval NE improvement.
• Production deployments (Meta Ads Ranking): six sparse semantic ID features deliver −0.071% Eval NE offline, and live A/B tests demonstrate ≈ 0.15% top-line online NE gain.
• Variance reduction: A/A experiments show 43% lower prediction variance for identical input pairs, indicating enhanced representation stability.
• Click-loss under swap: as more prefix codes are shared between swapped items, the CTR loss rate decreases monotonically, validating that semantic token collisions facilitate knowledge transfer (Zheng et al., 2 Apr 2025).
• Cross-domain GenCDR with domain-aware prefix-tree decoding achieves state-of-the-art NDCG@10 and reduced inference cost, with stable memory footprint independent of the candidate item pool size (Hu et al., 11 Nov 2025).

6. Algorithmic Trade-offs and Limitations

• RQ-VAE codebook and embedding table sizes must be carefully chosen. Increasing the number of quantization levels ($L$) or codebook size ($K$) increases semantic granularity but expands table size; hashing or partitioning techniques are used to scale to production settings (Zheng et al., 2 Apr 2025, Singh et al., 2023).
• Training the RQ-VAE requires up-to-date multimodal content embeddings; codebooks may "drift" as item semantics or catalogs evolve, necessitating periodic retraining.
• Excessive prefix length (deep trees) can result in over-partitioning, leading to insufficient training per token and overfitting. Shallow prefixing may lose discriminative power.
• Construction and maintenance of prefix-trees is algorithmically straightforward, but requires support for fast traversal and efficient storage.
• Integration of behavioral signals (e.g., engagement-driven clustering) or dynamic codebook growth remains a subject of future research. The framework is extensible to candidate generation and other forms of sparse feature tokenization, such as for user IDs or categories (Zheng et al., 2 Apr 2025, Hu et al., 11 Nov 2025, Singh et al., 2023).

7. Comparative Insights and Research Outlook

Relative to traditional random or surface n-gram hashing, Semantic ID prefix-ngram introduces a principled hierarchical grouping that enables controlled, semantically meaningful embedding collisions. The design best utilizes both the expressiveness of learned content embedding space and the need for scalable parameterization:

• Hierarchical prefixing outperforms flat n-gram partitioning, as shown in attention metric ablations and intra-cluster variance analysis.
• Prefix-ngrams maintain valid semantic code sequences, avoid invalid or hallucinated n-grams, and reduce vocabulary explosion.
• Cross-domain settings benefit from adaptive domain-specific tokenization with universal and per-domain semantic adapters, leveraging prefix-trees for fast and accurate generative decoding (Hu et al., 11 Nov 2025).

Semantic ID prefix-ngram, as substantiated by empirical deployments and public benchmarks, addresses the axes of cardinality, drift, and tail modeling, integrates with sequential and cross-domain systems, and enables scalable, robust, and semantically aware recommendation modeling (Zheng et al., 2 Apr 2025, Hu et al., 11 Nov 2025, Singh et al., 2023).