Semantic Tokenization in Generative Recommendation

• Semantic tokenization is a method that transforms rich, multimodal, and collaborative signals into discrete tokens capturing item semantics.
• It employs advanced techniques such as quantization (RQ-VAE), contrastive, hierarchical, and multimodal approaches to improve candidate retrieval.
• These methods boost system performance by enhancing retrieval quality, efficiency, and scalability while addressing cold-start challenges.

Semantic tokenization in generative recommendation refers to the transformation of item representations—often derived from rich textual, multimodal, or collaborative signals—into sequences or sets of discrete tokens that capture semantic, behavioral, and sometimes hierarchical item characteristics. These semantic tokens serve as the foundation for generative models to perform candidate retrieval not by matching continuous embeddings or static IDs, but by directly generating item token sequences conditioned on user interactions. This paradigm shift enables more efficient, expressive, and generalizable recommender systems, as evidenced by a large and rapidly evolving body of research.

1. Foundations and Historical Context

Semantic tokenization emerged as a critical enabler for generative recommendation, a paradigm where next-item recommendation or retrieval is cast as an autoregressive sequence generation problem. Early methods used unique item identifiers as model output tokens, which limited generalization and failed to incorporate item semantics or facilitate efficient modeling of cold-start scenarios (Liu et al., 11 Sep 2024). The introduction of semantic tokenization reframed item representation: items are encoded as discrete token sequences or sets, derived by quantization of semantic-rich, often multimodal embeddings. These tokens form a reduced, more structured vocabulary reflecting underlying item properties, rather than opaque ID spaces.

The field has progressed from unsupervised, content-reconstruction-driven tokenizers (e.g., RQ-VAE, product quantization) (Ju et al., 29 Jul 2025) to advanced frameworks that integrate contrastive, hierarchical, contextual, and cross-modal techniques (Zhu et al., 23 Apr 2024, Zhai et al., 20 Jun 2025, 2510.22361). This evolution reflects a growing consensus: effective semantic tokenization is indispensable for generative recommenders to learn collaborative filtering signals, model item relationships, handle large-scale catalogs, and generalize to unseen or low-frequency items.

2. Methodologies for Semantic Tokenization

Quantization-Based Tokenization

Residual quantization and variational autoencoders (RQ-VAE) form the backbone of most early semantic tokenization pipelines (Zhu et al., 23 Apr 2024, Ju et al., 29 Jul 2025). Items, embedded via pretrained language or multimodal models, are recursively quantized across levels, each level providing a codeword (token) from a learnable codebook. Token assignments minimize Euclidean or cosine distance between the current residual and codebook entries, producing a tuple $[c_0, c_1, ..., c_{m-1}]$ representing the item's semantic code.

Formally, in RQ-VAE:

• At level $d$, residual $r_d$ is quantized as $c_d = \mathrm{argmin}_e \| r_d - e \|$,
• Residual update: $r_{d+1} = r_d - e_{c_d}$,
• The reconstruction is $\hat{z} = \sum_{d=0}^{m-1} e_{c_d}$.

Contrastive Tokenization

Contrastive learning objectives are incorporated to encourage code assignments that reflect not just per-item content reconstruction but also discriminative power across the item population (Zhu et al., 23 Apr 2024, Zhai et al., 20 Jun 2025). InfoNCE or NT-Xent losses are applied, where positive pairs (an item's original and reconstructed embeddings) are pulled together, while negatives (other items) are pushed apart:

$$\mathcal{L}_{cl} = - \log \frac{\exp(\langle x_0, \hat{x}_0 \rangle / \tau)}{\sum_{j=0}^K \exp(\langle x_0, \hat{x}_j \rangle / \tau)}$$

Contrastive tokenization, as in CoST (Zhu et al., 23 Apr 2024) and SimCIT (Zhai et al., 20 Jun 2025), captures both semantic and relational item information, improving representational quality for downstream generative retrieval.

Hierarchical and Structured Tokenization

Hierarchical quantization is employed to encode taxonomy, attribute, or content structure. HiD-VAE (Fang et al., 6 Aug 2025) supervises each quantization layer with corresponding semantic tags (e.g., category, subcategory), enforcing coarse-to-fine traceability in the codes. Uniqueness losses penalize code collisions, promoting interpretable, disentangled, and diversity-preserving item IDs.

Mixture-of-Experts and Multimodal Approaches

MMQ (Xu et al., 21 Aug 2025) introduces a multi-expert quantization scheme in which modality-shared and modality-specific experts operate in parallel to capture both cross-modality synergy and uniqueness. Orthogonal regularization ensures non-redundancy among experts, while behavior-aware fine-tuning dynamically adapts semantic IDs to reflect user interaction signals, bridging the semantic-behavioral gap.

Order-Agnostic and Parallel Tokenization

SETRec (Lin et al., 15 Feb 2025) and RPG (Hou et al., 6 Jun 2025) realize token sets/long unordered tuples to overcome the sequential dependency and inefficiency of autoregressive generation. Sparse attention masks and query-guided mechanisms enable simultaneous prediction of all identifier tokens, yielding significant gains in efficiency and scalability.

3. Integration with Generative Recommendation Models

Semantic tokenization constitutes the first phase in a two-stage generative recommendation pipeline: (1) mapping item content or collaborative features into token sequences/sets; (2) training a generative model (e.g., Transformer encoder-decoder) to produce target item tokens given user history.

Several frameworks integrate semantic tokenization with generation:

• End-to-End Optimization: ETEGRec (Liu et al., 9 Sep 2024) couples the tokenizer and recommender via aligned losses (sequence-item, preference-semantic), optimized alternately for stability.
• Unified Models: STORE (Liu et al., 11 Sep 2024) uses a single LLM backbone for both text-to-token (semantic tokenization) and token-to-token (generative recommendation), reducing system complexity.
• Disentangled Fusion: DiscRec (Liu et al., 18 Jun 2025) employs dual-branch modules, with semantic and collaborative signals processed in parallel and adaptively fused, leveraging item-level position embeddings to address token–item misalignment.

Auxiliary techniques include curriculum learning using multi-identifier tokenizations for pretraining (Zheng et al., 6 Apr 2025), universal tokenizers for cross-domain transfer (Zheng et al., 6 Apr 2025), and chunk-level “act-with-think” modeling that unifies semantic explanation with behavioral prediction (Wang et al., 30 Jun 2025).

4. Impact on Performance and Scalability

Empirical evidence consistently demonstrates that advanced semantic tokenization methods substantially improve both retrieval quality and system efficiency:

Model/Method	Key Dataset	Notable Improvement	Tokenization Features
CoST (Zhu et al., 23 Apr 2024)	MIND	+43.7% NDCG@5	Contrastive quantization over RQ-VAE
SETRec (Lin et al., 15 Feb 2025)	4 Amazon domains	8–18× faster inference	Order-agnostic set tokenization
RPG (Hou et al., 6 Jun 2025)	Sports/Beauty/etc.	+12.6% NDCG@10, 15× speedup	Parallel long semantic IDs
HiD-VAE (Fang et al., 6 Aug 2025)	Beauty/Sports/etc.	+35% Recall@5 (Beauty)	Hierarchical supervised codes, de-colliding
SimCIT (Zhai et al., 20 Jun 2025)	AMap/POI	+16.2% Recall@10 (AMap)	Multi-modal contrastive quantization
MMQ (Xu et al., 21 Aug 2025)	E-commerce	+0.9% revenue, +4.33% Conv	Multimodal, multi-expert, behavior-adaptive

These gains arise from enhanced code expressiveness, robust generalization to cold-start/long-tail items, and reduced computation/memory via compact token representations. The order-agnostic and parallel generation paradigms further reduce latency bottlenecks associated with autoregressive decoding.

5. Challenges, Objective Alignment, and Recent Advances

A recurring challenge is aligning the objectives and semantics of separately trained tokenizers and generation models. As identified in DECOR (Liu et al., 22 Aug 2025), static tokenization optimized for semantic reconstruction can yield suboptimal, context-insensitive tokens when used in collaborative, context-driven generative modeling. DECOR proposes decomposed embedding fusion (combining frozen codebook embeddings with collaborative representations) and contextualized token composition (dynamically refining token meaning with attention over candidate codes and historical context), directly addressing the semantic–collaborative objective gap.

Additional advances focus on:

• Contextual Tokenization: ActionPiece (Hou et al., 19 Feb 2025) employs context-aware BPE-style token merging and set permutation regularization to reflect contextual and unordered feature sets, improving robustness and capturing nuanced item-action semantics.
• Pure Semantic Indexing: Unique, non-colliding semantic IDs are produced using relaxed candidate selection algorithms (ECM/RRS (Zhang et al., 19 Sep 2025)), avoiding non-semantic conflict-resolution tokens and yielding improvements in both overall and cold-start settings.
• Explainability and Interpretable Paths: HiD-VAE (Fang et al., 6 Aug 2025)’s hierarchical supervision enables each code token to be mapped to human-understandable item tags, facilitating transparency and control in recommendation explanations.

6. Multimodal, Universal, and Joint Task Extensions

Semantic tokenization research increasingly spans multimodal (text, images, spatial information), multilingual, and cross-task (search, retrieval, recommendation) settings:

• Multimodal Tokenization: UTGRec (Zheng et al., 6 Apr 2025) and MMQ (Xu et al., 21 Aug 2025) employ tree-structured codebooks and multi-expert architectures to encode and reconstruct content across modalities, ensuring token transferability and robustness.
• Joint Search and Recommendation: Discretizing joint embedding spaces (via RQ-KMeans, SVD fusion) supports unified semantic IDs effective across both search and recommendation (Penha et al., 14 Aug 2025).
• Generalization and Transfer: Universal/tokenizer approaches are empirically validated to outperform domain-specific solutions on new domains, supporting the design of scalable, adaptive generative recommender architectures (Zheng et al., 6 Apr 2025).

7. Practical Implementation Considerations

Implementation of semantic tokenization frameworks involves:

• Selecting/pretraining strong content or multimodal encoders (e.g., Sentence-T5, BERT).
• Designing and training hierarchical, contrastive, or multi-expert quantization modules or autoencoders.
• Aligning or fusing pretrained codebook (semantic) embeddings with dynamic, learnable collaborative embeddings using strategies such as decomposed fusion (Liu et al., 22 Aug 2025) or dual-branch gating (Liu et al., 18 Jun 2025).
• Constructing inference pipelines that enable conditional, constrained, or parallel token generation, beam search with ID-tree verification, or graph-based decoding for valid, efficient lookup (Hou et al., 6 Jun 2025).
• Integrating auxiliary losses and curriculum/data sampling mechanisms—e.g., multi-identifier pretraining routines, modality-aligned reconstruction, and transfer-aware adaptation.
• Ensuring code assignment uniqueness and avoiding semantic collisions in large-scale settings via candidate matching or recursive assignment algorithms (Zhang et al., 19 Sep 2025).

A plausible implication is that continued progress will require careful balancing of semantic fidelity, discriminative expressiveness, computational efficiency, and code interpretability. Unified and disentangled modeling of collaborative and semantic signals, such as in discRec (Liu et al., 18 Jun 2025) and DECOR (Liu et al., 22 Aug 2025), is likely to remain an active area for innovation.

---

Semantic tokenization has become foundational for generative recommendation owing to its ability to furnish expressive, compact, and generalizable representations of items. The field continues to advance rapidly in algorithmic sophistication, system architecture integration, and empirical performance, with implications for scalable, explainable, and cross-domain recommendation systems.