RecBase: Generative Rec Model

• RecBase is a generative foundational model that addresses language-centric pretraining limitations by using recommendation-aligned objectives on heterogeneous, cross-domain data.
• Its unified item tokenizer employs hierarchical quantization with curriculum learning to create domain-invariant, discrete item representations.
• Autoregressive sequence modeling in RecBase efficiently captures user interaction dynamics, achieving competitive zero-shot performance and reduced inference latency.

RecBase is a generative foundational model for recommendation tasks, specifically designed to address the limitations of traditional language-centric pretraining in cross-domain and zero-shot recommendation scenarios. The model employs a recommendation-oriented pretraining objective using a large, heterogeneous, cross-domain corpus and introduces a unified item tokenizer to align item semantics across varied domains.

1. Motivation and Conceptual Framework

The foundational motivation of RecBase is the observed mismatch between LLM (LM) pretraining and the requirements of item-centric recommendation tasks. Conventional LLM-based methods rely predominantly on semantic-level language understanding, which is not sufficient for capturing dynamic, item-level user interests and interactions across domains. In contrast, RecBase is constructed to bridge this gap by employing domain-agnostic pretraining, leveraging structured user interaction sequences and unified item representations. Representation discrepancies (where plain text-mapped inputs fail to encode essential item features) and the "knowledge gap" (where classical LMs inadequately model co-occurrence dynamics) inform the RecBase design (Zhou et al., 3 Sep 2025).

2. Model Architecture

RecBase consists of three principal architectural components:

1. Unified Feature Representation: Item textual descriptions are transformed to dense embeddings using models such as NV-Embed. These embeddings are discretized into hierarchical concept identifiers.
2. Unified Item Tokenizer: Employing a hierarchical quantization approach—specifically, Residual Quantized Variational Autoencoder (RQ-VAE), referred to as CL-VAE when augmented with curriculum learning. Given embedding $e$, encoding and quantization proceed as follows:
   • Encoding: $z = \mathcal{E}(e)$
   • Quantization at depth $d$: $c_d = \underset{k}{\arg\min} \|r_d - e_k\|$, $r_{d+1} = r_d - e_{c_d}$
   • Items are thus represented as sequences $s_i = (s_i^1, ..., s_i^m)$
3. Autoregressive Sequence Modeling: A transformer-based model predicts the next item token via joint probability factorization:

$$P(s_t \mid S_{<t}) = \prod_{j=1}^m P(s_t^j \mid s_t^{<j}, S_{<t})$$

This hierarchical encoding scheme enables structured representation and robust vocabulary sharing across domains.

3. Pretraining Methodology

Pretraining is performed on 15 large-scale, heterogeneous datasets spanning domains such as news, e-commerce, videos, books, gaming, and hotel reviews. Each item is processed through:

• Unified text formatting, concatenating attributes and reviews
• Embedding model (e.g., NV-Embed) followed by CL-VAE to generate hierarchical concept IDs
• The transformer is trained to model sequential item transitions, using negative log-likelihood loss:

$$\mathcal{L} = -\sum_{t=1}^n \sum_{j=1}^m \log P(s_t^{j*} \mid s_t^{<j*}, S_{<t})$$

Unlike conventional approaches, explicit negative sampling is omitted in favor of modeling genuine user interaction sequences, enhancing the capture of natural behavioral dynamics.

4. Unified Item Tokenizer: Mechanics and Impact

The unified item tokenizer is central to RecBase, mapping continuous item embeddings to discrete, hierarchical concept IDs. Through curriculum learning, the tokenizer is trained from coarse to fine levels, reducing codebook collapse and boosting semantic coverage. Its benefits include:

• Structured Multi-level Representation: Each item’s semantics is captured in both coarse and fine granular discrete identifiers.
• Domain-Invariant Token Vocabulary: Vocabulary sharing across domains enables improved generalization and transfer.
• Minimized Codebook Collapse: This mitigates common issues in VQ-based representations and maintains diverse concept coverage. A plausible implication is that this mechanism allows RecBase to quickly adapt to new domains and unseen items due to distributed, reusable token mappings.

5. Empirical Evaluation

RecBase is benchmarked in zero-shot settings on eight real-world datasets: MIND (news), MovieLens (movies), MicroLens (short videos), Goodreads (books), Yelp (local services), Steam (gaming), H{paper_content}M, and HotelRec. Key results are as follows:

Variant	Parameters	AUC Score	Comparison Baseline
RecBase_large	1.5B	~0.6063	Surpasses BERT_base, GPT-3.5
RecBase_base	313M	--	Matches LLM baselines

• RecBase_large matches or exceeds the ranking accuracy of LLM models up to 7B parameters, particularly showing substantial gains on datasets such as H{paper_content}M and Steam.
• Inference latency is reduced due to the unified token space, supporting deployment in latency-sensitive production scenarios. This suggests the model is both effective and computationally efficient, with strong cross-domain generalization.

6. Implications for Recommendation System Design

The RecBase approach underscores several critical themes for the recommendation research community:

• Hierarchical Discrete Representations: These permit granular semantic modeling across heterogeneous domains, crucial for cold-start and few-shot scenarios.
• Autoregressive Pretraining on Interaction Sequences: Captures complex sequential dependencies and item-item co-occurrence structures beyond text semantics.
• Scalability and Efficiency: RecBase demonstrates lower computational cost and faster inference than conventional LLM-based recommendation systems. A plausible implication is that explicit recommendation-aligned pretraining, coupled with advanced tokenization methods, sets a new baseline for zero-shot recommender system robustness.

7. Future Directions

Proposed future research avenues include:

• Integration of Multimodal Signals: Extending the unified tokenizer to incorporate images, audio, and modalities with heterogeneous input characteristics.
• Data Sparsity and Bias Mitigation: Exploring advanced fine-tuning and debiasing strategies to further improve performance for long-tail items and new users.
• Dynamic Tokenizer Adaptation: Investigating token space adjustment and cross-domain transfer learning as new domains and types of items arise. A plausible implication is that convergence of unified tokenization and generative pretraining will facilitate even greater domain adaptability.

Conclusion

RecBase offers a generative, recommendation-centric foundation model with hierarchical discrete item representation and autoregressive pretraining, outperforming strong LLM-based baselines in zero-shot, cross-domain tasks (Zhou et al., 3 Sep 2025). Its unified item tokenizer and efficient architecture present a scalable, robust approach for next-generation recommendation systems, with strong implications for multimodal integration and dynamic domain adaptation.