Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation

Abstract: LLMs (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations. We present a systematic analysis of this strategy: through spectral and geometric diagnostics, we show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover. These findings suggest that \emph{token initialization} is a key bottleneck when extending LMs with new vocabularies. Motivated by this diagnosis, we propose the \emph{Grounded Token Initialization Hypothesis}: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general-purpose knowledge for novel-token domains. We operationalize this hypothesis as GTI (Grounded Token Initialization), a lightweight grounding stage that, prior to fine-tuning, maps new tokens to distinct, semantically meaningful locations in the pretrained embedding space using only paired linguistic supervision. Despite its simplicity, GTI outperforms both mean initialization and existing auxiliary-task adaptation methods in the majority of evaluation settings across multiple generative recommendation benchmarks, including industry-scale and public datasets. Further analyses show that grounded embeddings produce richer inter-token structure that persists through fine-tuning, corroborating the hypothesis that initialization quality is a key bottleneck in vocabulary extension.

• The paper introduces the Grounded Token Initialization (GTI) hypothesis to address semantic collapse from mean token embedding, enabling distinct and semantically coherent token representations.
• The GTI method employs a lightweight grounding stage with bidirectional linguistic supervision that aligns new tokens with the pretrained embedding space, ensuring higher effective rank.
• Experiments on industrial-scale and public benchmarks demonstrate significant gains, including a +21.63% improvement in P@5 and enhanced Recall and NDCG metrics.

Grounded Token Initialization for New Vocabulary in LLMs for Generative Recommendation

Introduction

Expanding the vocabulary of pretrained LMs is a pervasive challenge for domain adaptation, especially for high-cardinality generative retrieval and recommendation scenarios. The standard methodology—initializing new token embeddings to the mean of the pretrained vocabulary—induces severe geometric degeneracy: new tokens collapse into a single point in embedding space, eradicating distinctions essential for downstream generative reasoning. The paper "Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation" (2604.02324) presents a rigorous spectral and geometric analysis of this degeneration, introduces the Grounded Token Initialization (GTI) hypothesis, and proposes a concrete, lightweight grounding stage that aligns new tokens with the pretrained space using bidirectional linguistic supervision. Extensive evaluation on industrial-scale and public generative recommendation benchmarks demonstrates robust and consistent gains from the proposed approach.

Token-Embedding Collapse and the GTI Hypothesis

Standard mean initialization (mean-of-vocab) offers no semantic separation among novel tokens, leading to a degenerate, low-rank subspace—a phenomenon confirmed by pairwise cosine similarity matrices and rapid spectral decay of the embedding matrix after fine-tuning. Figure 1 displays: mean initialization collapses new tokens while random initialization creates noisy, unstructured representations, both unsatisfactory for complex downstream alignment.

Figure 1: Mean initialization collapses new SID tokens, while grounding distributes them into semantically distinct regions of the pretrained embedding space.

Figure 2 provides clear evidence: only GTI produces differentiated intra-block structure with meaningful lexical affinities, as opposed to the uniformity or noise found with the standard alternatives.

Figure 2: Pairwise cosine-similarity matrices show that GTI creates semantically structured SID embeddings, in contrast to mean or random strategies.

Singular value analysis (Figure 7a, Figure 4) reveals that mean initialization yields rapid spectral decay and dimensional collapse, while GTI produces embeddings with higher effective rank and a broader span within the pretrained manifold, indicating expressive capacity suited to complex domain semantics.

Figure 3: GTI yields slower spectral decay and higher representational similarity between SID embeddings and the ground-truth codebook.

This motivates the GTI hypothesis: linguistically grounding new tokens in the pretrained embedding space before any fine-tuning enables LMs to immediately leverage their learned geometry and attention/operator structure for new domain semantics, sidestepping the non-identifiable starting point induced by mean or random initialization.

Grounded Token Initialization: Approach

GTI comprises an intermediate grounding stage (Figure 5). Here, the LM backbone is frozen, and only the new token embedding matrix is optimized. Alignment is enforced via bidirectional paired supervision using natural language descriptions and their associated Semantic-IDs (SIDs). Specifically, prompt templates (see Appendix) are instantiated to require the LM to both generate SIDs from text, and reconstruct natural language from SIDs, ensuring that token representations capture semantic structure in both the input and output directions.

Figure 5: Overview of the grounding stage; only SID embeddings are trained with the backbone frozen, using paired text–SID supervision.

The grounding phase produces an embedding initialization for the new tokens that is semantically coherent, differentiated, and aligned with the geometry of the pretrained space. After grounding, all model parameters are unfrozen and standard supervised fine-tuning commences as usual.

Experimental Results

Empirical validation covers both a large-scale real-world candidate retrieval setting and the public Vibrent Clothes Rental benchmark. GTI is benchmarked against:

• Vanilla SFT (mean initialization + SFT): Canonical baseline.
• LC-Rec: Auxiliary-task adaptation (multi-task fine-tuning with language modeling and SID supervision).
• GTI: The proposed grounding initialization, combined with standard SFT.

Across all K in top-K evaluation, all relevance thresholds, and both metrics (Precision, Recall, NDCG), GTI produces dominant or competitive performance. For instance, on industrial-scale data, GTI delivers a +21.63% relative P@5 gain over SFT and a +15.25% additional gain over the auxiliary-task LC-Rec at strict Good-Match (Table 1; see also Figure 6 and 4). Gains are particularly pronounced at small K, which is crucial for practical retrieval effectiveness.

Figure 6: GTI consistently achieves larger relative Precision and NDCG gains as candidate pool size varies, especially at low K.

Figure 7: On the Vibrent dataset, GTI also attains superior Recall@K and NDCG@K compared to baselines.

On public data, with no auxiliary objectives, GTI alone leads to substantially higher Recall and competitive or better NDCG, confirming that the initialization bottleneck is primary and not fully addressable via end-to-end or auxiliary-task fine-tuning.

Further, inspection of embedding structure post-fine-tuning (Figure 8) shows that semantic hierarchy and block structures persist only under grounded initialization, while alternatives revert to flat or noisy geometries.

Figure 8: After SFT, only GTI maintains blockwise hierarchical structure among SID embeddings—mean and random initializations yield less informative similarity patterns.

Quantitative representational similarity analysis (RSA) establishes that GTI yields the highest correlation between RQ-VAE ground-truth codebook and learned embeddings (Figure 7b), demonstrating superior preservation of pretrained semantic relations.

Implications and Future Directions

The results support the claim that token initialization is a first-order bottleneck in vocabulary extension for LMs: supervised fine-tuning is unable to recover expressive, high-rank, semantically informative structure if the initial geometry is collapsed. GTI’s decoupling of embedding learning from backbone adaptation systematically solves this issue, providing a semantically coherent basis for subsequent task learning.

For generative recommendation, this enables scalable injection of thousands of SIDs with minimal degradation or need for slow, large-scale multi-task training. The framework is generic and directly applicable to any domain requiring vocabulary expansion with semantic grounding, e.g., tool use, visual concepts, named entities, or multimodal augmentation.

GTI also forms a bridge to recent findings on contrastive representation collapse—the spectral and geometric trends diagnosed here closely parallel those in self-supervised learning studies, suggesting principled connections between init-bottleneck, capacity utilization, and semantic transfer in deep architectures.

Potential extensions include automated construction of richer grounding corpora, adaptation to multi-lingual/multi-modal scenarios, and application to foundation models in non-recommendation domains where continuous vocabulary expansion is required.

Conclusion

The paper offers a rigorous treatment of the token initialization bottleneck in generative recommendation and LLM vocabulary extension. By identifying and quantifying semantic collapse under mean initialization and introducing a lightweight, effective grounding stage, it establishes that proper semantic grounding of new tokens is foundational for leveraging pretrained knowledge and achieving downstream task success. These findings suggest that grounding initialization should become a standard component in the LM vocabulary extension pipeline, with implications spanning recommendation, search, and beyond.