Grounded Token Initialization Hypothesis

• The paper demonstrates that initializing new tokens at semantically meaningful positions enables models to leverage existing knowledge more effectively.
• GTI operationalizes this by using paired text descriptions to ground novel tokens before fine-tuning, ensuring alignment with pretrained embedding spaces.
• Empirical results show significant gains in metrics like Precision@K and NDCG, validating the superiority of semantically informed initialization over standard methods.

Searching arXiv for the cited papers and closely related work on grounded token initialization. The Grounded Token Initialization Hypothesis is the claim that, when a pretrained LLM is extended with novel tokens, those tokens should first be placed at distinct, semantically meaningful locations in the pretrained embedding space so that downstream optimization can exploit the model’s existing general-purpose knowledge rather than rebuild token geometry from a collapsed or off-manifold start (Chen et al., 2 Apr 2026). In its canonical formulation, the hypothesis was introduced for Semantic-ID tokens in generative recommendation, but closely related ideas recur in cross-lingual tokenizer conversion, tool-token learning, and morphology-aware vocabulary expansion. Across these settings, token initialization is treated not as a neutral preprocessing step but as a primary determinant of whether new symbols become usable, discriminative, and compatible with pretrained representations (Remy et al., 2023, Li et al., 17 Jun 2025, Teklehaymanot et al., 23 Mar 2026).

1. Definition and conceptual scope

The hypothesis is stated explicitly as follows: “linguistically grounding novel tokens in an LM’s pretrained embedding space, before fine-tuning, better enables the model to leverage its general-purpose knowledge for novel-token domains” (Chen et al., 2 Apr 2026). The claim is therefore not merely that new tokens need non-random vectors, but that their initial positions should already reflect the semantics associated with those tokens.

In the original formulation, “linguistically grounding” has a concrete operational meaning. Each new domain token is paired with natural-language descriptions such as titles, descriptions, or candidate profiles. During a dedicated grounding stage, the LM backbone is frozen and the model is trained on both text-to-token and token-to-text generation, so that only the newly introduced embedding rows are adjusted. A new token is considered grounded when, inside the frozen LM, its behavior is consistent with the natural-language semantics associated with it. This makes grounding an embedding-space alignment procedure rather than a general fine-tuning heuristic (Chen et al., 2 Apr 2026).

A central implication is that vocabulary extension should be understood as a representation-learning problem. Mean initialization places new tokens “on the manifold” of existing embeddings, and random initialization preserves token distinctness, but neither strategy guarantees that the pretrained network can interpret the new symbols in a semantically coherent way. The hypothesis proposes that semantic placement prior to end-to-end training is the missing intermediate step.

2. Origin in generative recommendation and the diagnosis of mean initialization

The hypothesis emerged from the setting of generative recommendation / generative retrieval, where items are first embedded into a continuous space and then discretized by an RQ-VAE into Semantic IDs (SIDs). An item embedding $\mathbf{z}_i \in \mathbb{R}^d$ is quantized into a sequence of code indices $(c_1,\dots,c_L)$ by residual quantization, and each codebook entry becomes a new vocabulary token such as <a_1> or <b_1>. The LM is then trained autoregressively to generate the SID sequence conditioned on a user history or query (Chen et al., 2 Apr 2026).

The standard heuristic for these new tokens is mean-of-vocabulary initialization:

$$\mathbf{e}_{c} := \frac{1}{|\mathcal{V}_{\mathrm{text}}|}\sum_{v \in \mathcal{V}_{\mathrm{text}}}\mathbf{e}_v, \quad \forall\, c \in \mathcal{V}_{\mathrm{SID}}.$$

This assigns exactly the same vector to every new token. In the SID block of the embedding matrix, the result is rank-$1$ initialization: all SID embeddings lie in a one-dimensional subspace, pairwise cosine similarities are nearly $1$, and the SID–SID cosine block is a near-uniform high-similarity block. The paper characterizes this as a degenerate subspace (Chen et al., 2 Apr 2026).

The key empirical finding is that supervised fine-tuning does not fully repair this collapse. After fine-tuning from mean initialization, the SID embedding matrix still exhibits rapid singular-value decay and low effective rank; pairwise cosine similarity remains flat or noisy; and Representational Similarity Analysis with the oracle RQ-VAE codebook yields lower Pearson $r$ and Spearman $\rho$ than GTI initialization. By contrast, GTI produces slower spectral decay, higher effective rank, richer pairwise structure, and higher RSA after fine-tuning. The diagnosis is that initialization quality is a structural bottleneck: once thousands of new tokens start from a collapsed geometry, gradient descent only partially unfolds the subspace (Chen et al., 2 Apr 2026).

3. GTI as the canonical operationalization

The canonical implementation of the hypothesis is GTI (Grounded Token Initialization), introduced as a separate stage before supervised fine-tuning. The procedure extends the vocabulary, partitions the embedding matrix into original and new rows, constructs a grounding corpus of paired examples $(x_i,y_i)$ with natural-language descriptions $x_i$ and canonical new-token sequences $y_i$, adds reversed pairs $(c_1,\dots,c_L)$0, freezes all LM parameters except the new embedding rows, and optimizes a standard autoregressive LM objective over the grounding corpus (Chen et al., 2 Apr 2026).

The grounding objective is:

$(c_1,\dots,c_L)$1

with the crucial restriction that only $(c_1,\dots,c_L)$2 is updated during grounding. Because the embedding matrix and LM head are tied, changing these rows affects both how the model reads new tokens and how it predicts them. The inductive bias is explicit: find new embeddings that make the frozen LM perform text$(c_1,\dots,c_L)$3token mapping, then unfreeze the model for standard downstream fine-tuning (Chen et al., 2 Apr 2026).

Empirically, GTI outperforms both mean initialization and auxiliary-task adaptation baselines in the majority of evaluation settings. On the industrial candidate-retrieval dataset, all numbers are reported as relative percentage gain over MI + Vanilla SFT. For Precision@K (Good Match), MI + Multi-task SFT (LC-Rec) yields +6.38% at P@5, +5.20% at P@10, and +3.87% at P@20, whereas GTI + Multi-task SFT yields +21.63%, +13.59%, and +8.16% respectively. For NDCG@5, LC-Rec yields +6.94% while GTI + Multi-task reaches +17.88%. On the public Vibrent Clothes Rental dataset, GTI + Vanilla SFT also exceeds MI + Multi-task SFT (LC-Rec) on several metrics, including Recall@20 (+26.02% versus +13.41%) and Recall@10 (+22.03% versus +11.86%), while NDCG is competitive and better for $(c_1,\dots,c_L)$4; for example, NDCG@20 is +12.23% for GTI versus +11.30% for LC-Rec. These performance gains track the geometric diagnostics: GTI starts with differentiated structure and preserves it through fine-tuning rather than attempting to reconstruct it from a rank-$(c_1,\dots,c_L)$5 collapse (Chen et al., 2 Apr 2026).

4. Related instantiations across domains

Work outside recommendation does not always use the same label, but several recent methods instantiate closely related assumptions about semantically anchored initialization. In “Tik-to-Tok”, target-language tokens are initialized by semantically similar source-language tokens using a word translation dictionary, a synthetic bilingual bigram corpus, and bilingual fastText. Each target token embedding is set as a weighted average of source-token embeddings under a one-to-many mapping. In the low-resource Frisian setting, the method reduces MLM loss at initialization from 9.11 under random initialization to 5.50 for the full dictionary + fastText strategy; after two epochs the same ordering remains, with 7.06, 5.02, 4.23, and 3.79 for random, shared-tokens, dictionary-only, and dictionary + fastText respectively. In the Dutch setting, converted models trained with only ~15% (= ~7GB) of Dutch OSCAR data match or exceed monolingual Dutch baselines on several downstream tasks, indicating that semantically grounded embedding transfer improves both data efficiency and final performance (Remy et al., 2023).

In “Re-Initialization Token Learning for Tool-Augmented LLMs”, tool tokens are initialized from textual priors derived from tool names or descriptions. If the tool label is a single in-vocabulary token, its embedding is used directly; otherwise, embeddings of the constituent tokens are pooled, and the learnable tool-token matrix is initialized with this prior and regularized by an $(c_1,\dots,c_L)$6 penalty toward it. The ablation most directly tied to the hypothesis compares initialization from irrelevant vocabulary with initialization from tool names. On GSM8K-XL, LLaMA2-70B improves from 0.4526 to 0.4683, and LLaMA2-13B improves from 0.2306 to 0.2324, when tool tokens are initialized from semantically relevant names rather than irrelevant words. Against ToolkenGPT, the full method also improves GSM8K-XL accuracy from 0.4789 to 0.5422 on 70B and from 0.2165 to 0.2852 on 13B, and improves VirtualHome plan success from 0.76 to 0.78 on 70B and from 0.68 to 0.72 on 13B (Li et al., 17 Jun 2025).

In “LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation”, novel tokens are grounded in morphemes and character n-grams rather than paired descriptions or tool labels. Words are segmented into morphemes, morpheme representations are composed from FastText n-gram embeddings, a linear projection aligns these vectors to the XLM-R embedding space, and a regularization term penalizes drift from the initialized values during Language-Adaptive Pretraining. On Amharic and Tigrinya, XLM-R + LGSE + LAPT outperforms XLM-R + FOCUS + LAPT and XLM-R + Random + LAPT across all reported tasks: average QA F1 rises to 78.25 versus 76.65 and approximately 70.0, average text-classification accuracy to 76.5 versus 74.45 and 70.35, and average NER F1 to 79.2 versus 77.8 and 72.15 (Teklehaymanot et al., 23 Mar 2026).

Taken together, these systems suggest a broader family of grounded initialization methods: dictionary-grounded transfer, description-grounded tool tokens, and morphology-grounded subword embeddings all replace arbitrary or collapsed starts with embeddings that already encode a domain-specific semantic prior.

5. Mechanistic and theoretical perspectives

The recommendation paper frames mean initialization as an instance of dimensional collapse, and that language recurs in broader work on initialization. In “Small Initialization Matters for LLMs”, parameter initialization scale is exposed through a $(c_1,\dots,c_L)$7-rule,

$(c_1,\dots,c_L)$8

and smaller initialization is shown to improve pretraining and benchmark performance, with the largest gains on reasoning-demanding tasks. For a 1.5B model, moving from $(c_1,\dots,c_L)$9 to $$\mathbf{e}_{c} := \frac{1}{|\mathcal{V}_{\mathrm{text}}|}\sum_{v \in \mathcal{V}_{\mathrm{text}}}\mathbf{e}_v, \quad \forall\, c \in \mathcal{V}_{\mathrm{SID}}.$$0 raises ARC-C from 27.28 to 29.01, TriviaQA from 8.80 to 13.00, HellaSwag from 38.74 to 43.38, GSM8K from 2.70 to 7.60, MATH500 from 6.00 to 10.00, and the average across the listed benchmarks from 22.47 to 24.98. The paper also reports that token-level gains concentrate on non-trivial, context-constrained predictions rather than all tokens uniformly. This suggests that initialization controls not only whether new symbols are learnable, but also which kinds of token-level structure are preferentially amplified during training (Hang et al., 16 Jun 2026).

A complementary line of evidence appears in “Transformers Are Born Biased”, which argues that transformers are behaviorally structured even at random initialization. On random input sequences, untrained decoder-only transformers exhibit extreme token preferences: in Nano GPT-2, one seed yields a top token appearing in 6.75% of 10,000 random sequences of length 1024, and in a 1.2B GPT-2 the top token appears in 10.2% of such sequences. The proposed mechanism is a contraction of token representations along a random seed-dependent direction driven by asymmetric nonlinear activations in MLP sublayers and amplified by self-attention; the resulting bias persists through training and is stable enough to support SeedPrint, a fingerprinting method that distinguishes models differing only in random initialization (Li et al., 5 Feb 2026).

These results do not establish the Grounded Token Initialization Hypothesis in the same operational form as GTI, but they materially change its theoretical backdrop. A plausible implication is that vocabulary extension methods are not inserting tokens into a neutral space. They are intervening in an embedding geometry that is already anisotropic, seed-dependent, and developmentally consequential. GTI-like procedures can therefore be interpreted as controlled attempts to align novel symbols with the pretrained geometry the rest of the network already exploits.

6. Limitations, misconceptions, and open questions

The strongest direct validation of the hypothesis remains domain-specific. In the original GTI study, the core experiments are in generative recommendation, with paired textual supervision available for each item or candidate. The authors note several limits explicitly: GTI depends on paired text, may weaken when descriptions are missing or noisy, is constrained by the expressivity of the frozen LM during grounding, and was compared mainly against mean initialization, random initialization, and LC-Rec-style multi-task adaptation rather than a wide family of alternative initializers. They also frame broader applications—entity IDs in KG-augmented LMs, domain-specific code tokens, multimodal tokens, and combinations with PEFT methods or contrastive losses—as open directions rather than settled results (Chen et al., 2 Apr 2026).

Related work exposes analogous constraints. Tik-to-Tok relies on a translation dictionary and is evaluated mainly on Germanic and Romance language settings; LGSE depends on morphologically annotated resources; and TokenLearning requires a semantically meaningful textual label for each tool token as well as task-specific tuning of the regularization coefficient. Across these papers, grounded initialization is consistently stronger when the grounding signal is informative and structurally appropriate, and weaker when the lexical or descriptive prior is sparse, noisy, or only loosely connected to the token’s functional role (Remy et al., 2023, Li et al., 17 Jun 2025, Teklehaymanot et al., 23 Mar 2026).

A common misconception is to equate every token-level use of the term grounding with token initialization. “Preference-grounded Token-level Guidance for LLM Fine-tuning” grounds sequence-level preference into token-level rewards and weighted MLE, addressing the granularity mismatch between sequence-level evaluation and token-level training. That work is adjacent in spirit but distinct in object: it does not study initialization of newly introduced vocabulary embeddings (Yang et al., 2023).

The current state of the topic is therefore uneven but coherent. Within the tested settings—Semantic IDs in recommendation, source-to-target token transfer, tool tokens, and morphology-aware vocabulary expansion—the hypothesis is strongly supported by both performance and geometry. Beyond those settings, the broader claim remains plausible rather than established: initialization appears to matter far more than standard practice assumes, but the general laws governing where new tokens should live in LM space, how strongly they should be constrained there, and when grounding should be lexical, descriptive, cross-lingual, or multimodal remain open research questions.