IDIOMoE: Item-ID + Oral-Language MoE Model

• The paper introduces IDIOMoE, which splits each transformer FFN into a text expert and an item expert via static token-type routing to improve recommendation accuracy.
• IDIOMoE maintains separate pathways for collaborative filtering signals and linguistic semantics, preserving the language understanding of the pretrained model.
• Empirical results show up to 30% gains in NDCG and HR with minimal latency overhead, demonstrating effective integration of recommendation and language tasks.

Searching arXiv for the primary paper and key related works mentioned in the provided data. Item-ID + Oral-language Mixture-of-Experts LLM (IDIOMoE) is a recommendation-oriented adaptation of a pretrained decoder-only transformer that treats item interaction histories as a native dialect within the language space while keeping that dialect distinct from natural language through token-type Mixture-of-Experts routing. Introduced in "Catalog-Native LLM: Speaking Item-ID Dialect with Less Entanglement for Recommendation" (Shirkavand et al., 30 Sep 2025), the method is motivated by the mismatch between collaborative filtering, which is highly accurate and efficient but semantically opaque, and pretrained LLMs, which provide rich world knowledge and instruction-following but do not natively model implicit user preferences from user–item interaction patterns. IDIOMoE addresses this by splitting each transformer block’s Feed Forward Network into a text expert and an item expert, with static routing based on token type, so that collaborative and linguistic signals remain in the same token space without being forced through the same FFN parameters (Shirkavand et al., 30 Sep 2025).

1. Position within recommendation and language modeling

Modern recommendation systems increasingly need to combine collaborative filtering and LLMs. In the formulation associated with IDIOMoE, collaborative filtering supplies predictive accuracy and efficiency, while LLMs provide expressive and generalizable reasoning, natural-language query handling, and transparent explanations (Shirkavand et al., 30 Sep 2025). The underlying problem is that collaborative signals are token-efficient but semantically opaque, whereas LLMs are semantically rich but struggle to model implicit user preferences when trained only on textual inputs.

The model is framed as a response to prior attempts to inject item IDs into an LLM vocabulary, including CoVE, CLLM4Rec, and URM, which are described as tending to entangle collaborative signals with linguistic semantics, causing destructive interference and degrading both recommendation accuracy and language fluency (Shirkavand et al., 30 Sep 2025). IDIOMoE instead treats item-ID sequences as a native dialect distinct from natural language, yet in the same token space.

This suggests that the central conceptual move is not merely vocabulary expansion, but architectural separation of pathways for two token regimes that must coexist in a single autoregressive model. A plausible implication is that the design targets unified generative modeling of recommendation and language tasks without requiring collaborative representations to inherit ordinary lexical semantics.

2. Core architectural design

The base model is a decoder-only transformer such as Qwen2.5-0.5B, in which the FFN in each block is replaced by a two-expert module (Shirkavand et al., 30 Sep 2025). The text expert is the unmodified pretrained FFN, and the item expert is a newly initialized FFN of matching or optionally reduced width. Multihead attention, layer norms, and positional embeddings are shared across both experts.

Routing is implemented by a static token-type gate: $$g(x)=1 \text{ if } x \text{ is a text token}, \qquad g(x)=0 \text{ if } x \text{ is an item-ID token.}$$

The combined module is: $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$

Because only one expert fires per token, total FLOPs and peak memory remain almost the same as the original model (Shirkavand et al., 30 Sep 2025). The architecture therefore differs from a conventional sparse MoE in which routing is typically learned dynamically across multiple experts. Here, static routing assigns a consistent role to each expert: text flows through an untouched expert intended to preserve language understanding, while item tokens flow through a dedicated expert intended to model collaborative patterns.

The paper’s ablations support that design choice. Replacing static token-type routing with a switch-style dynamic router causes a large performance collapse, reported as $-59\%$ NDCG@10 (Shirkavand et al., 30 Sep 2025). This is presented as evidence that a fixed text/item split maximizes specialization and minimizes cross-modal interference.

3. Item-ID representation and the catalog-native dialect

IDIOMoE extends the tokenizer with special <|item-📒|> tokens, one per catalog item (Shirkavand et al., 30 Sep 2025). Each item token is mapped by a hybrid embedding layer to a trainable vector $e_{\mathrm{item}} \in \mathbb{R}^d$, optionally factorized for large catalogs, plus, if desired, a frozen semantic bias from item titles.

During continued pretraining or fine-tuning, item tokens are routed through the item expert, allowing the network to learn what the paper calls a catalog-native dialect in the same embedding space as natural language (Shirkavand et al., 30 Sep 2025). In this setting, item IDs are not treated as external retrieval keys or solely as metadata anchors; they become first-class autoregressive tokens.

This suggests that the dialect metaphor is operational rather than rhetorical. The model does not translate item interactions into ordinary text, nor does it isolate them in a separate recommender module. Instead, it constrains how they are processed inside the transformer by preserving a shared autoregressive sequence model while separating FFN specialization by token type. A plausible implication is that item IDs can participate in long-context sequence modeling with text without requiring semantic reinterpretation as natural-language entities.

4. Training objective and optimization regime

The core training objective is standard next-token prediction over mixed sequences of text and item IDs (Shirkavand et al., 30 Sep 2025). For a training sequence $x_{1:T}$, the negative log-likelihood is

$$\mathcal{L}_{\mathrm{NLL}}=-\sum_{t=1}^{T}\log p_\theta\bigl(x_t \mid x_{<t}\bigr).$$

No additional BPR or pairwise ranking loss is used; collaborative patterns are reported to emerge from generative modeling of users’ historical item sequences interleaved with text (Shirkavand et al., 30 Sep 2025). This is a notable design decision because many recommendation systems depend on auxiliary ranking objectives or contrastive formulations, whereas IDIOMoE remains within a pure autoregressive likelihood framework.

The optimization setup uses AdamW with linear warmup followed by cosine decay, batch size 128, and maximum context length 1,024 tokens (Shirkavand et al., 30 Sep 2025). In large-scale runs, item experts and item embeddings are trainable, while the text expert and attention parameters may remain frozen for stability.

The production recommendations are consistent with this regime. They include freezing the pretrained text expert and attention blocks, fine-tuning only item experts and item embedding parameters, using static token-type gating, inserting MoE FFNs in the last transformer layers, adjusting item-expert width to catalog scale, warming up item experts on item-only sequences for a fraction of training and then mixing in text gradually, and retaining the original LLM head for text tokens while using a shared hybrid head so the model can generate item IDs directly (Shirkavand et al., 30 Sep 2025).

5. Empirical evaluation and comparative results

The reported evaluation covers public Amazon subsets—Games, Instruments, Arts, Sports, Beauty, and Toys—as well as larger 2023 catalogs for Books, Beauty, and Toys, and a proprietary industrial dataset with hundreds of millions of users and tens of thousands of items (Shirkavand et al., 30 Sep 2025). Baselines include classical CF and sequential models such as GRU4Rec, SASRec, BERT4Rec, FDSA, S3-Rec, TIGER, VQ-Rec, MISSRec, MQL4GRec, IDGenRec, and HSTU, together with LLM-based methods such as P5/P5-CID, VIP5, ReAT, and E4SRec. Three controlled variants on the same backbone are also compared: ID-Transformer, Text-Attr LLM, and Item-LLM (Shirkavand et al., 30 Sep 2025).

The evaluation metrics are NDCG@10, HR@10, and, on the proprietary dataset, MRR (Shirkavand et al., 30 Sep 2025). The reported results are summarized below.

Setting	Reported result	Metric
Six small Amazon domains	IDIOMoE ranks first among all LLM-based methods; example on Games: $+0.1102$ HR@10 vs. $0.1089$ for HSTU	HR@10
Large Amazon catalogs	Leading LLM and competitive with the best CF models; example on Books: $0.0419$	NDCG@10
Proprietary dataset vs. SASRec	$+27.1\%$, $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$0, $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$1	NDCG@10, HR@10, MRR

The paper also states that, with the recommended design choices, IDIOMoE achieves up to $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$2 relative gains in NDCG and HR while incurring only a few-percent latency overhead at real-world context lengths (Shirkavand et al., 30 Sep 2025).

These findings are presented as evidence that separating semantic and collaborative pathways improves recommendation quality without sacrificing the language understanding of the pretrained model. A plausible implication is that the method is especially relevant where a single model must both interpret text and predict item continuations.

6. Ablations, capacity controls, and mechanistic analysis

The ablation study distinguishes architectural specialization from raw parameter growth. Under matched parameter count, Wide-FFN, append/prepend extra blocks, and LoRA adapters yield far smaller gains or degrade compared with IDIOMoE (Shirkavand et al., 30 Sep 2025). The stated interpretation is that structured specialization, rather than parameter count alone, drives the improvement.

Item expert width is varied by shrink factors $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$3. On small benchmarks, a moderate shrink by $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$4 balances efficiency and performance, whereas on the industrial dataset any shrink hurts, implying that large-scale catalogs need full expert capacity (Shirkavand et al., 30 Sep 2025). MoE placement is also investigated: activating MoE in different layer subsets reveals that the last 8 layers benefit most, with an example gain of $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$5 NDCG@10, since deeper layers encode task-specific patterns (Shirkavand et al., 30 Sep 2025).

The mechanistic analysis follows Geva et al. (2022) and treats each FFN’s output weights $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$6 as defining value-vectors $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$7 (Shirkavand et al., 30 Sep 2025). Cosine alignment is compared against item embeddings $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$8 and text embeddings $$\mathrm{FFN}_{\mathrm{IDIOMoE}}(x)=g(x)\,\mathrm{FFN}_{\mathrm{text}}(x)+\bigl(1-g(x)\bigr)\,\mathrm{FFN}_{\mathrm{item}}(x).$$9: $-59\%$0

The analysis defines: $-59\%$1

$-59\%$2

$-59\%$3

The reported result is that, in IDIOMoE, the item expert’s neurons have higher positive item-text affinity, stronger category purity, and more clustered rows—especially in deep layers—than a non-MoE baseline (Shirkavand et al., 30 Sep 2025). This suggests that the architecture induces internal organization that is more catalog-specific and less linguistically entangled.

7. Interpretation, scope, and common points of confusion

A common misconception would be to regard IDIOMoE as simply an LLM with extra item tokens. The paper’s argument is narrower and more specific: vocabulary expansion without MoE corresponds to a distinct controlled baseline, Item-LLM, whereas IDIOMoE’s defining feature is the separation of the pretrained FFN into a text expert and an item expert with static token-type routing (Shirkavand et al., 30 Sep 2025). The method is therefore not reducible to tokenizer extension alone.

Another possible confusion concerns whether the model abandons ordinary language modeling in favor of a recommender-specific loss. The stated objective remains standard next-token prediction over mixed sequences, and no additional BPR or pairwise ranking loss is used (Shirkavand et al., 30 Sep 2025). Recommendation behavior is intended to emerge from generative modeling rather than from an externally imposed ranking objective.

The model is also not described as a fully dynamic MoE system. In contrast, static routing is explicitly favored over dynamic routing because the fixed text/item split gives each expert a consistent role and is empirically associated with better results (Shirkavand et al., 30 Sep 2025). Likewise, shared attention, layer norms, and positional embeddings indicate that the architecture preserves a common transformer backbone while localizing specialization to the FFN pathway.

Within the broader landscape of recommendation-language integration, IDIOMoE is best understood as a disentanglement strategy. It seeks to retain the text understanding of the pretrained model while making item interaction histories legible as a catalog-native dialect. This suggests a model family in which recommendation is not attached to an LLM as an external module, but encoded as a structurally separated modality inside the same autoregressive sequence model (Shirkavand et al., 30 Sep 2025).