DECOR: Decomposed Contextual Token Representations

• DECOR is a framework for sequential recommendation that decomposes token representations into frozen pretrained semantic embeddings and learned collaborative embeddings.
• It employs a T5-based generative model to autoregressively generate item token sequences, dynamically adapting token meanings based on user history.
• The method addresses prefix ambiguity and enhances effective embedding utilization, leading to notable improvements in recommendation accuracy.

Decomposed Contextual Token Representations, abbreviated DECOR, most specifically denotes a framework for sequential generative recommendation in which item tokens are represented by a combination of preserved pretrained semantics and newly learned collaborative signals, while their realized meaning is further adapted to user interaction context at generation time (Liu et al., 22 Aug 2025). In this formulation, items are not predicted as atomic IDs; instead, each item is mapped to a short semantic ID and a T5-based generative recommender autoregressively generates the target item’s token sequence. DECOR was introduced to address a structural mismatch in the standard two-stage pipeline: semantic tokenization is pretrained for reconstruction of metadata-derived embeddings, whereas recommender training is optimized for user interaction modeling. A broader research context includes methods that decompose, attribute, or structurally refine contextual token states in LLMs and diffusion conditioning pipelines, but those works differ materially in objective, architecture, and the meaning attached to “decomposition” (Oh et al., 2023).

1. Problem formulation and motivation

DECOR is defined on the sequential recommendation setting in which a user has an interaction history

$\mathbf{S}^{u} = [i_1,i_2,\dots,i_{t-1}],$

and the task is to predict the next item $i_t$ (Liu et al., 22 Aug 2025). Rather than predicting $i_t$ directly as an item ID, the framework assumes that each item $i$ has already been tokenized into a semantic ID

$\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$

so that recommendation becomes sequence generation: $$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$

The method is a response to two limitations of the prevailing two-stage paradigm. The first is suboptimal static tokenization. In systems such as TIGER, item metadata are encoded into pretrained semantic embeddings and then quantized by an RQ-VAE into discrete semantic IDs, but those token assignments remain fixed after tokenizer pretraining (Liu et al., 22 Aug 2025). Because the tokenizer is optimized for semantic reconstruction rather than recommendation quality, its assignments can be poor for sequential recommendation, particularly when multiple items share the same early prefix tokens despite differing in recommendation meaning. The paper emphasizes prefix ambiguity as the main failure mode.

The second limitation is discarded pretrained semantics. Even when semantic IDs are derived from pretrained tokenizers and codebooks, downstream recommenders typically use discrete token IDs whose embeddings are randomly initialized and then learned purely from interaction data (Liu et al., 22 Aug 2025). In that regime, the semantic knowledge captured during tokenizer pretraining is effectively overwritten or ignored. DECOR therefore attempts to preserve pretrained semantic structure while making token representations context-sensitive enough for collaborative prediction.

The tokenizer inherited by DECOR follows the TIGER setup and uses an RQ-VAE semantic indexer with multiple residual quantization levels. Given an item metadata embedding $\mathbf{x}\in\mathbb{R}^d$, the residual quantization process is

$\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$

$$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$

$$q_l(\mathbf{z}_{l-1}) = \arg\min_{\mathbf{e}\in \mathbf{C}_l}\|\mathbf{z}_{l-1}-\mathbf{e}\|_2.$$

The quantized representation is reconstructed as

$i_t$0

and the tokenizer is pretrained with

$i_t$1

where

$i_t$2

and

$i_t$3

with $i_t$4 (Liu et al., 22 Aug 2025).

2. Decomposition into pretrained and collaborative embeddings

The core representational claim of DECOR is that a token should not be forced into a single embedding space. Instead, each token is represented by two distinct components: a pretrained semantic embedding from the frozen RQ-VAE codebook, and a collaborative embedding learned from recommendation supervision (Liu et al., 22 Aug 2025). This is the part of the framework that gives the method its “decomposed” character in the strict sense.

The semantic indexer is an $i_t$5-layer RQ-VAE with codebooks

$i_t$6

The pretrained semantic embedding space is

$i_t$7

where each $i_t$8 is a frozen codebook for layer $i_t$9. For a token index $i_t$0, the semantic embedding is obtained by lookup: $i_t$1 A separate collaborative embedding space is learned as

$i_t$2

These two views are aligned by projection into a shared latent space followed by normalization: $i_t$3 where $i_t$4. They are then concatenated and mapped back into model space: $i_t$5 with $i_t$6 (Liu et al., 22 Aug 2025).

Several properties follow directly from this design. First, the pretrained codebooks remain frozen during recommender training, so DECOR explicitly retains tokenizer semantics instead of relearning them from scratch. Second, the collaborative embeddings are free to absorb recommendation-specific structure. Third, the fusion is neither additive nor gate-based; it is concatenation followed by a learned linear fusion layer. The paper characterizes this as a modality-aware architecture in which semantic and collaborative signals are kept separate until the final fusion projection.

This decomposition is used at the encoder input side of the downstream recommender. The recommender itself is a T5-based generative model, and the RQ-VAE tokenizer is pretrained with the same hidden size as the recommender for compatibility; in the reported experiments,

$i_t$7

(Liu et al., 22 Aug 2025).

3. Contextualized token composition

DECOR’s second main component is Contextualized Token Composition, which addresses the rigidity of fixed semantic IDs without changing the token assignments themselves (Liu et al., 22 Aug 2025). The semantic IDs remain cached and stable, but the embedding associated with a token is dynamically reinterpreted as a function of user history and generation context. This distinguishes DECOR both from purely static semantic-ID methods and from end-to-end retokenization approaches such as ETEGRec.

For a target item $i_t$8 with semantic ID sequence

$i_t$9

and historical context $i$0, DECOR computes a context-aware token representation

$i$1

Here $i$2 is a context vector derived from the user’s history, and $i$3 is the candidate token set used for composition.

The context vector is built from the fused embeddings of the historical context sequence: $i$4 where each

$i$5

DECOR applies attention pooling: $i$6 with

$i$7

$i$8

Thus the conditioning signal is sequence-level, but it is used to adapt token-level representations.

The candidate set $i$9 is defined as all tokens from the same RQ-VAE codebook layer (Liu et al., 22 Aug 2025). This layer-constrained scope preserves quantization hierarchy: a token in codebook layer 1 composes only over other layer-1 codes, and similarly for later layers. The composition itself is a soft attention mechanism: $\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$0

$\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$1

The composed embedding is then interpolated with the original static embedding: $\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$2

The paper reports best composition weights of $\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$3 on Scientific and $\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$4 on Instrument and Game (Liu et al., 22 Aug 2025). It also introduces a separate mechanism for the beginning-of-sequence case, because the first RQ-VAE token often captures coarse category-level semantics and no target tokens have yet been generated. For this purpose, DECOR adds $\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$5 learnable BOS query vectors

$\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$6

and composes the BOS embedding as

$\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$7

The tuned BOS query counts were $\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$8 on Scientific and $\mathbf{c}_i = [c_{i,1}, c_{i,2}, \dots, c_{i,L}],$9 on Instrument and Game (Liu et al., 22 Aug 2025).

A plausible implication is that DECOR treats token identity as simultaneously stable and contextual: stable because the semantic ID is unchanged, contextual because the realized embedding is recomputed from same-layer alternatives according to the user history.

4. Learning procedure, implementation, and empirical results

DECOR does not introduce a new recommendation loss beyond the autoregressive generation objective; recommendation-stage training is performed end-to-end through the same next-item token generation objective used by the downstream recommender (Liu et al., 22 Aug 2025). The pretrained codebooks remain frozen, while the collaborative embeddings, fusion parameters, context-pooling network, composition projections, BOS queries, and T5 recommender parameters are trained in the recommendation stage.

The experiments use three Amazon Review subsets after 5-core filtering: Scientific, Instrument, and Game (Liu et al., 22 Aug 2025). Sequence length is capped at 20. Dataset sizes are:

• Scientific: 50,985 users, 25,848 items, 412,947 interactions
• Instrument: 57,439 users, 24,587 items, 511,836 interactions
• Game: 94,762 users, 25,612 items, 814,586 interactions

Evaluation uses leave-one-out splitting, with the last interaction for test, the second last for validation, and the rest for training, under full-ranking evaluation. Metrics are Recall@5, Recall@10, NDCG@5, and NDCG@10. Baselines include traditional sequential recommenders—Caser, GRU4Rec, SASRec, BERT4Rec, FDSA, and S$$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$0Rec—generative recommenders with static tokenization—P5-SID, P5-CID, TIGER, LETTER, and CoST—and the dynamic tokenizer-recommender model ETEGRec. Implementation details reported by the paper are: Sentence-T5 base as text encoder, T5 as generative recommender, learning rate $$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$1, AdamW with cosine schedule, dropout $$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$2, batch size 256, max epochs 200, beam size 50, warmup 10,000, early stop patience 20, a single NVIDIA Tesla A40, and seed 2025 (Liu et al., 22 Aug 2025).

On all three datasets, DECOR reports the best results. On Scientific, DECOR achieves

$$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$3

The paper highlights this as +13.8% NDCG@10 on Scientific over ETEGRec. On Instrument, DECOR reaches

$$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$4

On Game, it reports

$$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$5

(Liu et al., 22 Aug 2025).

The ablations directly support DECOR’s two-component design. Adding Token Comp. over TIGER improves NDCG@10 from $$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$6 to $$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$7 on Scientific, from $$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$8 to $$p(\mathbf{c}_{i_t}\mid \mathbf{X}^{u}) = \prod_{l=1}^{L} p(c_{i_t,l}\mid \mathbf{X}^{u}, c_{i_t,1}, \dots, c_{i_t,l-1}).$$9 on Instrument, and from $\mathbf{x}\in\mathbb{R}^d$0 to $\mathbf{x}\in\mathbb{R}^d$1 on Game. The Pretrained Only variant also improves over TIGER, raising NDCG@10 from $\mathbf{x}\in\mathbb{R}^d$2 to $\mathbf{x}\in\mathbb{R}^d$3 on Scientific, from $\mathbf{x}\in\mathbb{R}^d$4 to $\mathbf{x}\in\mathbb{R}^d$5 on Instrument, and from $\mathbf{x}\in\mathbb{R}^d$6 to $\mathbf{x}\in\mathbb{R}^d$7 on Game. The full model, combining pretrained semantics, token composition, and BOS queries, yields the best NDCG@10 values of $\mathbf{x}\in\mathbb{R}^d$8, $\mathbf{x}\in\mathbb{R}^d$9, and $\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$0, respectively (Liu et al., 22 Aug 2025).

Two analyses are especially diagnostic. First, a case study on prefix ambiguity in the Scientific dataset uses t-SNE visualizations to show that a shared prefix such as $\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$1 corresponds to a single fixed embedding under static tokenization, whereas DECOR’s composed prefix embeddings scatter by context and align better with valid next-token candidates (Liu et al., 22 Aug 2025). Second, the paper measures codebook utilization and shows that many first-layer codebook entries are underused during tokenizer pretraining—Layer-1 usage under TIGER is 26.6% on Scientific, 27.97% on Instrument, and 25.67% on Game—but after DECOR composition effective active embedding usage rises to 51.06%, 100.00%, and 100.00%, respectively. This supports the claim that same-layer attention composition increases effective embedding utilization.

The method is intended to be lightweight relative to the backbone. The added costs are reported as $\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$2 per item for fusion, $\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$3 for context pooling, and $\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$4 per token for token composition. Because $\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$5 and $\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$6 are treated as constants, the paper states that overhead scales linearly with sequence length and remains small relative to Transformer self-attention,

$\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$7

(Liu et al., 22 Aug 2025).

5. Relation to adjacent decomposition-based token representation research

DECOR belongs to a larger set of attempts to make contextual token representations more structured, interpretable, or semantically stable, but adjacent methods decompose different objects and serve different end tasks.

In autoregressive language-model interpretability, “Token-wise Decomposition of Autoregressive LLM Hidden States for Analyzing Model Predictions” gives an exact additive decomposition of hidden states and logits by source token (Oh et al., 2023). For a target position $\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$8, the final logits are written as

$\mathbf{z}_0 = \text{Encoder}(\mathbf{x}),$9

and token influence is analyzed by subtracting $$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$0 from the final logit vector. That method decomposes the realized forward pass of a decoder Transformer into token-attributable hidden-state and logit contributions. Its primary use is prediction analysis, not adaptive token representation learning.

In encoder-side explanation, DecompX propagates decomposed token representations through attention, residual, layer normalization, feed-forward layers, and the classification head of encoder-only Transformers (Modarressi et al., 2023). Its core identity is

$$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$1

This is again a representation decomposition, but it is aimed at class-specific attribution rather than semantic-ID recommendation.

In lexical semantic representation learning, “Injecting Wiktionary to improve token-level contextual representations using contrastive learning” targets a different failure mode: standard contextual embeddings are too shaped by local sentential context for sense comparison (Mosolova et al., 2024). That work does not explicitly factor a token vector into separate latent components; instead it fine-tunes bert-base-uncased with supervised contrastive learning over lexicographic example sentences. A central result is that fine-tuning plus PCA with whitening to 100 dimensions yields 71.4% on the original WiC test set, compared with 69.6% for MirrorWiC, in the unsupervised setting. The overlap with DECOR lies in the goal of making token neighborhoods more semantically stable rather than in explicit factorization.

In structured contextualization for language modeling, “Contextually Structured Token Dependency Encoding for LLMs” augments token embeddings with a learned dependency matrix $$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$2 (Blades et al., 30 Jan 2025). The basic update is

$$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$3

The paper can be read as a partial decomposition into base token content and dependency-structured context, but it does not define distinct subspaces or disentangled components in the stronger sense. Its reported evidence includes improvements in dependency consistency that grow with sequence length, from 4.5% at 20 tokens to 42.6% at 200 tokens.

Taken together, these papers indicate that “decomposition” in token representation research is not a single method family but a spectrum. DECOR in recommendation separates semantic prior and collaborative adaptation; the autoregressive and encoder interpretability papers separate source-token contributions; the lexical semantic work reshapes token geometry without explicit factorization; and the dependency-encoding model injects a relational component into otherwise standard token states. This suggests that DECOR’s distinctive feature is not decomposition alone, but decomposition coupled to contextual reinterpretation of fixed semantic IDs.

6. Terminology, scope, and limitations

A recurring source of confusion is that DECOR is also the title of a distinct text-to-image customization method, “DECOR: Decomposition and Projection of Text Embeddings for Text-to-Image Customization” (Jang et al., 2024). That paper uses the acronym for DECOmposition and pRojection, not for Decomposed Contextual Token Representations. Its setting is LoRA-based text-to-image customization, where the method analyzes a CLIP text encoder output matrix

$$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$4

decomposes it with singular value decomposition, and applies the inference-time projection

$$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$5

to suppress unwanted semantics in the LoRA branch. That method is therefore about conditioning refinement for diffusion models, not sequential recommendation.

The recommendation DECOR has several explicit limits. It still depends on a pretrained semantic tokenizer; if the initial tokenizer is poor, DECOR adapts embeddings but does not change the semantic ID assignments themselves (Liu et al., 22 Aug 2025). Performance is sensitive to the composition strength $$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$6: the paper reports that moderate values, particularly $$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$7–$$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$8, generally work best, whereas very high $$\mathbf{z}_l = \mathbf{z}_{l-1} - q_l(\mathbf{z}_{l-1}), \quad l=1,\dots,L,$$9 such as $$q_l(\mathbf{z}_{l-1}) = \arg\min_{\mathbf{e}\in \mathbf{C}_l}\|\mathbf{z}_{l-1}-\mathbf{e}\|_2.$$0 can cause sharp degradation because the model over-relies on composition and individual token embeddings become undertrained. The method also introduces extra modules and hyperparameters, especially BOS query count and fusion projections. Finally, the reported evaluation is confined to Amazon sequential recommendation datasets; broader generalization to other domains or extreme cold-start settings is not directly established.

Within those limits, DECOR contributes a precise architectural answer to the two-stage misalignment problem in generative recommendation. It preserves pretrained semantics through frozen RQ-VAE codebooks, adds a separate collaborative embedding space, fuses those two streams explicitly, and dynamically composes same-layer token alternatives conditioned on user history (Liu et al., 22 Aug 2025). The resulting representation is neither a static semantic lookup nor an end-to-end retokenized code. It is a token state with two named sources of information—semantic and collaborative—and a context-conditioned realization at inference and training time.