Text-Enhanced Facet-Aware Pre-training

• The paper introduces a pre-training module that initializes item representations with facet-specific textual features to mitigate cold-start and semantic gap issues.
• It leverages a frozen Transformer text encoder, a shared projection layer, and alternating supervised contrastive learning to disentangle multi-facet attributes.
• Empirical results on datasets like ML-20m demonstrate notable gains in HR@20 and NDCG@20 compared to conventional random ID initialization.

Searching arXiv for the specified paper and closely related work on text-enhanced or facet-aware pre-training. Text-Enhanced Facet-Aware Pre-training is a pre-training module for sequential recommendation that initializes item representations from textual metadata in a way that is explicitly aligned with a downstream multi-facet architecture. Introduced as part of FAME+, the approach addresses two linked limitations of conventional sequential recommendation pipelines: randomly initialized ID embeddings do not encode item content, and a single embedding vector is often insufficient to represent the multi-faceted nature of items such as movies or products. Its central mechanism is to use a frozen pre-trained text encoder, a shared projection layer, and multiple facet-specific heads trained with alternating supervised contrastive learning so that each head captures a distinct semantic factor before sequential training begins (Liu et al., 18 Jan 2026).

1. Conceptual scope and problem formulation

In the FAME+ formulation, the immediate motivation is the cold-start and semantic-gap problem. Randomly initialized ID embeddings lack any notion of item content and are especially weak on long-tail items. At the same time, the downstream recommendation model assumes that users may express preferences over multiple facets of an item, such as movie genres and starring actors, or product category, brand, and price. A single static identifier embedding is therefore treated as an inadequate substrate for modeling varied user intent (Liu et al., 18 Jan 2026).

The pre-training module addresses this by explicitly carving an item’s textual metadata into $H$ separate facet subspaces. The intended downstream consequence is that each attention head in FAME begins from semantically meaningful features rather than from a content-agnostic lookup. After pre-training, the concatenation of these facet sub-embeddings replaces the usual ID embedding in the sequential recommender.

A defining property of the method is that it is not framed as generic language-model pre-training. The pre-training stage does not use masked language modeling or classification. Instead, it uses ground-truth categorical labels for each facet and optimizes a supervised contrastive objective separately for each facet. This makes the procedure explicitly disentangling rather than merely text-informed.

2. Representational architecture

The architecture starts from a frozen, pre-trained Transformer text encoder, exemplified in the paper by BERT-base-uncased. For each item $i$, textual metadata are assembled into a templated string such as “title:… description:… genres:… directors:… cast:…” or analogous fields on Amazon. The encoder produces a pooled $[\mathrm{CLS}]$ representation

$$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$

This vector is then mapped into the recommender space by a shared MLP layer,

$$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$

The shared projection is followed by $H$ independent linear heads, each responsible for one facet-specific subspace of dimension $D/H$:

$$\widetilde{\mathbf{z}}_i^{(h)} = \mathbf{W}^{(h)}\mathbf{h}_i + \mathbf{b}^{(h)}, \qquad \mathbf{z}_i^{(h)} = \frac{\widetilde{\mathbf{z}}_i^{(h)}}{\|\widetilde{\mathbf{z}}_i^{(h)}\|_2}\in\mathbb{R}^{D/H}.$$

The $\ell_2$ normalization places each facet vector on the unit hypersphere. In the stated design, this normalization stabilizes contrastive training and prevents magnitude from serving as a shortcut. After pre-training, the final item embedding is formed by concatenation,

$$\mathbf{e}_i' = \bigl[ \mathbf{z}_i^{(1)} \| \mathbf{z}_i^{(2)} \| \cdots \| \mathbf{z}_i^{(H)} \bigr]\in\mathbb{R}^{D}.$$

This architecture formalizes “facet awareness” as a decomposition of the item representation into disjoint learned subspaces rather than as post hoc interpretation of a monolithic embedding.

3. Alternating supervised contrastive pre-training

The pre-training objective is organized around facet-specific supervised contrastive learning. For each facet $i$0, every item has a categorical label $i$1. Mini-batches $i$2 are constructed for one facet at a time, and positives for an anchor item $i$3 are defined as other items in the batch that share the same facet label:

$i$4

All other samples in the batch serve as comparison items,

$i$5

The supervised contrastive loss for head $i$6 is

$i$7

A mask-based implementation is also described: a binary mask $i$8 iff $i$9 and $[\mathrm{CLS}]$0, and the numerator and denominator are computed with tensor-masked dot products (Liu et al., 18 Jan 2026).

A central procedural feature is alternation. The model does not optimize all $[\mathrm{CLS}]$1 simultaneously in one batch, because that would not guarantee positives for all facets. Instead, training cycles through the facets. At step $[\mathrm{CLS}]$2, the method samples a facet-specific batch, computes $[\mathrm{CLS}]$3, and back-propagates only through the shared projection and the current facet head while all other heads remain frozen. An epoch is considered complete after each facet has been updated once.

This alternating scheme is the core of the “text-enhanced” mechanism in FAME+: text provides the semantic source signal, but the pre-training objective forces that signal to separate according to facet labels before any sequential modeling is performed.

4. Disentanglement mechanism and optimization regime

The paper attributes facet disentanglement to three design choices rather than to an explicit orthogonality penalty. First, the projection heads are independent: each $[\mathrm{CLS}]$4 acts only on facet $[\mathrm{CLS}]$5, and there are no shared weights across facets at the projection stage. Second, hyperspherical normalization imposes $[\mathrm{CLS}]$6, so improvement must come from directional changes in each head. Third, alternation with stratified batch construction forces each head to specialize on one facet at a time (Liu et al., 18 Jan 2026).

This point is important because a common simplification would be to describe the method as learning orthogonal facets. The stated formulation is narrower. There are no explicit orthogonality constraints across heads; the claim is instead that the independent losses push facets to capture different factors of variation in practice.

Training uses a stratified $[\mathrm{CLS}]$7 sampler. For facet $[\mathrm{CLS}]$8, the method chooses $[\mathrm{CLS}]$9 distinct classes and samples $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$0 items per class, yielding batch size $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$1. The example configuration is $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$2, $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$3, $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$4, which guarantees $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$5 positives for every anchor. The training procedure cycles through all classes over epochs in a “fair” epoch strategy so that even rare classes appear.

The reported hyperparameters are tightly specified: number of facets $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$6 matched to downstream FAME, with an example $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$7; temperature $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$8; pre-training epochs $$\mathbf{e}_i = \mathrm{Encoder}(\mathcal{T}_i)\in\mathbb{R}^{D_T}.$$9; shared MLP hidden size $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$0; optimizer AdamW with learning rate $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$1 and weight decay $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$2; and no learning-rate scheduler. The use of a fixed small learning rate is stated explicitly.

Another common misconception is that the method is a general-purpose text encoder fine-tuned during recommendation training. The actual integration is more constrained. After pre-training, each item receives a fixed facet-aware vector $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$3, the text encoder and MLP are frozen, the projection weights are discarded, and only the resulting item embeddings are kept as a static lookup table for the downstream recommender.

5. Integration into sequential recommendation and empirical behavior

Within FAME, the pre-trained embedding $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$4 replaces the usual ID lookup $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$5. All subsequent attention, gating, and Mixture-of-Experts routing operate on these pre-trained features, and the whole FAME model is then fine-tuned end-to-end with cross-entropy loss (Liu et al., 18 Jan 2026).

The empirical evidence in the paper separates the effect of raw text initialization from facet-aware pre-training. On the ML-20m dataset, the main ablation is reported as follows.

Variant	HR@20	NDCG@20
FAME (ID-init)	0.3021	0.1383
FAME+ raw text	0.3097	0.1602
FAME+ facet aware	0.3397	0.1608

These numbers indicate that simple text initialization already improves over random ID initialization, but facet-aware structuring produces the largest gain in HR@20. The smaller change from $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$6 to $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$7 in NDCG@20 suggests that the pre-training effect is not uniform across ranking metrics.

The Beauty dataset ablation further distinguishes among facet combinations. Reported HR@20 / NDCG@20 values are $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$8 for FAME, $$\mathbf{h}_i = \sigma\bigl(\mathbf{W}_{\mathrm{shared}}\mathbf{e}_i + \mathbf{b}_{\mathrm{shared}}\bigr)\in\mathbb{R}^{D}.$$9 for FAME+ raw, $H$0 for FAME+ with $H$1, $H$2 for $H$3, and $H$4 for $H$5. In this setting, $H$6 is the best-performing pair. A plausible implication is that the usefulness of facet-aware pre-training depends on whether the chosen facets correspond to behaviorally salient distinctions in the downstream domain.

The paper also reports a representational analysis by t-SNE. Raw BERT embeddings remain highly entangled and show no clear clusters, whereas after pre-training, items sharing the same facet label form tight clusters in each subspace. At the system level, FAME+ is reported to consistently outperform all baselines by $H$7–$H$8 relative on HR@20/NDCG@20 across four datasets, and the pre-training module yields an extra $H$9–$D/H$0 lift over FAME with random initialization.

6. Relation to adjacent text-enhanced pre-training paradigms

Text-Enhanced Facet-Aware Pre-training belongs to a wider family of methods that use text as a supervisory source for structured representation learning, but its operational meaning of “facet” is domain-specific. In FAME+, facets are item metadata dimensions aligned with downstream attention heads. In other domains, related papers use comparable language for different representational partitions.

In facial-linguistic pre-training, GPTFace uses large-scale weakly correlated face-caption data and jointly optimizes masked image/language modeling and image-text matching. The paper states that span MILM forces recovery of attribute words such as “smiling,” “blonde hair,” and “beard,” while image-text matching tightens image-text alignment at the attribute level; it summarizes the result as learning fine-grained facet representations from weakly aligned web data (Li et al., 21 Oct 2025). In this context, “facet” denotes facial attributes rather than recommendation metadata.

In text-attributed heterogeneous graphs, THLM extends BERT pre-training with a context graph prediction task and neighbor-based text augmentation for text-poor nodes. Its objective combines MLM with binary classification of context-graph membership, and its text augmentation concatenates neighbor texts for nodes with limited textual content (Zou et al., 2023). This is not framed as facet disentanglement, but it shares the idea that textual signals can be reorganized to compensate for structural sparsity or semantic imbalance.

In sparse text-attributed graphs, S²Aligner explicitly adopts facet-aware factorization by decomposing graph-text representations into semantic and structural components, using structure-oriented reconstruction with a consistency gate, and adding sparsity-aware cross-domain risk balancing (Wang et al., 18 May 2026). Here the facets are “semantic” and “structural” branches, and the design is motivated by missing or noisy text anchors rather than by multiple item attributes.

In multimodal document and scene-text understanding, TAP incorporates OCR tokens as an explicit modality and pre-trains with MLM, image-text matching, and relative position prediction in order to align question words, object regions, and scene text (Yang et al., 2020). Its notion of text awareness is tri-modal grounding rather than facet disentanglement, but it illustrates a closely related principle: when the downstream task depends on a particular semantic aspect of the input, pre-training can be made more effective by structuring text-conditioned representations around that aspect.

Taken together, these works suggest that “text-enhanced” and “facet-aware” are not fixed architectural labels. They designate a design pattern in which text is used to impose semantically meaningful partitions on learned representations before downstream optimization. In FAME+, that partition is operationalized through $D/H$1 supervised contrastive subspaces aligned with recommendation heads; in adjacent literature, the same general principle appears as facial attribute alignment, semantic-structural decomposition, topology-aware augmentation, or scene-text grounding.