SAILRec: LLM-based Semantic Recommender

• The paper introduces SAILRec which integrates collaborative filtering with LLMs via dual-side semantic alignment and hierarchical attention steering for binary click prediction.
• It employs a dual-side alignment mechanism that maps user and item embeddings into the LLM’s semantic space using a collaborative Q-Former, enabling precise preference matching.
• Empirical results on MovieLens-1M and Amazon-Book demonstrate consistent improvements in AUC, UAUC, NDCG, and MAP compared to baseline methods.

SAILRec is an LLM-based recommender designed for binary click prediction in settings where a user’s interaction history and a candidate item are presented in a prompt, and the model predicts whether the user would click or like the item. Its central premise is that enhancing a LLM with collaborative embeddings from user-item interactions does not by itself ensure that those embeddings will be used properly during inference. The framework therefore combines dual-side semantic alignment with hierarchical attention steering so that collaborative signals become both semantically interpretable to the LLM and selectively emphasized at the layers where they are most useful for preference matching and decision-making (Wu et al., 3 Jun 2026).

1. Problem formulation and motivation

SAILRec addresses a specific failure mode in LLM-based recommendation: the mismatch between the availability of collaborative knowledge and the actual utilization of that knowledge inside the Transformer. Prior LLM-based recommenders can leverage text semantics effectively, but they are weaker at exploiting collaborative signals derived from user-item interaction data. In this formulation, the recommendation task is binary click prediction, and the LLM predicts logits at the answer position for “Yes” and “No” (Wu et al., 3 Jun 2026).

The framework is motivated by the observation that prior methods such as CoLLM inject collaborative embeddings as soft tokens, while SeLLa-Rec shows that semantic alignment can reduce the gap between collaborative and language spaces. SAILRec argues that a further issue remains: even if collaborative embeddings are inserted and aligned, the model still requires control over when those embeddings influence inference. This leads to two design requirements: collaborative embeddings must be understandable within the LLM’s semantic space, and their influence must be regulated across layers rather than applied uniformly (Wu et al., 3 Jun 2026).

The architecture therefore consists of three components: a collaborative filtering model that produces user and item collaborative embeddings, dual-side semantic alignment that maps those embeddings into the LLM’s semantic space, and hierarchical attention steering that controls attention to the injected collaborative tokens across Transformer depth. The paper uses matrix factorization as the collaborative model and freezes the resulting user and item embeddings after collaborative pretraining (Wu et al., 3 Jun 2026).

2. Diagnostic attention analysis

A distinguishing feature of SAILRec is that its architectural choices are explicitly motivated by a diagnostic attention analysis on MovieLens-1M. The analysis inspects attention from the answer position to three token classes: user-side collaborative embeddings, item-side collaborative embeddings, and historical item-title tokens. The reported finding is that collaborative embeddings are used in a depth-dependent manner: attention to them is low in shallow layers and rises in middle and deep layers (Wu et al., 3 Jun 2026).

This depth profile is interpreted as evidence that collaborative knowledge is not primarily useful for early lexical or contextual encoding. Instead, it contributes more strongly in later stages associated with preference matching and final recommendation decisions. The analysis also finds that collaborative usage is alignment-sensitive. Without semantic alignment, attention patterns are less orderly; item-side-only alignment improves late-layer accessibility; and dual-side alignment yields the most balanced use, with stronger user-side attention at the top layers and better recommendation performance (Wu et al., 3 Jun 2026).

These observations directly motivate SAILRec’s two core mechanisms. Dual-side alignment is intended to make collaborative embeddings legible to the model’s semantic representations, while hierarchical steering is intended to suppress collaborative interference in shallow layers and strengthen collaborative evidence in deep layers. A plausible implication is that SAILRec is not merely an embedding-injection method; it is a method for coordinating semantic and collaborative evidence over depth.

3. Model architecture and recommendation objective

The collaborative backbone in SAILRec is matrix factorization, which produces frozen user and item embeddings

$$\mathbf{e}^{c}_{u}\in\mathbb{R}^{d_c},\qquad \mathbf{e}^{c}_{i}\in\mathbb{R}^{d_c}.$$

These embeddings are the source of collaborative knowledge injected into the LLM (Wu et al., 3 Jun 2026).

The recommendation objective is defined through answer-position logits for “Yes” and “No”: $$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$ Training uses binary cross-entropy: $$\begin{aligned} \mathcal{L}_{\mathrm{rec}} = {} & -y_{u,i}\log \hat{y}_{u,i} \ & -(1-y_{u,i})\log(1-\hat{y}_{u,i}). \end{aligned}$$ This formulation keeps the recommendation task in a standard supervised setting while using the LLM as the main inference engine over textual and collaborative context (Wu et al., 3 Jun 2026).

The operational pipeline is three-stage. First, matrix factorization is trained on recommendation data and then frozen. Second, the user and item mapping modules are warmed up using contrastive semantic alignment. Third, the LLM is supervised fine-tuned with LoRA while jointly optimizing the mapping modules under the hierarchical attention steering mechanism (Wu et al., 3 Jun 2026).

The paper uses Qwen2-7B as the backbone LLM, a collaborative embedding dimension of 256, and an LLM hidden size of 3584. Additional reported implementation details include a stage-2 alignment learning rate of 1e-3, alignment batch size 128, InfoNCE temperature 0.07, stage-3 learning rate chosen from $8\times10^{-5}$ to $2\times10^{-4}$, weight decay 0.01, Adam $\beta_2$ of 0.95, warmup ratio 0, max sequence length 768, MovieLens-1M batch size 256 with 2 epochs, Amazon-Book batch size 2000 with 1 epoch, and NVIDIA A6000 GPUs for hardware (Wu et al., 3 Jun 2026).

4. Dual-side semantic alignment

Dual-side semantic alignment is the first major mechanism in SAILRec. It is “dual-side” because user and item collaborative embeddings are aligned to semantic targets in different ways, reflecting the asymmetry between items, which usually have interpretable text semantics, and users, whose IDs are anonymous (Wu et al., 3 Jun 2026).

On the item side, the target representation is the LLM’s semantic embedding of the item title. The item-side collaborative embedding is passed through a lightweight mapper based on a Q-Former-like module, called the Collaborative Q-Former, or C-QFormer. This produces a high-dimensional token aligned with the LLM hidden size. The mapped item token is then aligned to item-title semantics with InfoNCE. The stated rationale is that the LLM already has reasonably good semantic knowledge about item titles, so collaborative item vectors should be grounded in that representation (Wu et al., 3 Jun 2026).

On the user side, the problem is harder because a user ID has no direct linguistic meaning. SAILRec addresses this by building semantic user profiles from historical interactions using three phrase-level codebooks: style, emotion, and ideology. For each semantic direction $r\in\{A,B,C\}$, the codebook is

$\mathcal{C}^{r}=\{c^{r}_{1},\ldots,c^{r}_{K_r}\}.$

Historical items are weighted by position and time decay: $$w_j=w^{\mathrm{pos}_j}w^{\mathrm{time}_j},\qquad \tilde{w}_j=\frac{w_j}{\sum_{\ell=1}^{T}w_{\ell}}.$$ The model then aggregates tag votes and selects the top tag for each direction: $$\begin{aligned} S^{r}_{u}(c) &= \sum_{j=1}^{T} \tilde{w}_{j} \mathbb{I}\left[\phi^{r}(i^{u}_{j})=c\right], \ p^{r}_{u} &= \arg\max_{c\in\mathcal{C}^{r}} S^{r}_{u}(c). \end{aligned}$$ This yields the semantic profile

$$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$0

The three profile tags are encoded by the frozen LLM semantic encoder to produce targets

$$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$1

and the user-side C-QFormer maps the frozen matrix-factorization embedding into three aligned collaborative tokens

$$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$2

Alignment is then performed with slot-wise InfoNCE: $$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$3 The final user-side alignment loss averages across the three semantic directions (Wu et al., 3 Jun 2026).

The conceptual role of this design is explicit in the paper: rather than aligning a user embedding to a single vague textual proxy, SAILRec aligns it to a structured semantic profile capturing different facets of preference. The ablation study further reports that removing codebook-based user profiling and replacing it with a single whole-history representation degrades performance, indicating that structured semantic anchors are more effective than a monolithic sequence embedding (Wu et al., 3 Jun 2026).

5. Hierarchical attention steering

Hierarchical attention steering is SAILRec’s second major mechanism and governs how collaborative tokens are used across Transformer depth. The layers are partitioned into shallow, middle, and deep groups. Attention is computed as

$$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$4

where $$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$5 is the causal mask and $$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$6 is an attention bias applied only to collaborative token positions $$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$7 (Wu et al., 3 Jun 2026).

In shallow layers, SAILRec applies suppression: $$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$8 This negative bias discourages early reliance on collaborative tokens so that the model can first process lexical and contextual semantics from the prompt (Wu et al., 3 Jun 2026).

In middle layers, no steering is applied, so $$\hat{y}_{u,i} = \frac{\exp(o_{\text{Yes}})}{\exp(o_{\text{Yes}})+\exp(o_{\text{No}})}.$$9. This leaves the model free to combine textual and collaborative information without intervention (Wu et al., 3 Jun 2026).

In deep layers, SAILRec applies enhancement: $$\begin{aligned} \mathcal{L}_{\mathrm{rec}} = {} & -y_{u,i}\log \hat{y}_{u,i} \ & -(1-y_{u,i})\log(1-\hat{y}_{u,i}). \end{aligned}$$0 The deep bias $$\begin{aligned} \mathcal{L}_{\mathrm{rec}} = {} & -y_{u,i}\log \hat{y}_{u,i} \ & -(1-y_{u,i})\log(1-\hat{y}_{u,i}). \end{aligned}$$1 is depth-dependent; the paper uses a triangular schedule and increases the positive bias toward the top layers. The reported hyperparameters are a shallow layer ratio of 30%, a deep layer ratio of 20%, shallow bias $$\begin{aligned} \mathcal{L}_{\mathrm{rec}} = {} & -y_{u,i}\log \hat{y}_{u,i} \ & -(1-y_{u,i})\log(1-\hat{y}_{u,i}). \end{aligned}$$2, and maximum deep bias 2.0 (Wu et al., 3 Jun 2026).

The steering schedule analysis compares suppression, no steering, and enhancement patterns across depth. The default SAIL-S-N-E schedule—suppress in shallow layers, leave middle layers untouched, enhance in deep layers—is reported as best overall, while uniform steering across all layers performs worse. This supports the claim that collaborative information should be introduced selectively rather than uniformly (Wu et al., 3 Jun 2026).

Masking studies reinforce this interpretation. By setting attention bias to $$\begin{aligned} \mathcal{L}_{\mathrm{rec}} = {} & -y_{u,i}\log \hat{y}_{u,i} \ & -(1-y_{u,i})\log(1-\hat{y}_{u,i}). \end{aligned}$$3 for selected collaborative token paths during inference, the paper tests whether specific paths matter. Masking user-side or item-side collaborative tokens degrades performance, masking both degrades performance most, and masking shallow, middle, or deep access all causes deterioration in different ways. On Amazon-Book, masking user-side collaborative tokens causes AUC to fall from 0.8393 to 0.5553, which the paper interprets as evidence that user-side preference abstraction is especially important in sparse and long-tailed recommendation settings (Wu et al., 3 Jun 2026).

6. Empirical results, ablations, and limitations

SAILRec is evaluated on MovieLens-1M and Amazon-Book using AUC, UAUC, NDCG, and MAP. After preprocessing, MovieLens-1M contains 33,891 train, 10,401 valid, and 7,331 test instances, with 839 users and 3,256 items; Amazon-Book contains 727,468 train, 25,747 valid, and 25,747 test instances, with 22,967 users and 34,154 items. The baselines are grouped into collaborative filtering methods, LLM-only recommenders, and collaborative-enhanced LLM recommenders (Wu et al., 3 Jun 2026).

The main reported outcomes are summarized below.

Dataset	AUC	UAUC	NDCG	MAP
MovieLens-1M	0.7571	0.7410	0.8985	0.7999
Amazon-Book	0.8393	0.7506	0.9017	0.8524

On MovieLens-1M, SAILRec improves over the strongest baseline by +1.43 AUC points, +2.77 UAUC points, +1.00 NDCG points, and +1.69 MAP points. On Amazon-Book, the gains are +2.02 AUC points, +1.96 UAUC points, +0.77 NDCG points, and +1.05 MAP points. The paper emphasizes that the improvements are consistent across both datasets and all metrics (Wu et al., 3 Jun 2026).

The ablation study attributes these gains to the combination of alignment and steering. Removing hierarchical attention steering degrades performance, showing that aligned embeddings alone are insufficient. Removing both user- and item-side alignment causes a substantial drop. Removing only user- or only item-side alignment also harms results, with user-side alignment described as especially important because user IDs are not linguistically grounded. Training ablations further show that freezing the mapping modules and training only LoRA, or updating mapping modules without proper joint optimization with LoRA, is weaker than the full model (Wu et al., 3 Jun 2026).

Qualitative analyses in the appendix include t-SNE visualizations and similarity heatmaps. These show that the three user-side collaborative tokens are not redundant, that they occupy distinct regions in embedding space, that item-side tokens align more clearly with item-title semantics, and that user-side tokens have weaker but differentiated semantic associations. This suggests that dual-side alignment creates semantically distinct collaborative representations rather than merely duplicating one latent factor across multiple tokens (Wu et al., 3 Jun 2026).

Two limitations are explicitly noted. First, the steering schedule is fixed; the layer-wise bias schedule is predefined and may not be optimal for every sample or backbone. Second, although the collaborative tokens are semantically aligned, they remain continuous embeddings rather than directly human-readable concepts. The paper therefore suggests future work on sample-adaptive or continuous steering and on improving interpretability of collaborative token representations (Wu et al., 3 Jun 2026).