SAILRec: Steering LLM Attention to Dual-Side Semantically Aligned Collaborative Embeddings for Recommendation

Abstract: Recent LLM-based recommenders enhance LLMs with collaborative embeddings from user-item interactions, but making such embeddings available does not ensure their proper use during inference. Through a diagnostic attention analysis, we find that the utilization of collaborative embeddings is depth-dependent and alignment-sensitive, suggesting that LLMs need to balance their internal semantic knowledge with external collaborative knowledge. To address this issue, we propose SAILRec, an LLM-based recommender that improves this balance through dual-side semantic alignment and hierarchical attention steering. The former aligns item-side embeddings with item-text semantics and user-side embeddings with codebook-based semantic profiles, while the latter suppresses premature shallow-layer collaborative interference and strengthens collaborative evidence in deeper decision layers. Experiments on MovieLens-1M and Amazon-Book show that SAILRec consistently outperforms representative baselines, with ablation and masking analyses validating its key designs.

• The paper introduces a framework that aligns user and item collaborative embeddings with LLM semantics via dual-side alignment and hierarchical attention steering.
• It demonstrates that steering attention from shallow to deep layers enhances collaborative signal integration and improves performance on datasets like MovieLens-1M and Amazon-Book.
• Experiments reveal that removing either alignment or steering components significantly degrades performance, underscoring the framework’s effectiveness in leveraging collaborative knowledge.

SAILRec: Dual-Side Semantic Alignment and Hierarchical Attention Steering for LLM-Based Recommendation

Introduction

LLMs are increasingly used for recommendation due to their semantic understanding and in-context reasoning capabilities. However, LLMs are inherently limited in modeling collaborative information—signals arising from user-item interactions typical of collaborative filtering (CF) approaches—because user and item IDs lack intrinsic semantic meaning to the LLM. Although previous works inject external collaborative embeddings as input tokens, these embeddings often remain semantically misaligned with the LLM and may be underutilized without effective cross-modal integration strategies. "SAILRec: Steering LLM Attention to Dual-Side Semantically Aligned Collaborative Embeddings for Recommendation" (2606.04514) systematically addresses this bottleneck by proposing a unified framework that jointly performs dual-side semantic alignment (on both user and item collaborative embeddings) and hierarchical attention steering within an LLM-based recommender, yielding strong empirical improvements and nuanced insights into collaborative knowledge utilization.

Methodological Framework

Diagnostic Insights: Collaboration is Depth-Dependent and Alignment-Sensitive

Through attention analysis on MovieLens-1M, the paper demonstrates that naive collaborative embedding injection leads to shallow and suboptimal attention utilization. Specifically, collaborative embedding attention is suppressed until intermediate-to-deep transformer layers, and their effective contribution relies on semantic alignment.

Figure 1: Mean attention from the answer position to key token groups under different semantic alignment settings on MovieLens-1M. Dual-side alignment yields deeper, more balanced collaborative attention.

This analysis motivates two core questions for LLM-based recommendation: (1) Are external collaborative embeddings understandable within the LLM's semantic space? (2) At which layers should collaborative information contribute to prediction?

SAILRec Architecture

SAILRec answers these with a pipeline that combines dual-sided semantic alignment and hierarchical, layer-wise attention control.

Figure 2: Overall architecture of SAILRec showing dual-side alignment, attention steering, and collaborative embedding injection.

Dual-Side Semantic Alignment:

• User-Side: Since user IDs lack explicit semantics, user collaborative embeddings are aligned with codebook-constructed semantic profiles. These profiles are built via aggregation of historical user interactions using three phrase-level semantic codebooks covering style, emotion, and ideology. Alignment is achieved via a lightweight, query-based "Collaborative Q-Former" (C-QFormer) using slotwise InfoNCE loss.
• Item-Side: Item collaborative embeddings are mapped to the LLM semantic space by aligning with frozen LLM encodings of item texts (titles), thus directly grounding collaborative signals with item semantics via InfoNCE-based contrastive learning.

  Figure 3: Example codebook tags for MovieLens-1M illustrating multi-aspect codebook structure (style, emotion, ideology) used for user-side semantic alignment.

Hierarchical Attention Steering:

• Transformer layers are partitioned into shallow (early), middle, and deep (top) groups.
• Attention to collaborative tokens is suppressed in shallow layers to avoid interfering with lexical and basic contextual modeling, unmodified in middle layers to allow natural interaction, and enhanced (via positive bias) in deep layers to strengthen collaborative evidence immediately prior to prediction.

Training Regimen:

• Three-stage: (1) Train/freeze CF model (matrix factorization), (2) Warm up C-QFormers via dual-side contrastive alignment, (3) Supervised fine-tuning with LoRA adaptation, jointly optimizing LLM and C-QFormers under attention steering.

  Figure 4: Sample prompt for SAILRec training, indicating explicit collaborative token positions and semantic anchors.

Layer-Wise Collaborative Utilization

Layer-wise probe analyses validate that attention steering combined with semantic alignment robustly improves deep-layer collaborative utilization, balancing collaborative and semantic signals for effective user-item matching.

Figure 5: Layer-wise attention to semantic and collaborative tokens in SAILRec on MovieLens-1M, illustrating controlled and depth-selective collaborative integration.

Figure 6: Layer-wise answer attention to semantic and collaborative tokens in SAILRec on Amazon-Book, illustrating comparable trends across domains.

Empirical Evaluation

Extensive experiments on MovieLens-1M and Amazon-Book demonstrate:

• Superior overall performance: SAILRec outperforms competitive CF, LLM-only, and collaborative-enhanced LLM baselines on AUC, UAUC, NDCG, and MAP by up to 2.77 percentage points on UAUC and 1.69 on MAP (MovieLens-1M), and similar or larger margins on Amazon-Book.
• Robustness in warm/cold splits: Gains are largest on warm-start scenarios where collaborative evidence is plentiful, but SAILRec retains strong (and often best) competitive performance in cold-start regimes by leveraging semantic alignment.

  Figure 7: Warm/cold start performance; SAILRec provides strong gains under warm settings, competitive in cold.

• Ablation and masking analyses: Removing user-side alignment, item-side alignment, codebook structure, or hierarchical attention steering consistently degrades performance; inference-time masking of collaborative token attention leads to large drops, particularly on sparse domains and for user-side masking (e.g., >0.27 AUC loss on Amazon-Book when masking user-side tokens).
• Steering schedule analysis: Only the three-stage S-N-E (suppress–none–enhance) schedule consistently achieves highest and most stable user-discriminative performance; single-strategy or inverted schedules underperform.

Qualitative and Representation Analysis

Visualization of learned collaborative tokens and alignment targets (e.g., t-SNE, heatmaps) reinforces that dual-side semantic alignment encourages interpretable and specialized representations for user and item collaborative tokens. User-side C-QFormer tokens cluster along distinct semantic axes (style, emotion, ideology), and alignment with codebook-anchored targets is visually and quantitatively clear.

Figure 8: t-SNE visualization shows differentiated user-side collaborative tokens along three codebook directions.

Implications and Future Directions

Practical and Theoretical Implications

• Semantic-bridged collaborative integration: The results indicate that collaborative embedding integration with LLMs should not be approached as naive concatenation; instead, semantic bridging via explicit alignment and steerable attention unlocks collaborative gains while retaining semantic modeling capacity.
• Layer-wise utilization as inductive bias: The observed "collaborative late, semantic early" principle matches known patterns in transformer-based language modeling, suggesting a universal design motif when integrating non-linguistic information into LLMs.
• Personalized, interpretable recommendation: The codebook-based user-side alignment allows for interpretable and modular user preference modeling, mediating between opaque embeddings and explicit semantic anchors.

Limitations and Research Directions

• Predefined attention schedules may be suboptimal across domains or samples; adaptive, data- or sample-dependent steering could further improve performance.
• Collaborative token interpretability remains limited by their continuous embedding nature; development of explicitly interpretable or discretized collaborative representations (cf. approaches like TokenRec and BinLLM) is a promising future direction.
• Results call for further backbone-specific tuning and analysis, particularly for newer LLM families (e.g., Qwen3), to fully harness the benefits of semantic alignment and attention control.
• The modularity of the SAILRec approach may facilitate its extension to multi-task or multi-modal recommendation and interactive/causal recommendation settings.

Conclusion

SAILRec establishes a principled framework for collaborative information utilization in LLM-based recommendation, combining dual-side semantic alignment with hierarchical attention steering for robust, interpretable, and effective integration of collaborative signals. Empirical and qualitative analyses confirm that SAILRec achieves consistent improvements over state-of-the-art baselines in both dense and sparse, warm and cold scenarios, setting a new bar for integrating collaborative and semantic knowledge under the LLM recommendation paradigm.