Improving LLM-based Recommendation with Self-Hard Negatives from Intermediate Layers

Abstract: LLMs have shown great promise in recommender systems, where supervised fine-tuning (SFT) is commonly used for adaptation. Subsequent studies further introduce preference learning to incorporate negative samples into the training process. However, existing methods rely on sequence-level, offline-generated negatives, making them less discriminative and informative when adapting LLMs to recommendation tasks with large negative item spaces. To address these challenges, we propose ILRec, a novel preference fine-tuning framework for LLM-based recommendation, leveraging self-hard negative signals extracted from intermediate layers to improve preference learning. Specifically, we identify self-hard negative tokens from intermediate layers as fine-grained negative supervision that dynamically reflects the model's preference learning process. To effectively integrate these signals into training, we design a two-stage framework comprising cross-layer preference optimization and cross-layer preference distillation, enabling the model to jointly discriminate informative negatives and enhance the quality of negative signals from intermediate layers. In addition, we introduce a lightweight collaborative filtering model to assign token-level rewards for negative signals, mitigating the risk of over-penalizing false negatives. Extensive experiments on three datasets demonstrate ILRec's effectiveness in enhancing the performance of LLM-based recommender systems.

• The paper introduces ILRec, which mines token-level self-hard negatives from intermediate LLM layers to supply richer training signals.
• ILRec integrates cross-layer preference optimization, distillation, and collaborative reward regularization to boost ranking accuracy.
• Experiments on Amazon Review datasets show ILRec outperforms DPO-based methods in Hit@K and NDCG@K metrics, while reducing training overhead.

Improving LLM-based Recommender Systems via Self-Hard Negatives from Intermediate Layers

Introduction

LLMs have emerged as a viable paradigm for sequential recommendation tasks due to their expressive sequence modeling and instruction-following abilities. However, aligning LLMs to complex recommendation spaces, especially in large negative item scenarios, remains suboptimal because existing fine-tuning and alignment methods largely use static, coarse sequence-level negative examples with limited informativeness and adaptivity. The paper "Improving LLM-based Recommendation with Self-Hard Negatives from Intermediate Layers" (2602.17410) introduces the ILRec framework, which systematically addresses these shortcomings by leveraging token-level, dynamically self-generated hard negatives extracted from intermediate layers of LLMs. This approach yields more informative supervision, enhances learning efficiency, and empirically outperforms established baselines in full-candidate and candidate-ranking recommendation scenarios.

Figure 1: ILRec addresses the indiscriminativeness, uninformative nature, and inefficiency of traditional negative sampling by extracting fine-grained and informative negative signals.

Motivation and Analysis

Existing LLM-based recommendation methods rely on policy optimization algorithms such as DPO and their variants, leveraging negatives that are either randomly sampled or generated from prior policy snapshots. This strategy suffers from several key limitations:

• Indiscriminativeness: Sparse sequence-level negatives do not sufficiently regularize or shape the output distribution, particularly problematic as the item space grows.
• Uninformativeness: Negatives quickly become stale, not reflecting the current model's weaknesses or learning trajectory, thus offering diminishing returns as models are trained.
• Inefficiency: Additional computational and alignment steps are required to sample and utilize negatives, often at significant cost.

ILRec proposes to mine self-hard negatives from intermediate LLM layers. These layers, although weaker than the final output, encode competitive hypotheses that act as naturally hard (but not degenerate) negatives for the final layer’s optimization.

The ILRec Framework

ILRec introduces a two-stage training procedure encompassing (1) token-level self-hard negative extraction from intermediate layers and (2) cross-layer preference fine-tuning. An auxiliary lightweight CF model is further introduced for collaborative reward regularization, mitigating over-penalization of false negatives.

Figure 2: The ILRec training framework, including self-hard negative extraction, cross-layer preference optimization, distillation, and collaborative reward regularization.

Self-Hard Negative Extraction

For each prediction step, ILRec ensembles logits across $k$ consecutive intermediate layers directly below the output layer, producing a soft token distribution. High-probability tokens (excluding ground-truth) whose values exceed a dynamic threshold proportional to the ground-truth’s probability are selected as fine-grained self-hard negatives. This design offers:

• Dense, token-level supervision that reflects candidate ambiguity in item decoding.
• Adaptive difficulty as intermediate layers track the model’s current learning state.

Cross-Layer Preference Fine-Tuning

ILRec integrates two synergistic mechanisms:

• Cross-Layer Preference Optimization (CPO): Penalizes the final layer’s output probability for self-hard negatives via a modified cross-entropy loss, with penalty weights reflecting their hardness.
• Cross-Layer Preference Distillation (CPD): Aligns intermediate layers’ token distributions to that of the final layer via KL divergence, ensuring extracted negatives remain informative and synchronized with final-layer learning.

Collaborative Reward Regularization (CRR)

To mitigate the potential penalization of tokens correlated with user preference (so-called "false negatives"), CRR employs a CF model to score each penalized token. Token-level rewards derived from CF statistics modulate penalty intensity, integrating collaborative signals and enhancing the negative mining process.

Experimental Results

ILRec was benchmarked on multiple Amazon Review datasets (Musical Instruments, Arts, Video Games), demonstrating consistent improvements across Hit@K and NDCG@K metrics under both BIGRec and LC-Rec paradigms. ILRec achieves substantial gains over sequence-level DPO-based optimizers such as SDPO and SPRec, particularly in full ranking scenarios with large negative spaces.

Ablation and Analytical Studies

Ablations indicate that all core modules (CPO, CPD, CRR) significantly contribute to performance. The choice of intermediate layer depth is crucial: utilizing several of the deepest intermediate layers yields the most reliable hard negatives, corroborated by cross-backbone experiments (LLama-3.1-8B, Qwen2-0.5B, etc.). ILRec consistently outperforms alternative strategies of using only the final output or shallow layer negatives.

Figure 3: ILRec yields consistent gains across different LLM model backbones.

Figure 4: Performance is sensitive to the number of intermediate layers selected for negative extraction.

Extension to Candidate Ranking and Efficiency

ILRec also offers improvements on small-scale candidate ranking tasks, even when the available negative space is significantly reduced.

Figure 5: ILRec achieves higher accuracy on candidate-ranking tasks compared to SFT and DPO-based approaches.

Efficiency analysis reveals that ILRec, by integrating negative mining within the supervised fine-tuning process, requires fewer training epochs and dramatically reduces sampling overhead compared to DPO variants, without sacrificing performance.

Theoretical and Practical Implications

By introducing cross-layer, token-level negative mining, the proposed ILRec framework advances preference alignment for LLM-based recommenders on several dimensions:

• Theoretical: Positions intermediate layers as a latent curriculum for hard negative sampling, analogizing weak-to-strong supervision and model self-contradiction as useful training signals.
• Practical: Enables effective recommendation model training in massive item spaces, with improved scalability, sample efficiency, and ease of integration into standard SFT workflows.
• Future Directions: The modular nature of ILRec (CPO, CPD, CRR) provides a template for further research on self-supervised alignment signals and collaborative-integration in generative recommendation. Extending negative mining to other generative sequence models or conversational systems is a compelling direction.

Conclusion

ILRec demonstrates that mining self-hard negatives from intermediate LLM layers, combined with cross-layer optimization and collaborative reward regularization, delivers robust, scalable improvements for LLM-based recommender systems (2602.17410). This work underscores the importance of fine-grained, adaptive supervision for preference learning in large candidate spaces, and points to new possibilities for cross-layer self-improvement in deep sequence models.