Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation (2110.05730v2)

Abstract: Recent advancements of sequential deep learning models such as Transformer and BERT have significantly facilitated the sequential recommendation. However, according to our study, the distribution of item embeddings generated by these models tends to degenerate into an anisotropic shape, which may result in high semantic similarities among embeddings. In this paper, both empirical and theoretical investigations of this representation degeneration problem are first provided, based on which a novel recommender model DuoRec is proposed to improve the item embeddings distribution. Specifically, in light of the uniformity property of contrastive learning, a contrastive regularization is designed for DuoRec to reshape the distribution of sequence representations. Given the convention that the recommendation task is performed by measuring the similarity between sequence representations and item embeddings in the same space via dot product, the regularization can be implicitly applied to the item embedding distribution. Existing contrastive learning methods mainly rely on data level augmentation for user-item interaction sequences through item cropping, masking, or reordering and can hardly provide semantically consistent augmentation samples. In DuoRec, a model-level augmentation is proposed based on Dropout to enable better semantic preserving. Furthermore, a novel sampling strategy is developed, where sequences having the same target item are chosen hard positive samples. Extensive experiments conducted on five datasets demonstrate the superior performance of the proposed DuoRec model compared with baseline methods. Visualization results of the learned representations validate that DuoRec can largely alleviate the representation degeneration problem.

• The paper proposes DuoRec, which integrates contrastive learning to tackle representation degeneration in sequential recommendation.
• It utilizes model-level augmentation with Dropout to generate semantically faithful item embeddings, outperforming SASRec and BERT4Rec.
• Experimental results across multiple benchmarks show that DuoRec achieves a more uniform embedding distribution and significant gains in HR and NDCG metrics.

Analyzing the Impact of Contrastive Learning on Representation Degeneration in Sequential Recommendation

The paper presents a focused examination of the representation degeneration problem that arises in sequential recommendation systems, specifically addressing the issue with an innovative approach using contrastive learning. The primary concern highlighted is the tendency for item embeddings to converge into an anisotropic shape when processed by advanced deep learning models such as Transformers and BERT. This convergence results in item embeddings displaying high semantic similarities, which can impede the ability of recommendation models to differentiate between items effectively.

Key Contributions

The authors propose a novel model named DuoRec, which integrates contrastive learning to mitigate the representation degeneration problem. The model modifies the item embeddings' distribution, striving for a more uniform representation. Central to DuoRec is the implementation of a contrastive regularization mechanism that stems from both theoretical foundations and empirical discoveries.

Critically, the paper sheds light on how existing contrastive learning methodologies often rely on data-level augmentations—commonly involving item cropping or reordering—which may fail to maintain semantic consistency. In response, DuoRec employs a model-level augmentation using Dropout, enabling the generation of more semantically faithful representations. A novel sampling strategy is also introduced, targeting sequences that share the same target item as hard positive samples to enhance training effectiveness.

Experimental Insights and Results

The paper extensively tests DuoRec across five benchmark datasets, including subsets of the Amazon dataset and the Yelp dataset, highlighting its robust performance. DuoRec demonstrates superior efficacy when compared to baseline models, such as SASRec and BERT4Rec, achieving significant improvements in metrics like HR@5, HR@10, NDCG@5, and NDCG@10 across various datasets. These results underscore the model's ability to enhance the effectiveness of recommendation systems significantly.

Visualizations utilized within the experimental analysis illustrate that DuoRec achieves a more uniform distribution of item embeddings. This shift is evidenced by slower decreases in singular values when compared to traditional methods, validating the model's capability in addressing the anisotropic distribution observed in earlier models.

Theoretical and Practical Implications

The paper's focus on tackling the representation degeneration problem through contrastive learning offers both theoretical and practical implications. Theoretically, it advances the understanding of how item embeddings behave under current sequential models and how contrastive learning can be harnessed to reshape these representations. Practically, DuoRec provides a framework for improving sequential recommendation systems, offering a pathway to more precise and semantically enriched item recommendations.

Moreover, the incorporation of both supervised and unsupervised positive sampling within contrastive learning contexts serves as a potential inspiration for future research. This dual approach showcases how integrating semantic information can further refine the quality of sequence embeddings.

Future Directions

The research opens avenues for further exploration into augmenting sequential recommendation frameworks with novel contrastive learning techniques. Future work could investigate additional augmentation strategies that maintain semantic consistency across various data domains or explore hybrid models that synergize contrastive learning with other learning paradigms. Additionally, the scalability and computational efficiency of such models could be assessed, particularly in large-scale recommendation scenarios.

In summary, the paper makes significant strides in addressing the representation degeneration problem in sequential recommendation systems. Through utilising contrastive learning, DuoRec not only enhances embedding distribution but also sets a benchmark for upcoming advancements in recommendation systems, strengthening both their theoretical foundations and practical applications.