Dual Correction Strategy for Ranking Distillation in Top-N Recommender System (2109.03459v4)
Abstract: Knowledge Distillation (KD), which transfers the knowledge of a well-trained large model (teacher) to a small model (student), has become an important area of research for practical deployment of recommender systems. Recently, Relaxed Ranking Distillation (RRD) has shown that distilling the ranking information in the recommendation list significantly improves the performance. However, the method still has limitations in that 1) it does not fully utilize the prediction errors of the student model, which makes the training not fully efficient, and 2) it only distills the user-side ranking information, which provides an insufficient view under the sparse implicit feedback. This paper presents Dual Correction strategy for Distillation (DCD), which transfers the ranking information from the teacher model to the student model in a more efficient manner. Most importantly, DCD uses the discrepancy between the teacher model and the student model predictions to decide which knowledge to be distilled. By doing so, DCD essentially provides the learning guidance tailored to "correcting" what the student model has failed to accurately predict. This process is applied for transferring the ranking information from the user-side as well as the item-side to address sparse implicit user feedback. Our experiments show that the proposed method outperforms the state-of-the-art baselines, and ablation studies validate the effectiveness of each component.
- Speeding up the xbox recommender system using a euclidean transformation for inner-product spaces. In RecSys.
- Learning efficient object detection models with knowledge distillation. In NeurIPS.
- Unsupervised Proxy Selection for Session-based Recommender Systems. In SIGIR.
- Born again neural networks. arXiv preprint arXiv:1805.04770 (2018).
- Neural collaborative filtering. In WWW.
- Distilling the knowledge in a neural network. NIPS (2015).
- Jun Hu and Ping Li. 2018. Collaborative multi-objective ranking. In CIKM.
- Collaborative filtering for implicit feedback datasets. In ICDM.
- DE-RRD: A Knowledge Distillation Framework for Recommender System. In CIKM.
- Item-side Ranking Regularized Distillation for Recommender System. Information Sciences (2021). https://doi.org/10.1016/j.ins.2021.08.060
- Topology Distillation for Recommender System. In KDD.
- Semi-supervised learning for cross-domain recommendation to cold-start users. In CIKM.
- Wang-Cheng Kang and Julian McAuley. 2019. Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation. In CIKM.
- Dual neural personalized ranking. In WWW.
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
- Deep Rating Elicitation for New Users in Collaborative Filtering. In WWW.
- Bidirectional Distillation for Top-K Recommender System. In WWW.
- Bootstrapping User and Item Representations for One-Class Collaborative Filtering. SIGIR (2021).
- Collaborative Distillation for Top-N Recommendation. ICDM (2019).
- Discrete Content-Aware Matrix Factorization. In KDD.
- Discrete factorization machines for fast feature-based recommendation. arXiv preprint arXiv:1805.02232 (2018).
- An experimental evaluation of point-of-interest recommendation in location-based social networks. Proceedings of the VLDB Endowment (2017).
- Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 (2019).
- BPR: Bayesian personalized ranking from implicit feedback. In UAI.
- Fitnets: Hints for thin deep nets. In arXiv.
- Jiaxi Tang and Ke Wang. 2018. Ranking distillation: Learning compact ranking models with high performance for recommender system. In KDD.
- Collaborative topic regression with social regularization for tag recommendation. In IJCAI.
- Binarized collaborative filtering with distilling graph convolutional networks. IJCAI (2019).
- Listwise approach to learning to rank: theory and algorithm. In ICML.
- Discrete Collaborative Filtering. In SIGIR.
- Youngjune Lee (5 papers)
- Kee-Eung Kim (24 papers)