Deep Uncertainty-Based Explore for Index Construction and Retrieval in Recommendation System

Abstract: In recommendation systems, the relevance and novelty of the final results are selected through a cascade system of Matching -> Ranking -> Strategy. The matching model serves as the starting point of the pipeline and determines the upper bound of the subsequent stages. Balancing the relevance and novelty of matching results is a crucial step in the design and optimization of recommendation systems, contributing significantly to improving recommendation quality. However, the typical matching algorithms have not simultaneously addressed the relevance and novelty perfectly. One main reason is that deep matching algorithms exhibit significant uncertainty when estimating items in the long tail (e.g., due to insufficient training samples) items.The uncertainty not only affects the training of the models but also influences the confidence in the index construction and beam search retrieval process of these models. This paper proposes the UICR (Uncertainty-based explore for Index Construction and Retrieval) algorithm, which introduces the concept of uncertainty modeling in the matching stage and achieves multi-task modeling of model uncertainty and index uncertainty. The final matching results are obtained by combining the relevance score and uncertainty score infered by the model. Experimental results demonstrate that the UICR improves novelty without sacrificing relevance on realworld industrial productive environments and multiple open-source datasets. Remarkably, online A/B test results of display advertising in Shopee demonstrates the effectiveness of the proposed algorithm.

• The paper introduces the UICR algorithm that integrates uncertainty modeling into the matching stage to balance relevance and novelty.
• It employs a three-component pipeline—UN-Index, UN-Retrieval, and UN-Model—to refine traditional approaches and handle long-tail items effectively.
• Experimental evaluations on datasets like Shopee demonstrate improvements such as a 4.80% revenue increase and a 2.59% rise in CTR over existing methods.

Deep Uncertainty-Based Index Construction and Retrieval in Recommendation Systems

Abstract: The paper introduces an innovative approach to improving recommendation systems by balancing relevance and novelty through the incorporation of uncertainty modeling. Conventional recommendation systems follow a three-stage process of Matching -> Ranking -> Strategy. The novelty and relevance of the items presented depend fundamentally on the results obtained from the initial matching stage. However, typical matching algorithms often fail to optimize both relevance and novelty of recommendations, especially for long-tail items. To address this challenge, this paper presents the UICR (Uncertainty-based Index Construction and Retrieval) algorithm, which integrates uncertainty modeling into the matching stage.

Introduction: This research explores a critical issue within industrial-scale recommendation systems: balancing relevance and novelty. By leveraging a deep model to retrieve items that align with user interests, traditional methods often struggle to account for the uncertainty associated with long-tail items. Deep models are typically trained on point estimations which may exhibit high uncertainty for infrequently interacted items. This uncertainty complicates both the model training and the subsequent index construction and retrieval processes.

Methodology:

Problem Definition

The goal is to reframe the matching phase from merely retrieving top candidates to effectively balancing relevance and novelty through uncertainty. The UICR method is defined to handle a large-scale candidate set and ensure the retrieval of items that accurately reflect user interests while maintaining high novelty.

Uncertainty-based Matching Pipeline

The proposed UICR framework contains three main components:

• UN-Index: This component constructs an index that takes uncertainty into account. Traditional nearest-neighbor algorithms, like HNSW, are refined to ensure paths with higher confidence are retained.
• UN-Retrieval: This leverages the UN-Index to integrate relevance and uncertainty scores during retrieval, thereby improving the balance between relevance and novelty.
• UN-Model: The model is trained for both relevance and uncertainty estimate scores, supporting the processes of UN-Index and UN-Retrieval.

Uncertainty-based Index Construction

Using item-to-item uncertainty, the proposed method adjusts the distance metric to retain items with low variance yet high relevance. This adaptation significantly enhances retrieval precision.

Uncertainty-based Retrieval Algorithm

The retrieval process combines traditional scoring methods with uncertainty-based measurements. The score fusion mechanism is formulated to enhance novelty without compromising relevance.

Uncertainty-based Model Training

The model effectively estimates the relevance and variance of both item-to-item and user-to-item interactions. The DUAL (Deep Uncertainty-Aware Learning) method integrates these uncertain estimates seamlessly into the modeling process, ensuring efficient and accurate relevance predictions.

Experiments:

Datasets and Metrics

Evaluations are conducted on three datasets: Amazon, Taobao, and an industrial Shopee dataset, using metrics such as Recall, CateEntropy, and NewCateRatio to assess the balance between relevance and novelty.

Performance Evaluation

UICR demonstrates superior performance compared to traditional methods like DSSM, YouTubeDNN, and NANN, showing significant improvements in both Recall and novelty metrics. Specifically, the Shopee dataset indicates a $4.80\%$ improvement in Revenue and $2.59\%$ in CTR following the deployment of UICR.

Ablation Study

Further analysis through ablation studies highlights the critical contributions of each component within the UICR framework. Specifically, the impact of the uncertainty estimation model for item-to-item (UN-Index) and user-to-item (UN-Retrieval) are individually validated, collectively contributing to overall system performance.

Discussion:

Practical Implications

The UICR's method for incorporating uncertainty into the recommender system's matching stage has considerable implications for real-world applications. It enables recommendation systems to deliver more balanced and diverse recommendation sets, which is crucial for long-term user engagement and system sustainability.

Future Directions

While the presented UICR method offers substantial improvements, future research can investigate further refinements in the uncertainty estimation process and explore more computationally efficient ways to integrate these methods into large-scale production environments.

Conclusion:

This paper presents a significant advancement in recommendation system methodologies by introducing the UICR algorithm, emphasizing the balance of relevance and novelty through uncertainty modeling. This method not only enhances the quality of the index but also augments novelty in recommendations, ultimately providing a more diversified and engaging user experience. The UICR framework, validated through extensive experimentation, demonstrates measurable improvements over existing methods, making it a valuable contribution to the field.