Sequential Recommendation via Stochastic Self-Attention (2201.06035v2)

Abstract: Sequential recommendation models the dynamics of a user's previous behaviors in order to forecast the next item, and has drawn a lot of attention. Transformer-based approaches, which embed items as vectors and use dot-product self-attention to measure the relationship between items, demonstrate superior capabilities among existing sequential methods. However, users' real-world sequential behaviors are \textit{\textbf{uncertain}} rather than deterministic, posing a significant challenge to present techniques. We further suggest that dot-product-based approaches cannot fully capture \textit{\textbf{collaborative transitivity}}, which can be derived in item-item transitions inside sequences and is beneficial for cold start items. We further argue that BPR loss has no constraint on positive and sampled negative items, which misleads the optimization. We propose a novel \textbf{STO}chastic \textbf{S}elf-\textbf{A}ttention~(STOSA) to overcome these issues. STOSA, in particular, embeds each item as a stochastic Gaussian distribution, the covariance of which encodes the uncertainty. We devise a novel Wasserstein Self-Attention module to characterize item-item position-wise relationships in sequences, which effectively incorporates uncertainty into model training. Wasserstein attentions also enlighten the collaborative transitivity learning as it satisfies triangle inequality. Moreover, we introduce a novel regularization term to the ranking loss, which assures the dissimilarity between positive and the negative items. Extensive experiments on five real-world benchmark datasets demonstrate the superiority of the proposed model over state-of-the-art baselines, especially on cold start items. The code is available in \url{https://github.com/zfan20/STOSA}.

• The paper introduces STOchastic Self-Attention (STOSA) that models items as stochastic Gaussian distributions to capture uncertainty in user behavior.
• It employs a novel Wasserstein self-attention mechanism to effectively measure item-item relationships and enhance collaborative transitivity.
• Extensive experiments on five benchmarks show significant improvements in ranking metrics, particularly for cold start and dynamic user scenarios.

Overview of Sequential Recommendation via Stochastic Self-Attention

The paper "Sequential Recommendation via Stochastic Self-Attention" addresses significant challenges in sequential recommendation systems by introducing stochastic self-attention as a novel approach. This research aims to improve recommendation systems by better understanding and predicting user behaviors that are inherently uncertain and dynamic. The existing transformer-based techniques leverage fixed vector embeddings and dot-product self-attention mechanisms, which struggle to capture the uncertainty in user behaviors and the collaborative transitivity among items, particularly beneficial for cold start scenarios.

Key Contributions

1. Stochastic Embeddings: The paper introduces STOchastic Self-Attention (STOSA), where each item is represented as a stochastic Gaussian distribution, incorporating both mean and covariance. This representation is crucial for embodying the uncertainty in user-item interactions.
2. Wasserstein Self-Attention: STOSA features a pioneering Wasserstein Self-Attention module, utilizing the Wasserstein distance to effectively capture item-item relationships and incorporate uncertainty. This distance metric adheres to the triangle inequality, thereby facilitating collaborative transitivity.
3. Enhanced BPR Loss: The research further enhances the Bayesian Personalized Ranking (BPR) loss function by incorporating a regularization term. This term ensures the dissimilarity between positive and negative samples, thus optimizing ranking accuracy.

Experimental Findings

Extensive experiments conducted on five benchmark datasets demonstrate the superiority of STOSA compared to conventional and state-of-the-art methods in sequential recommendation tasks. The proposed model shows significant improvements, particularly in handling cold start items and users with varying interest patterns. Numerical results highlight that STOSA outperforms baselines across several metrics, including Recall@1, Recall@5, NDCG@5, and MRR.

Implications and Future Directions

The implications of this research extend both practically and theoretically. By modeling items through stochastic distributions, STOSA provides a more flexible framework for sequential recommendations, effectively handling dynamic and uncertain user behaviors. The adoption of Wasserstein self-attention opens avenues for further exploration into alternative distance measures and their applications in deep learning models.

Future research could explore extending this approach to broader domains beyond recommender systems, examining its applicability in other sequential data modeling tasks. Additionally, optimizing computational efficiency and scalability of stochastic attention mechanisms remains a critical area for further investigation.

In conclusion, this paper contributes significantly to the advancement of recommendation systems by introducing novel methods that address key limitations in existing models, fostering better understanding and adaptation to real-world user behavior dynamics.