Neural Collaborative Filtering vs. Matrix Factorization Revisited (2005.09683v2)

Abstract: Embedding based models have been the state of the art in collaborative filtering for over a decade. Traditionally, the dot product or higher order equivalents have been used to combine two or more embeddings, e.g., most notably in matrix factorization. In recent years, it was suggested to replace the dot product with a learned similarity e.g. using a multilayer perceptron (MLP). This approach is often referred to as neural collaborative filtering (NCF). In this work, we revisit the experiments of the NCF paper that popularized learned similarities using MLPs. First, we show that with a proper hyperparameter selection, a simple dot product substantially outperforms the proposed learned similarities. Second, while a MLP can in theory approximate any function, we show that it is non-trivial to learn a dot product with an MLP. Finally, we discuss practical issues that arise when applying MLP based similarities and show that MLPs are too costly to use for item recommendation in production environments while dot products allow to apply very efficient retrieval algorithms. We conclude that MLPs should be used with care as embedding combiner and that dot products might be a better default choice.

• The paper presents an empirical reassessment showing that a well-tuned dot product model significantly outperforms MLP-based neural collaborative filtering in recommendation systems.
• The paper demonstrates that MLPs require substantial model capacity and data to approximate simple dot product functions, resulting in higher RMSE when applied to high-dimensional embeddings.
• The paper highlights the computational efficiency of dot products over MLPs and advocates for their use in real-time recommendation systems to achieve cost savings and improved performance.

Analysis of "Neural Collaborative Filtering vs. Matrix Factorization Revisited"

This paper revisits an important paradigm in collaborative filtering using embedding-based models—specifically evaluating Neural Collaborative Filtering (NCF) against traditional Matrix Factorization (MF) techniques. This exploration is significant as NCF, which utilizes learned embeddings processed through a multilayer perceptron (MLP), is often considered state-of-the-art in recommendation systems, primarily due to its perceived ability to learn complex, non-linear relationships.

Main Contributions and Findings

1. Empirical Reassessment: The authors present an empirical reevaluation of the experiments from the original NCF work. They discover that, contrary to earlier assertions, a well-tuned dot product model significantly outperforms the MLP-based approach in various scenarios when tested for item retrieval from users' implicit feedback datasets (e.g., Movielens and Pinterest datasets).
2. Challenges in Learning with MLPs: The paper exposes the challenges inherent in using MLPs to approximate simple functions such as dot products. Even though MLPs are universal function approximators, the authors demonstrate, both theoretically and empirically, that learning a dot product requires a substantial model capacity and considerable data. They substantiate this claim by deriving RMSE values significantly above the expected threshold when MLPs are tasked with learning dot-product equivalents in high-dimensional embeddings.
3. Efficiency in Deployment: A critical insight offered is the practical implications of the computational cost associated with MLPs. Dot products are not only computationally less expensive (scaling linearly with the embedding dimension $d$), but they also benefit from efficient algorithms for maximum inner product search, which are pivotal in large-scale real-time recommendations.
4. Recommendations for Model Selection: Based on numerical results, the authors recommend against the default use of MLPs for embedding combination in production environments, advocating that dot products suffice unless the dataset is particularly large or the embedding space is minimal.

Implications of Research

The implications of this research are multi-faceted. Theoretically, it challenges the narrative that MLPs, due to their general approximative capabilities, are inherently superior. It advocates for the adoption of simpler, more robust methods unless specific conditions necessitate more complex architectures. Practically, this could mean substantial cost savings and increased efficiency for industry applications, specifically in contexts requiring rapid, online computation of recommendations.

Future Directions

1. Robustness of Dot Products in Various Settings: Further studies could explore the robustness of dot product-based recommendations across diverse types of datasets and user interactions, extending beyond the implicit feedback evaluated in this paper.
2. Exploration of Hybrid Models: While this paper downplays the success of NeuMF, future research might explore alternative hybrid paradigms that leverage the strengths of both MLPs and traditional linear methods in specific contexts where mixed signal processing is beneficial.
3. Extension to Advanced Neural Models: Given the success of dot products in NLP models like Transformers, research could investigate architectures where dot products are integrated into more advanced deep learning paradigms beyond recommender systems.

In summary, this paper advocates a reconsideration of model complexities in favor of tried-and-true techniques like dot products, which, given proper tuning, match or exceed the performance of sophisticated neural architectures like MLPs in the field of embedding-based recommender systems.