The MovieLens Beliefs Dataset: Collecting Pre-Choice Data for Online Recommender Systems (2405.11053v3)
Abstract: An increasingly important aspect of designing recommender systems involves considering how recommendations will influence consumer choices. This paper addresses this issue by introducing a method for collecting user beliefs about un-experienced items - a critical predictor of choice behavior. We implemented this method on the MovieLens platform, resulting in a rich dataset that combines user ratings, beliefs, and observed recommendations. We document challenges to such data collection, including selection bias in response and limited coverage of the product space. This unique resource empowers researchers to delve deeper into user behavior and analyze user choices absent recommendations, measure the effectiveness of recommendations, and prototype algorithms that leverage user belief data, ultimately leading to more impactful recommender systems. The dataset can be found at https://grouplens.org/datasets/movielens/ml_belief_2024/.
- The connection between popularity bias, calibration, and fairness in recommendation. In Proceedings of the 14th ACM conference on recommender systems. 726–731.
- Gediminas Adomavicius and Alexander Tuzhilin. 2005. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE transactions on knowledge and data engineering 17, 6 (2005), 734–749.
- The Economics of Recommender Systems: Evidence from a Field Experiment on MovieLens. In Proceedings of the 24th ACM Conference on Economics and Computation (London, United Kingdom) (EC ’23). Association for Computing Machinery, New York, NY, USA, 117. https://doi.org/10.1145/3580507.3597677
- Deconstructing the Filter Bubble: User Decision-Making and Recommender Systems. Fourteenth ACM Conference on Recommender Systems (2020), 82–91.
- Novelty and diversity in recommender systems. In Recommender systems handbook. Springer, 603–646.
- Social comparisons and contributions to online communities: A field experiment on movielens. American Economic Review 100, 4 (2010), 1358–98.
- Beyond accuracy: evaluating recommender systems by coverage and serendipity. Proceedings of the fourth ACM conference on Recommender systems (2010), 257–260.
- F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TIIS) 5, 4 (2015), 1–19.
- Marius Kaminskas and Derek Bridge. 2016. Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Transactions on Interactive Intelligent Systems (TiiS) 7, 1 (2016), 1–42.
- Investigating serendipity in recommender systems based on real user feedback. In Proceedings of the 33rd annual acm symposium on applied computing. 1341–1350.
- Challenges of serendipity in recommender systems. WEBIST 2016: Proceedings of the 12th International conference on web information systems and technologies. Volume 2, ISBN 978-989-758-186-1 (2016).
- Being accurate is not enough: how accuracy metrics have hurt recommender systems. CHI’06 extended abstracts on Human factors in computing systems (2006), 1097–1101.
- Exploring the filter bubble: the effect of using recommender systems on content diversity. Proceedings of the 23rd International Conference on World Wide Web (2014), 677–686.
- Recommendations as treatments: Debiasing learning and evaluation. Proceedings of the International Conference on Machine Learning (2016), 1670–1679.
- Harald Steck. 2018. Calibrated recommendations. Proceedings of the 12th ACM conference on recommender systems (2018), 154–162.
- Saúl Vargas and Pablo Castells. 2011. Rank and relevance in novelty and diversity metrics for recommender systems. Proceedings of the fifth ACM conference on Recommender systems (2011), 109–116.
- Leveraging missing ratings to improve online recommendation systems. Journal of marketing research 43, 3 (2006), 355–365.