2000 character limit reached
Long-Term Value of Exploration: Measurements, Findings and Algorithms
Published 12 May 2023 in cs.IR | (2305.07764v2)
Abstract: Effective exploration is believed to positively influence the long-term user experience on recommendation platforms. Determining its exact benefits, however, has been challenging. Regular A/B tests on exploration often measure neutral or even negative engagement metrics while failing to capture its long-term benefits. We here introduce new experiment designs to formally quantify the long-term value of exploration by examining its effects on content corpus, and connecting content corpus growth to the long-term user experience from real-world experiments. Once established the values of exploration, we investigate the Neural Linear Bandit algorithm as a general framework to introduce exploration into any deep learning based ranking systems. We conduct live experiments on one of the largest short-form video recommendation platforms that serves billions of users to validate the new experiment designs, quantify the long-term values of exploration, and to verify the effectiveness of the adopted neural linear bandit algorithm for exploration.
- Improved algorithms for linear stochastic bandits. Advances in neural information processing systems 24 (2011).
- Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning. PMLR, 1638–1646.
- Shipra Agrawal and Navin Goyal. 2013. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning. PMLR, 127–135.
- Excuseme: Asking users to help in item cold-start recommendations. In Proceedings of the 9th ACM Conference on Recommender Systems. 83–90.
- Finite-time analysis of the multiarmed bandit problem. Machine learning 47, 2 (2002), 235–256.
- Multiple randomization designs. arXiv preprint arXiv:2112.13495 (2021).
- Carousel personalization in music streaming apps with contextual bandits. In Proceedings of the 14th ACM Conference on Recommender Systems. 420–425.
- Olivier Chapelle and Lihong Li. 2011. An empirical evaluation of thompson sampling. Advances in neural information processing systems 24 (2011).
- Minmin Chen. 2021. Exploration in recommender systems. In Fifteenth ACM Conference on Recommender Systems. 551–553.
- Top-k off-policy correction for a REINFORCE recommender system. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 456–464.
- Values of User Exploration in Recommender Systems. In Fifteenth ACM Conference on Recommender Systems. 85–95.
- A Thompson sampling algorithm for cascading bandits. In The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 438–447.
- Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 208–214.
- Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198.
- An efficient algorithm for generalized linear bandit: Online stochastic gradient descent and thompson sampling. In International Conference on Artificial Intelligence and Statistics. PMLR, 1585–1593.
- Contextual bandits for adapting treatment in a mouse model of de novo carcinogenesis. In Machine learning for healthcare conference. PMLR, 67–82.
- Parametric bandits: The generalized linear case. Advances in Neural Information Processing Systems 23 (2010).
- Vime: Variational information maximizing exploration. Advances in neural information processing systems 29 (2016).
- Guido W Imbens and Donald B Rubin. 2015. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press.
- Using Exploration to Alleviate Closed Loop Effects in Recommender Systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2025–2028.
- Olivier Jeunen and Bart Goethals. 2021. Top-k contextual bandits with equity of exposure. In Proceedings of the 15th ACM Conference on Recommender Systems. 310–320.
- Degenerate feedback loops in recommender systems. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 383–390.
- Deep learning with logged bandit feedback. In International Conference on Learning Representations.
- Trustworthy online controlled experiments: A practical guide to a/b testing. Cambridge University Press.
- Matrix factorization techniques for recommender systems. Computer 42, 8 (2009), 30–37.
- Aravindh Krishnamoorthy and Deepak Menon. 2013. Matrix inversion using Cholesky decomposition. In 2013 signal processing: Algorithms, architectures, arrangements, and applications (SPA). IEEE, 70–72.
- On the Value of Prior in Online Learning to Rank. In International Conference on Artificial Intelligence and Statistics. PMLR, 6880–6892.
- Randomized exploration in generalized linear bandits. In International Conference on Artificial Intelligence and Statistics. PMLR, 2066–2076.
- Asymptotically efficient adaptive allocation rules. Advances in applied mathematics 6, 1 (1985), 4–22.
- John Langford and Tong Zhang. 2007. The epoch-greedy algorithm for multi-armed bandits with side information. Advances in neural information processing systems 20 (2007).
- A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web. 661–670.
- Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning. PMLR, 2071–2080.
- Customized nonlinear bandits for online response selection in neural conversation models. In Thirty-Second AAAI Conference on Artificial Intelligence.
- Explore, exploit, and explain: personalizing explainable recommendations with bandits. In Proceedings of the 12th ACM conference on recommender systems. 31–39.
- Nonstationary bandits with habituation and recovery dynamics. Operations Research 68, 5 (2020), 1493–1516.
- Dynamic online pricing with incomplete information using multiarmed bandit experiments. Marketing Science 38, 2 (2019), 226–252.
- Roger Penrose. 1955. A generalized inverse for matrices. In Mathematical proceedings of the Cambridge philosophical society, Vol. 51. Cambridge University Press, 406–413.
- Numerical recipes 3rd edition: The art of scientific computing. Cambridge university press.
- Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arXiv:1802.09127 (2018).
- Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web. 285–295.
- Short-term satisfaction and long-term coverage: Understanding how users tolerate algorithmic exploration. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 513–521.
- Mastering the game of Go with deep neural networks and tree search. nature 529, 7587 (2016), 484–489.
- Show Me the Whole World: Towards Entire Item Space Exploration for Interactive Personalized Recommendations. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 947–956.
- William R Thompson. 1933. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25, 3-4 (1933), 285–294.
- Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR) 52, 1 (2019), 1–38.
- Neural thompson sampling. arXiv preprint arXiv:2010.00827 (2020).
- Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning. PMLR, 11492–11502.
- Cascading bandits for large-scale recommendation problems. arXiv preprint arXiv:1603.05359 (2016).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
