Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

BARS-CTR: Open Benchmarking for Click-Through Rate Prediction (2009.05794v5)

Published 12 Sep 2020 in cs.IR and cs.AI

Abstract: Click-through rate (CTR) prediction is a critical task for many applications, as its accuracy has a direct impact on user experience and platform revenue. In recent years, CTR prediction has been widely studied in both academia and industry, resulting in a wide variety of CTR prediction models. Unfortunately, there is still a lack of standardized benchmarks and uniform evaluation protocols for CTR prediction research. This leads to non-reproducible or even inconsistent experimental results among existing studies, which largely limits the practical value and potential impact of their research. In this work, we aim to perform open benchmarking for CTR prediction and present a rigorous comparison of different models in a reproducible manner. To this end, we ran over 7,000 experiments for more than 12,000 GPU hours in total to re-evaluate 24 existing models on multiple datasets and settings. Surprisingly, our experiments show that with sufficient hyper-parameter search and model tuning, many deep models have smaller differences than expected. The results also reveal that making real progress on the modeling of CTR prediction is indeed a very challenging research task. We believe that our benchmarking work could not only allow researchers to gauge the effectiveness of new models conveniently but also make them fairly compare with the state of the arts. We have publicly released the benchmarking code, evaluation protocols, and hyper-parameter settings of our work to promote reproducible research in this field.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. 2014. The Criteo Dataset. https://www.kaggle.com/c/criteo-display-ad-challenge
  2. 2015. The Avazu Dataset. https://www.kaggle.com/c/avazu-ctr-prediction
  3. 2020. Neural Network Intelligence — An open source AutoML toolkit. https://github.com/Microsoft/nni
  4. 2021. The DeepCTR Package. https://github.com/shenweichen/DeepCTR
  5. Higher-Order Factorization Machines. In Annual Conference on Neural Information Processing Systems (NeurIPS). 3351–3359.
  6. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 785–794.
  7. FLEN: Leveraging Field for Scalable CTR Prediction. CoRR abs/1911.04690 (2019).
  8. Wide & Deep Learning for Recommender Systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems (DLRS@RecSys). 7–10.
  9. Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions. In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI). 3609–3616.
  10. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys). 191–198.
  11. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys). 101–109.
  12. ImageNet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). 248–255.
  13. Deep Session Interest Network for Click-Through Rate Prediction. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI). 2301–2307.
  14. Image Matters: Visually Modeling User Behaviors Using Advanced Model Server. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM). 2087–2095.
  15. An Embedding Learning Framework for Numerical Features in CTR Prediction. In The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). 2910–2918.
  16. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In International Joint Conference on Artificial Intelligence (IJCAI). 1725–1731.
  17. Xiangnan He and Tat-Seng Chua. 2017. Neural Factorization Machines for Sparse Predictive Analytics. In Proceedings of the 40th ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). 355–364.
  18. Practical Lessons from Predicting Clicks on Ads at Facebook. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, (ADKDD). 5:1–5:9.
  19. Practical Lessons from Predicting Clicks on Ads at Facebook. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising (ADKDD). 5:1–5:9.
  20. Interaction-Aware Factorization Machines for Recommender Systems. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI). 3804–3811.
  21. FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction. In Proceedings of ACM Conference on Recommender Systems (RecSys). 169–177.
  22. Field-aware Factorization Machines for CTR Prediction. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys). 43–50.
  23. Liang Lan and Yu Geng. 2019. Accurate and Interpretable Factorization Machines. In The AAAI Conference on Artificial Intelligence (AAAI). 4139–4146.
  24. Adversarial Multimodal Representation Learning for Click-Through Rate Prediction. In The Web Conference (WWW). 827–836.
  25. Interpretable Click-Through Rate Prediction through Hierarchical Attention. In The International Conference on Web Search and Data Mining (WSDM). 313–321.
  26. Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). 539–548.
  27. xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, (KDD). 1754–1763.
  28. Model Ensemble for Click Prediction in Bing Search Ads. In Proceedings of the 26th International Conference on World Wide Web Companion (WWW). 689–698.
  29. Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction. In The World Wide Web Conference, (WWW). 1119–1129.
  30. A Convolutional Click Prediction Model. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management (CIKM). 1743–1746.
  31. Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD). 1930–1939.
  32. Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. In Proceedings of the 41st International Conference on Research & Development in Information Retrieval (SIGIR). 1137–1140.
  33. Ad click prediction: a view from the trenches. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 1222–1230.
  34. Deep Spatio-Temporal Neural Networks for Click-Through Rate Prediction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD). 2078–2086.
  35. Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018. 1349–1357.
  36. Surabhi Punjabi and Priyanka Bhatt. 2018. Robust Factorization Machines for User Response Prediction. In Proceedings of the 2018 World Wide Web Conference on World Wide Web (WWW). 669–678.
  37. Product-Based Neural Networks for User Response Prediction. In Proceedings of the IEEE 16th International Conference on Data Mining (ICDM). 1149–1154.
  38. Lifelong Sequential Modeling with Personalized Memorization for User Response Prediction. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). 565–574.
  39. Steffen Rendle. 2010. Factorization Machines. In Proceedings of the 10th IEEE International Conference on Data Mining (ICDM). 995–1000.
  40. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th International Conference on World Wide Web (WWW). 521–530.
  41. Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 255–262.
  42. AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). 1161–1170.
  43. Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison. In Proceedings of the Fourteenth ACM Conference on Recommender Systems (RecSys). 23–32.
  44. Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations. In Proceedings of the Fourteenth ACM Conference on Recommender Systems (RecSys). 269–278.
  45. Holographic Factorization Machines for Recommendation. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI). 5143–5150.
  46. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In 7th International Conference on Learning Representations (ICLR).
  47. Deep & Cross Network for Ad Click Predictions. In Proceedings of the 11th International Workshop on Data Mining for Online Advertising (ADKDD). 12:1–12:7.
  48. Telepath: Understanding Users from a Human Vision Perspective in Large-Scale Recommender Systems. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI). 467–474.
  49. Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, (IJCAI). 3119–3125.
  50. Canran Xu and Ming Wu. 2020. Learning Feature Interactions with Lorentzian Factorization Machine. In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI). 6470–6477.
  51. Why Do We Click: Visual Impression-aware News Recommendation. In Proceedings of the 29th ACM International Conference on Multimedia (MM).
  52. Operation-aware Neural Networks for user response prediction. Neural Networks 121 (2020), 161–168.
  53. UNBERT: User-News Matching BERT for News Recommendation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI). 3356–3362.
  54. Deep Learning Based Recommender System: A Survey and New Perspectives. ACM Comput. Surv. 52, 1 (2019), 5:1–5:38.
  55. Deep Interest Evolution Network for Click-Through Rate Prediction. CoRR abs/1809.03672 (2018).
  56. Deep Interest Network for Click-Through Rate Prediction. In Proceedings of the SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD). 1059–1068.
  57. Ensembled CTR Prediction via Knowledge Distillation. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM). 2941–2958.
Citations (109)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now