Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Ensemble Shape Calibration: Multi-Field Post-hoc Calibration in Online Advertising (2401.09507v2)

Published 17 Jan 2024 in cs.LG

Abstract: In the e-commerce advertising scenario, estimating the true probabilities (known as a calibrated estimate) on Click-Through Rate (CTR) and Conversion Rate (CVR) is critical. Previous research has introduced numerous solutions for addressing the calibration problem. These methods typically involve the training of calibrators using a validation set and subsequently applying these calibrators to correct the original estimated values during online inference. However, what sets e-commerce advertising scenarios apart is the challenge of multi-field calibration. Multi-field calibration requires achieving calibration in each field. In order to achieve multi-field calibration, it is necessary to have a strong data utilization ability. Because the quantity of pCTR specified range for a single field-value (such as user ID and item ID) sample is relatively small, this makes the calibrator more difficult to train. However, existing methods have difficulty effectively addressing these issues. To solve these problems, we propose a new method named Deep Ensemble Shape Calibration (DESC). In terms of business understanding and interpretability, we decompose multi-field calibration into value calibration and shape calibration. We introduce innovative basis calibration functions, which enhance both function expression capabilities and data utilization by combining these basis calibration functions. A significant advancement lies in the development of an allocator capable of allocating the most suitable calibrators to different estimation error distributions within diverse fields and values. We achieve significant improvements in both public and industrial datasets. In online experiments, we observe a +2.5% increase in CVR and +4.0% in GMV (Gross Merchandise Volume). Our code is now available at: https://github.com/HaoYang0123/DESC.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. Calibration of machine learning models. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques. IGI Global, 128–146.
  2. CAN: feature co-action network for click-through rate prediction. In Proceedings of the fifteenth ACM international conference on web search and data mining. 57–65.
  3. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7–10.
  4. The YouTube video recommendation system. In Proceedings of the fourth ACM conference on Recommender systems. 293–296.
  5. Calibrating user response predictions in online advertising. In Machine Learning and Knowledge Discovery in Databases: Applied Data Science Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part IV. Springer, 208–223.
  6. Exploration in online advertising systems with deep uncertainty-aware learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2792–2801.
  7. Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoft’s bing search engine. Omnipress.
  8. On calibration of modern neural networks. In International conference on machine learning. PMLR, 1321–1330.
  9. DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:1703.04247 (2017).
  10. Distribution-free binary classification: prediction sets, confidence intervals and calibration. Advances in Neural Information Processing Systems 33 (2020), 3711–3723.
  11. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the eighth international workshop on data mining for online advertising. 1–9.
  12. MBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration. In Proceedings of the ACM Web Conference 2022. 2236–2246.
  13. FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction. In Proceedings of the 13th ACM Conference on Recommender Systems. 169–177.
  14. Smooth isotonic regression: a new method to calibrate predictive models. AMIA Summits on Translational Science Proceedings 2011 (2011), 16.
  15. Kevin B Korb. 1999. Calibration and the evaluation of predictive learners. In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence. Citeseer, 73–77.
  16. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems 32 (2019).
  17. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics. PMLR, 623–631.
  18. Verified uncertainty calibration. Advances in Neural Information Processing Systems 32 (2019).
  19. Obtaining Calibrated Probabilities with Personalized Ranking Models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 4083–4091.
  20. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 1137–1140.
  21. LightSAGE: Graph Neural Networks for Large Scale Item Retrieval in Shopee’s Advertisement Recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems. 334–337.
  22. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems 32 (2019).
  23. Field-aware calibration: a simple and empirically strong method for reliable probabilistic predictions. In Proceedings of The Web Conference 2020. 729–739.
  24. John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers 10, 3 (1999), 61–74.
  25. Deep crossing: Web-scale modeling without manually crafted combinatorial features. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 255–262.
  26. Billion-scale commodity embedding for e-commerce recommendation in alibaba. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 839–848.
  27. MaskNet: Introducing feature-wise multiplication to CTR ranking models by instance-guided mask. arXiv preprint arXiv:2102.07619 (2021).
  28. Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2645–2649.
  29. Practice on Effectively Extracting NLP Features for Click-Through Rate Prediction. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 4887–4893.
  30. Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, Vol. 1. 609–616.
  31. Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. 694–699.
  32. Deepintent: Learning attentions for online advertising with recurrent neural networks. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1295–1304.
  33. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference on machine learning. PMLR, 11117–11128.
  34. Optimized cost per click in taobao display advertising. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2191–2200.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Shuai Yang (140 papers)
  2. Hao Yang (328 papers)
  3. Zhuang Zou (1 paper)
  4. Linhe Xu (2 papers)
  5. Shuo Yuan (34 papers)
  6. Yifan Zeng (23 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets