Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Confidence-Aware Multi-Field Model Calibration (2402.17655v2)

Published 27 Feb 2024 in cs.LG

Abstract: Accurately predicting the probabilities of user feedback, such as clicks and conversions, is critical for advertisement ranking and bidding. However, there often exist unwanted mismatches between predicted probabilities and true likelihoods due to the rapid shift of data distributions and intrinsic model biases. Calibration aims to address this issue by post-processing model predictions, and field-aware calibration can adjust model output on different feature field values to satisfy fine-grained advertising demands. Unfortunately, the observed samples corresponding to certain field values can be seriously limited to make confident calibrations, which may yield bias amplification and online disturbance. In this paper, we propose a confidence-aware multi-field calibration method, which adaptively adjusts the calibration intensity based on confidence levels derived from sample statistics. It also utilizes multiple fields for joint model calibration according to their importance to mitigate the impact of data sparsity on a single field. Extensive offline and online experiments show the superiority of our method in boosting advertising performance and reducing prediction deviations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Don’t just blame over-parametrization for over-confidence: Theoretical analysis of calibration in binary classification. In ICML. PMLR, 566–576.
  2. Calibration of machine learning models. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques. IGI Global, 128–146.
  3. Simple and scalable response prediction for display advertising. ACM Transactions on Intelligent Systems and Technology (TIST) 5, 4 (2014), 1–34.
  4. Deep ctr prediction in display advertising. In ACM MM. 811–820.
  5. Morris H DeGroot and Stephen E Fienberg. 1983. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician) 32, 1-2 (1983), 12–22.
  6. Calibrating user response predictions in online advertising. In ECML-PKDD. Springer, 208–223.
  7. A survey of uncertainty in deep neural networks. Artificial Intelligence Review 56, Suppl 1 (2023), 1513–1589.
  8. Tilmann Gneiting and Adrian E Raftery. 2005. Weather forecasting with ensemble methods. Science 310, 5746 (2005), 248–249.
  9. On calibration of modern neural networks. In ICML. PMLR, 1321–1330.
  10. Calibrated Conversion Rate Prediction via Knowledge Distillation under Delayed Feedback in Online Advertising. In CIKM. 3983–3987.
  11. Calibration of Neural Networks using Splines. In ICLR. https://openreview.net/forum?id=eQe8DEWNN2W
  12. Practical lessons from predicting clicks on ads at facebook. In AdKDD. 1–9.
  13. On the importance of gradients for detecting distributional shifts in the wild. NeurIPS 34 (2021), 677–689.
  14. MBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration. In WWW. 2236–2246.
  15. Calibrating predictive model estimates to support personalized medicine. JAMIA 19, 2 (2012), 263–274.
  16. Soft calibration objectives for neural networks. NeurIPS 34 (2021), 29768–29779.
  17. Ranganath Krishnan and Omesh Tickoo. 2020. Improving model calibration with accuracy versus uncertainty optimization. NeurIPS 33 (2020), 18237–18248.
  18. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. NeurIPS 32 (2019).
  19. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In AISTATS. PMLR, 623–631.
  20. Verified uncertainty calibration. NeurIPS 32 (2019).
  21. Trainable calibration measures for neural networks from kernel mean embeddings. In ICML. PMLR, 2805–2814.
  22. Obtaining Calibrated Probabilities with Personalized Ranking Models. In AAAI, Vol. 36. 4083–4091.
  23. Estimating conversion rate in display advertising from past erformance data. In KDD. 768–776.
  24. Improving Multi-Scenario Learning to Rank in E-commerce by Exploiting Task Relationships in the Label Space. In CIKM. ACM, New York,NY,USA.
  25. Focal loss for dense object detection. In ICCV. 2980–2988.
  26. The devil is in the margin: Margin-based label smoothing for network calibration. In CVPR. 80–88.
  27. Ad click prediction: a view from the trenches. In KDD. 1222–1230.
  28. Predicting accurate probabilities with a ranking loss. In ICML, Vol. 2012. NIH Public Access, 703.
  29. Attended temperature scaling: a practical approach for calibrating deep neural networks. arXiv preprint arXiv:1810.11586 (2018).
  30. Calibrating deep neural networks using focal loss. NeurIPS 33 (2020), 15288–15299.
  31. When does label smoothing help? NeurIPS 32 (2019).
  32. Obtaining well calibrated probabilities using bayesian binning. In AAAI, Vol. 29.
  33. Brian Neelon and David B Dunson. 2004. Bayesian isotonic regression and trend analysis. Biometrics 60, 2 (2004), 398–406.
  34. Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In ICML. 625–632.
  35. Measuring Calibration in Deep Learning.. In CVPR workshops, Vol. 2.
  36. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. NeurIPS 32 (2019).
  37. Field-aware calibration: a simple and empirically strong method for reliable probabilistic predictions. In WWW. 729–739.
  38. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 (2017).
  39. John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers 10, 3 (1999), 61–74.
  40. On fairness and calibration. NeurIPS 30 (2017).
  41. Intra order-preserving functions for calibration of multi-class neural networks. NeurIPS 33 (2020), 13456–13467.
  42. Predicting clicks: estimating the click-through rate for new ads. In WWW. 521–530.
  43. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624 (2021).
  44. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In RecSys. 269–278.
  45. On calibration and out-of-domain generalization. NeurIPS 34 (2021), 2215–2227.
  46. Generalizing to unseen domains: A survey on domain generalization. TKDE (2022).
  47. Posterior Probability Matters: Doubly-Adaptive Calibration for Neural Predictions in Online Advertising. In SIGIR. 2645–2649.
  48. Non-parametric calibration for classification. In AISTATS. PMLR, 178–190.
  49. Edwin B Wilson. 1927. Probable inference, the law of succession, and statistical inference. JASA 22, 158 (1927), 209–212.
  50. Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In ICML, Vol. 1. 609–616.
  51. Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In KDD. 694–699.
  52. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In ICML. PMLR, 11117–11128.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Yuang Zhao (3 papers)
  2. Chuhan Wu (87 papers)
  3. Qinglin Jia (8 papers)
  4. Hong Zhu (52 papers)
  5. Jia Yan (14 papers)
  6. Libin Zong (1 paper)
  7. Linxuan Zhang (3 papers)
  8. Zhenhua Dong (76 papers)
  9. Muyu Zhang (2 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now