Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity (2309.06240v2)

Published 12 Sep 2023 in stat.ML, cs.LG, physics.chem-ph, and physics.data-an

Abstract: Reliable uncertainty quantification (UQ) in ML regression tasks is becoming the focus of many studies in materials and chemical science. It is now well understood that average calibration is insufficient, and most studies implement additional methods testing the conditional calibration with respect to uncertainty, i.e. consistency. Consistency is assessed mostly by so-called reliability diagrams. There exists however another way beyond average calibration, which is conditional calibration with respect to input features, i.e. adaptivity. In practice, adaptivity is the main concern of the final users of a ML-UQ method, seeking for the reliability of predictions and uncertainties for any point in features space. This article aims to show that consistency and adaptivity are complementary validation targets, and that a good consistency does not imply a good adaptivity. Adapted validation methods are proposed and illustrated on a representative example.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and Best Practices for Machine Learning in Chemistry. Trends in Chemistry, 3:146–156, 2021.
  2. Clarifying Trust of Materials Property Predictions using Neural Networks with Distribution-Specific Uncertainty Quantification. Mach. Learn.: Sci. Technol., 4:025019, 2023.
  3. Characterizing Uncertainty in Machine Learning for Chemistry. J. Chem. Inf. Model., 63:4012–4029, 2023.
  4. Materials cartography: A forward-looking perspective on materials representation and devising better maps. APL Mach. Learn., 1:020901, 2023.
  5. High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach. In International Conference on Machine Learning, pages 4075–4084. PMLR, 2018. URL: https://proceedings.mlr.press/v80/pearce18a.html.
  6. Fast and accurate uncertainty estimation in chemical machine learning. J. Chem. Theory Comput., 15:906–915, 2019.
  7. Uncertainty Quantification Using Neural Networks for Molecular Property Prediction. J. Chem. Inf. Model., 60:3770–3780, 2020.
  8. Methods for comparing uncertainty quantifications for material property predictions. Mach. Learn.: Sci. Technol., 1:025006, 2020.
  9. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Inform. Fusion, 76:243–297, 2021.
  10. A Survey of Uncertainty in Deep Neural Networks. arXiv:2107.03342, July 2021.
  11. Pairwise difference regression: A machine learning meta-algorithm for improved prediction and uncertainty quantification in chemical search. J. Chem. Inf. Model., 61:3846–3857, 2021. PMID: 34347460.
  12. CRUDE: Calibrating Regression Uncertainty Distributions Empirically. arXiv:2005.12496, March 2021.
  13. Robust and scalable uncertainty estimation with conformal prediction for machine-learned interatomic potentials. Mach. Learn.: Sci. Technol., 3:045028, November 2022.
  14. Materials Property Prediction with Uncertainty Quantification: A Benchmark Study. arXiv:2211.02235, November 2022.
  15. Deep ensemble inverse model for image-based estimation of solar cell parameters. APL Mach. Learn., 1:036108, 2023.
  16. Graph Neural Network Interatomic Potential Ensembles with Calibrated Aleatoric and Epistemic Uncertainty on Energy and Forces. arXiv:2305.16325, May 2023.
  17. W. He and Z. Jiang. A Survey on Uncertainty Quantification Methods for Deep Neural Networks: An Uncertainty Source Perspective. arXiv:2302.13425, February 2023.
  18. Machine learning guided optimal composition selection of niobium alloys for high temperature applications. APL Mach. Learn., 1:036102, 2023.
  19. Reliable neural networks for regression uncertainty estimation. Reliab. Eng. Syst. Saf., 229:108811, 2023.
  20. The Peril of Popular Deep Learning Uncertainty Estimation Methods. arXiv:2112.05000, 2021.
  21. P. Pernot. Prediction uncertainty validation for computational chemists. J. Chem. Phys., 157:144103, 2022.
  22. Evaluation of measurement data - Guide to the expression of uncertainty in measurement (GUM). Technical Report 100:2008, Joint Committee for Guides in Metrology, JCGM, 2008. URL: http://www.bipm.org/utils/common/documents/jcgm/JCGM_100_2008_F.pdf.
  23. Uncertainty associated with virtual measurements from computational quantum chemistry models. Metrologia, 41:369–375, 2004.
  24. Introduction to Active Thermochemical Tables: Several "key" enthalpies of formation revisited. J. Phys. Chem. A, 108:9979–9997, 2004.
  25. A quantitative uncertainty metric controls error in neural network-driven chemical discovery. Chem. Sci., 10:7913–7922, 2019.
  26. E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach. Learn., 110:457–506, 2021.
  27. A universal similarity based approach for predictive uncertainty quantification in materials science. Sci. Rep., 12:1–10, 2022.
  28. On Calibration of Modern Neural Networks. In International Conference on Machine Learning, pages 1321–1330. 2017. URL: https://proceedings.mlr.press/v70/guo17a.html.
  29. Accurate uncertainties for deep learning using calibrated regression. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2796–2804. PMLR, 10–15 Jul 2018. URL: https://proceedings.mlr.press/v80/kuleshov18a.html.
  30. Evaluating and Calibrating Uncertainty Prediction in Regression Tasks. Sensors, 22:5540, 2022.
  31. Calibrated uncertainty for molecular property prediction using ensembles of message passing neural networks. Mach. Learn.: Sci. Technol., 3:015012, 2022.
  32. V. Vovk. Conditional validity of inductive conformal predictors. In S. C. H. Hoi and W. Buntine, editors, Proceedings of the Asian Conference on Machine Learning, volume 25 of Proceedings of Machine Learning Research, pages 475–490, Singapore Management University, Singapore, 04–06 Nov 2012. PMLR. URL: https://proceedings.mlr.press/v25/vovk12.html.
  33. A. N. Angelopoulos and S. Bates. A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification. arXiv:2107.07511, July 2021.
  34. Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. J. Mach. Learn. Res., 22:3681–3722, 2021.
  35. Improving Conditional Coverage via Orthogonal Quantile Regression. arXiv:2106.00394, 2021.
  36. M. Reiher. Molecule-specific uncertainty quantification in quantum chemical studies. Isr. J. Chem., 62:e202100101, 2022.
  37. Uncertainty Estimates and Multi-hypotheses Networks for Optical Flow. In Computer Vision – ECCV 2018, pages 677–693. Springer, Cham, Switzerland, 2018.
  38. Evaluating scalable uncertainty estimation methods for deep learning-based molecular property prediction. J. Chem. Inf. Model., 60:2697–2717, 2020.
  39. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. arXiv:2011.09588, 2020.
  40. Individual Calibration with Randomized Forecasting. arXiv:2006.10288, June 2020.
  41. Well-calibrated regression uncertainty in medical imaging with deep learning. In T. Arbel, I. Ben Ayed, M. de Bruijne, M. Descoteaux, H. Lombaert, and C. Pal, editors, Proceedings of the Third Conference on Medical Imaging with Deep Learning, volume 121 of Proceedings of Machine Learning Research, pages 393–412. PMLR, 06–08 Jul 2020. URL: https://proceedings.mlr.press/v121/laves20a.html.
  42. Evaluating and Calibrating Uncertainty Prediction in Regression Tasks. arXiv:1905.11659v3, February 2020.
  43. Uncertainty quantification for predictions of atomistic neural networks. arXiv:2207.06916, July 2022.
  44. P. Pernot. Validation of uncertainty quantification metrics: a primer based on the consistency and adaptivity concepts. arXiv:2303.07170, 2023. arXiv:2303.07170.
  45. P. Pernot. The long road to calibrated prediction uncertainty in computational chemistry. J. Chem. Phys., 156:114109, 2022.
  46. A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning. arXiv:2109.02355, September 2021.
  47. P. Pernot and F. Cailliez. A critical review of statistical calibration/prediction models handling data inconsistency and model inadequacy. AIChE J., 63:4642–4665, 2017.
  48. R. T. Birge. The calculation of errors by the method of least squares. Phys. Rev., 40:207–227, 1932.
  49. Local Calibration: Metrics and Recalibration. arXiv:2102.10809, February 2021.
  50. Calibration after bootstrap for accurate uncertainty quantification in regression models. npj Comput. Mater., 8:1–9, 2022.
  51. P. Pernot. Stratification of uncertainties recalibrated by isotonic regression and its impact on calibration error statistics. arXiv:2306.05180, June 2023.
  52. Assessing differences between results determined according to the guide to the expression of uncertainty in measurement. J. Res. Nat. Inst. Stand. Technol., 115(6):453, 2010.
  53. B. Efron. Bootstrap Methods: Another Look at the Jackknife. Ann. Stat., 7:1–26, January 1979.
  54. B. Efron and R. Tibshirani. Statistical data analysis in the computer age. Science, 253:390–395, 1991.
  55. Validation Metrics for Deterministic and Probabilistic Data. J. Verif. Validation Uncertainty Quantif., 3:031002, 2018.
  56. P. Pernot. Properties of the ENCE and other MAD-based calibration metrics. arXiv:2305.11905, May 2023.
  57. P. Pernot. Confidence curves for UQ validation: probabilistic reference vs. oracle. arXiv:2206.15272, June 2022.
  58. Uncertain of uncertainties? A comparison of uncertainty quantification metrics for chemical data sets. ChemRxiv, September 2023.
  59. L. van der Maaten and G. Hinton. Visualizing data using t-sne. J Mach Learn Res, 9(8):2579–2605, 2008. URL: http://jmlr.org/papers/v9/vandermaaten08a.html.
  60. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426, February 2018.
  61. Uncertainty toolbox: an open-source library for assessing, visualizing, and improving uncertainty quantification. arXiv:2109.10254, September 2021.
  62. P. Pernot. Codes and data for the reproduction of the results of the present paper, 2023.
  63. R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2019. URL: http://www.R-project.org/.
  64. P. Pernot. Errviewlib-v1.7.3, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (1)
  1. Pascal Pernot (26 papers)
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now