Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
91 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
o3 Pro
5 tokens/sec
GPT-4.1 Pro
15 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
Gemini 2.5 Flash Deprecated
12 tokens/sec
2000 character limit reached

Meta-learning to Calibrate Gaussian Processes with Deep Kernels for Regression Uncertainty Estimation (2312.07952v1)

Published 13 Dec 2023 in stat.ML, cs.AI, and cs.LG

Abstract: Although Gaussian processes (GPs) with deep kernels have been successfully used for meta-learning in regression tasks, its uncertainty estimation performance can be poor. We propose a meta-learning method for calibrating deep kernel GPs for improving regression uncertainty estimation performance with a limited number of training data. The proposed method meta-learns how to calibrate uncertainty using data from various tasks by minimizing the test expected calibration error, and uses the knowledge for unseen tasks. We design our model such that the adaptation and calibration for each task can be performed without iterative procedures, which enables effective meta-learning. In particular, a task-specific uncalibrated output distribution is modeled by a GP with a task-shared encoder network, and it is transformed to a calibrated one using a cumulative density function of a task-specific Gaussian mixture model (GMM). By integrating the GP and GMM into our neural network-based model, we can meta-learn model parameters in an end-to-end fashion. Our experiments demonstrate that the proposed method improves uncertainty estimation performance while keeping high regression performance compared with the existing methods using real-world datasets in few-shot settings.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243–297, 2021.
  2. Multi-task feature learning. Advances in Neural Information Processing Systems, 2006.
  3. B. Bakker and T. Heskes. Task clustering and gating for Bayesian multitask learning. Journal of Machine Learning Research, 4:83–99, 2003.
  4. The need for uncertainty quantification in machine-assisted medical decision making. Nature Machine Intelligence, 1(1):20–23, 2019.
  5. Learning a synaptic learning rule. In International Joint Conference on Neural Networks, 1991.
  6. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations, 2018.
  7. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613–1622, 2015.
  8. Meta-calibration: Meta-learning of model calibration using differentiable expected calibration error. arXiv preprint arXiv:2106.09613, 2021.
  9. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems, 31, 2018.
  10. Uncertainty toolbox: an open-source library for assessing, visualizing, and improving uncertainty quantification. arXiv preprint arXiv:2109.10254, 2021.
  11. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. Advances in Neural Information Processing Systems, 34:10971–10984, 2021.
  12. Calibrated reliable regression using maximum mean discrepancy. Advances in Neural Information Processing Systems, 33:17164–17175, 2020.
  13. A. P. Dawid. The well-calibrated Bayesian. Journal of the American Statistical Association, 77(379):605–610, 1982.
  14. Fast calibrated additive quantile regression. Journal of the American Statistical Association, 116(535):1402–1412, 2021.
  15. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135, 2017.
  16. Meta-learning mean functions for Gaussian processes. arXiv preprint arXiv:1901.08098, 2019.
  17. Loss function based second-order jensen inequality and its application to particle variational inference. Advances in Neural Information Processing Systems, 34:6803–6815, 2021.
  18. Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pages 1050–1059, 2016.
  19. Deep Bayesian active learning with image data. In International Conference on Machine Learning, pages 1183–1192, 2017.
  20. J. Garcia and F. Fernández. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research, 45:515–564, 2012.
  21. Conditional neural processes. In International Conference on Machine Learning, pages 1704–1713, 2018.
  22. Neural processes. arXiv preprint arXiv:1807.01622, 2018.
  23. T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359–378, 2007.
  24. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321–1330, 2017.
  25. Meta-learning priors for efficient online Bayesian regression. In Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13, pages 318–337. Springer, 2020.
  26. Efficient and scalable multi-task regression on massive number of tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3763–3770, 2019.
  27. J. M. Hernández-Lobato and R. Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference on Machine Learning, pages 1861–1869, 2015.
  28. T. Iwata and Y. Tanaka. Few-shot learning for spatial regression via neural embedding-based Gaussian processes. Machine Learning, 111:1239–1257, 2022.
  29. Quantile-based bias correction and uncertainty quantification of extreme event attribution statements. Weather and Climate Extremes, 12:24–32, 2016.
  30. Statistical uncertainty quantification to augment clinical decision support: a first implementation in sleep medicine. NPJ Digital Medicine, 4(1):1–9, 2021.
  31. Attentive neural processes. In International Conference on Learning Representations, 2018.
  32. S. Kim and S.-Y. Yun. Calibration of few-shot classification tasks: Mitigating misconfidence from distribution mismatch. IEEE Access, 10:53894–53908, 2022.
  33. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
  34. D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
  35. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, pages 2796–2804, 2018.
  36. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning, pages 2805–2814, 2018.
  37. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
  38. A simple baseline for Bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
  39. Calibrated model-based deep reinforcement learning. In International Conference on Machine Learning, pages 4314–4323, 2019.
  40. Modular conformal calibration. In International Conference on Machine Learning, pages 15180–15195, 2022.
  41. Uncertainty quantification with statistical guarantees in end-to-end autonomous driving control. In IEEE International Conference on Robotics and Automation, pages 7344–7350, 2020.
  42. Obtaining well calibrated probabilities using Bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
  43. T. Nguyen and A. Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. arXiv preprint arXiv:2207.04179, 2022.
  44. A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with supervised learning. In International Conference on Machine Learning, pages 625–632, 2005.
  45. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
  46. Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems, 33:16108–16118, 2020.
  47. High-quality prediction intervals for deep learning: A distribution-free, ensembled approach. In International Conference on Machine Learning, pages 4075–4084, 2018.
  48. J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61–74, 1999.
  49. Gaussian Processes for Machine Learning. The MIT Press, 2005.
  50. S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017.
  51. Meta-learning reliable priors in the function space. Advances in Neural Information Processing Systems, 34:280–293, 2021.
  52. Reliable decisions with threshold calibration. Advances in Neural Information Processing Systems, 34:1831–1844, 2021.
  53. Prediction intervals: Split normal mixture from quality-driven deep ensembles. In Conference on Uncertainty in Artificial Intelligence, pages 1179–1187, 2020.
  54. J. Schmidhuber. Evolutionary principles in self-referential learning. on learning now to learn: The meta-meta-meta…-hook. Master’s thesis, Technische Universitat Munchen, Germany, 1987.
  55. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175, 2015.
  56. Reliable training and estimation of variance networks. Advances in Neural Information Processing Systems, 32, 2019.
  57. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 30, 2017.
  58. E. Snelson and Z. Ghahramani. Sparse Gaussian processes using pseudo-inputs. Advances in Neural Information Processing Systems, 18, 2005.
  59. Distribution calibration for regression. In International Conference on Machine Learning, pages 5897–5906, 2019.
  60. N. Tagasovska and D. Lopez-Paz. Single-model uncertainties for deep learning. Advances in Neural Information Processing Systems, 32, 2019.
  61. Time series clustering: A superior alternative for market basket analysis. In Proceedings of International Conference on Advanced Data and Information Engineering, pages 241–248. Springer, 2014.
  62. Adaptive deep kernel learning. arXiv preprint arXiv:1905.12131, 2019.
  63. Calibrating deep convolutional Gaussian processes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1554–1563, 2019.
  64. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pages 3630–3638, 2016.
  65. Conformal calibrators. In Conformal and Probabilistic Prediction and Applications, pages 84–99, 2020.
  66. Deep kernel learning. In International Conference on Artificial Intelligence and Statistics, pages 370–378, 2016.
  67. Calibrating cnns for few-shot meta learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2090–2099, 2022.
  68. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129–14142, 2020.
  69. Individual calibration with randomized forecasting. In International Conference on Machine Learning, pages 11387–11397, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now