Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Robust Calibration For Improved Weather Prediction Under Distributional Shift (2401.04144v1)

Published 8 Jan 2024 in cs.LG and cs.AI

Abstract: In this paper, we present results on improving out-of-domain weather prediction and uncertainty estimation as part of the \texttt{Shifts Challenge on Robustness and Uncertainty under Real-World Distributional Shift} challenge. We find that by leveraging a mixture of experts in conjunction with an advanced data augmentation technique borrowed from the computer vision domain, in conjunction with robust \textit{post-hoc} calibration of predictive uncertainties, we can potentially achieve more accurate and better-calibrated results with deep neural networks than with boosted tree models for tabular data. We quantify our predictions using several metrics and propose several future lines of inquiry and experimentation to boost performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Combination of machine learning algorithms for recommendation of courses in e-learning system based on historical data. Knowledge-Based Systems, 51:1–14, 2013.
  2. Invariant Risk Minimization. arXiv e-prints, art. arXiv:1907.02893, jul 2019.
  3. Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb.com/. Software available from wandb.com.
  4. Christopher M Bishop. Mixture density networks, 1994.
  5. Evaluating epidemic forecasts in an interval format. PLOS Computational Biology, 17(2):e1008618, 2021.
  6. Xgboost. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug 2016. doi: 10.1145/2939672.2939785. URL http://dx.doi.org/10.1145/2939672.2939785.
  7. Uncertainty-aware learning from demonstration using mixture density networks with sampling-free variance modeling, 2017.
  8. Uncertainty toolbox: an open-source library for assessing, visualizing, and improving uncertainty quantification. arXiv preprint arXiv:2109.10254, 2021.
  9. Gamma-ray Bursts as distance indicators through a machine learning approach. arXiv e-prints, art. arXiv:1907.05074, jul 2019. doi: 10.48550/arXiv.1907.05074.
  10. Agroconsultant: intelligent crop recommendation system using machine learning algorithms. In 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA), pages 1–6. IEEE, 2018.
  11. Ngboost: Natural gradient boosting for probabilistic prediction, 2020.
  12. James Fiedler. Simple modifications to improve tabular neural networks, 2021.
  13. Unsupervised Domain Adaptation by Backpropagation. arXiv e-prints, art. arXiv:1409.7495, sep 2014.
  14. Sankalp Gilda. Adaptive Kalman Filter-based Wavelet Shrinkage Denoising of Stellar Spectra. In American Astronomical Society Meeting Abstracts #233, volume 233 of American Astronomical Society Meeting Abstracts, page 420.08, jan 2019a.
  15. Sankalp Gilda. Feature Selection for Better Spectral Characterization or: How I Learned to Start Worrying and Love Ensembles. In Peter J. Teuben, Marc W. Pound, Brian A. Thomas, and Elizabeth M. Warner, editors, Astronomical Data Analysis Software and Systems XXVII, volume 523 of Astronomical Society of the Pacific Conference Series, page 67, oct 2019b. doi: 10.48550/arXiv.1902.07215.
  16. Sankalp Gilda. Beyond mirkwood: Enhancing SED Modeling with Conformal Predictions. arXiv e-prints, art. arXiv:2312.14212, dec 2023a. doi: 10.48550/arXiv.2312.14212.
  17. Sankalp Gilda. deep-REMAP: Parameterization of Stellar Spectra Using Regularized Multi-Task Learning. arXiv e-prints, art. arXiv:2311.03738, nov 2023b. doi: 10.48550/arXiv.2311.03738.
  18. Automatic Kalman-filter-based wavelet shrinkage denoising of 1D stellar spectra. Monthly Notices of the Royal Astronomical Society, 490(4):5249–5269, 09 2019. ISSN 0035-8711. doi: 10.1093/mnras/stz2577. URL https://doi.org/10.1093/mnras/stz2577.
  19. Parameterization of MARVELS Spectra Using Deep Learning. In American Astronomical Society Meeting Abstracts #231, volume 231 of American Astronomical Society Meeting Abstracts, page 349.02, jan 2018.
  20. Astronomical Image Quality Prediction based on Environmental and Telescope Operating Conditions. arXiv e-prints, art. arXiv:2011.03132, nov 2020. doi: 10.48550/arXiv.2011.03132.
  21. Unsupervised Domain Adaptation for Constraining Star Formation Histories. arXiv e-prints, art. arXiv:2112.14072, dec 2021a. doi: 10.48550/arXiv.2112.14072.
  22. Uncertainty-aware learning for improvements in image quality of the Canada–France–Hawaii Telescope. Monthly Notices of the Royal Astronomical Society, 510(1):870–902, 11 2021b. ISSN 0035-8711. doi: 10.1093/mnras/stab3243. URL https://doi.org/10.1093/mnras/stab3243.
  23. Mirkwood: fast and accurate sed modeling using machine learning. The Astrophysical Journal, 916(1):43, 2021c.
  24. SED Analysis using Machine Learning Algorithms. In American Astronomical Society Meeting Abstracts, volume 53 of American Astronomical Society Meeting Abstracts, page 119.03, jun 2021d.
  25. mirkwood: SED modeling using machine learning. Astrophysics Source Code Library, record ascl:2102.017, feb 2021e.
  26. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359–378, 2007.
  27. Revisiting deep learning models for tabular data, 2021.
  28. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
  29. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
  30. Antonio Lavecchia. Machine-learning approaches in drug discovery: methods and applications. Drug discovery today, 20(3):318–331, 2015.
  31. Positional normalization. In Advances in Neural Information Processing Systems, pages 1620–1632, 2019.
  32. On feature normalization and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12383–12392, 2021.
  33. Multivariate deep evidential regression. arXiv preprint arXiv:2104.06135, 2021.
  34. SED Fitting in the Modern Era: Fast and Accurate Machine-Learning Assisted Software. HST Proposal. Cycle 29, ID. #16626, jun 2021.
  35. Task-agnostic undesirable feature deactivation using out-of-distribution data. Advances in Neural Information Processing Systems, 34, 2021.
  36. The use of machine learning algorithms in recommender systems: A systematic review. Expert Systems with Applications, 97:205–227, 2018.
  37. Catboost: unbiased boosting with categorical features, 2019.
  38. Leslie N. Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464–472, 2017. doi: 10.1109/WACV.2017.58.
  39. Applications of machine learning in drug discovery and development. Nature reviews Drug discovery, 18(6):463–477, 2019.
  40. On Calibration and Out-of-domain Generalization. arXiv e-prints, art. arXiv:2102.10395, feb 2021.
  41. Gradient centralization: A new optimization technique for deep neural networks, 2020.
  42. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. arXiv e-prints, art. arXiv:1904.00962, apr 2019.
  43. CRUDE: Calibrating Regression Uncertainty Distributions Empirically. arXiv e-prints, art. arXiv:2005.12496, may 2020.
  44. From machine learning to deep learning: progress in machine intelligence for rational drug discovery. Drug discovery today, 22(11):1680–1685, 2017.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets