Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networks (2309.10976v1)

Published 20 Sep 2023 in cs.LG

Abstract: Safe deployment of graph neural networks (GNNs) under distribution shift requires models to provide accurate confidence indicators (CI). However, while it is well-known in computer vision that CI quality diminishes under distribution shift, this behavior remains understudied for GNNs. Hence, we begin with a case study on CI calibration under controlled structural and feature distribution shifts and demonstrate that increased expressivity or model size do not always lead to improved CI performance. Consequently, we instead advocate for the use of epistemic uncertainty quantification (UQ) methods to modulate CIs. To this end, we propose G-$\Delta$UQ, a new single model UQ method that extends the recently proposed stochastic centering framework to support structured data and partial stochasticity. Evaluated across covariate, concept, and graph size shifts, G-$\Delta$UQ not only outperforms several popular UQ methods in obtaining calibrated CIs, but also outperforms alternatives when CIs are used for generalization gap prediction or OOD detection. Overall, our work not only introduces a new, flexible GNN UQ method, but also provides novel insights into GNN CIs on safety-critical tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell., 2012.
  2. On the bottleneck of graph neural networks and its practical implications. In Proc. Int. Conf. on Learning Representations (ICLR), 2021.
  3. Size-invariant graph representations for graph classification extrapolations. In Proc. Int. Conf. on Machine Learning (ICML), 2021.
  4. Weight uncertainty in neural network. In Proc. Int. Conf. on Machine Learning (ICML), 2015.
  5. Sizeshiftreg: a regularization method for improving size-generalization in graph neural networks. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2022.
  6. Learning causally invariant representations for out-of-distribution generalization on graphs. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2022.
  7. Tree mover’s distance: Bridging graph metrics and stability of graph neural networks. In Proc. Adv. in Neural Information Processing Systems NeurIPS, 2022.
  8. Principal neighbourhood aggregation for graph nets. In NeurIPS, 2020.
  9. Torchmetrics - measuring reproducibility in pytorch, 2022. URL https://github.com/Lightning-AI/torchmetrics.
  10. A closer look at distribution shifts and out-of-distribution generalization on graphs. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
  11. Benchmarking graph neural networks. CoRR, 2020.
  12. Graph neural networks with learnable structural and positional representations. In Proc. Int. Conf. on Learning Representations (ICLR), 2022a.
  13. Long range graph benchmark. In Proc. Adv. in Neural Information Processing Systems NeurIPS, Datasets and Benchmark Track, 2022b.
  14. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proc. Int. Conf. on Machine Learning (ICML), 2016.
  15. Leveraging unlabeled data to predict out-of-distribution performance. In Proc. Int. Conf. on Learning Representations (ICLR), 2022.
  16. Utilising graph machine learning within drug discovery and development. CoRR, abs/2012.05716, 2020.
  17. GOOD: A graph out-of-distribution benchmark. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), Benchmark Track, 2022.
  18. Predicting with confidence on unseen distributions. In ICCV, 2021.
  19. On calibration of modern neural networks. In Proc. of the Int. Conf. on Machine Learning, (ICML), 2017.
  20. A generalization of vit/mlp-mixer to graphs. CoRR, abs/2212.13350, 2022.
  21. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proc. Int. Conf. on Learning Representations (ICLR), 2017.
  22. Deep anomaly detection with outlier exposure. In Proc. Int. Conf. on Learning Representations (ICLR), 2019.
  23. Unsolved problems in ML safety. CoRR, abs/2109.13916, 2021.
  24. Scaling out-of-distribution detection for real-world settings. In Proc. Int. Conf. on Machine Learning (ICML), 2022a.
  25. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proc. Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2022b.
  26. What makes graph neural networks miscalibrated? In Proc. Adv. in Neural Information Processing Systems NeurIPS, 2022.
  27. Predicting the generalization gap in deep networks with margin distributions. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  28. Jurygcn: Quantifying jackknife uncertainty on graph convolutional networks. In Proc. Int. Conf. on Knowledge Discovery & Data Mining, KDD, 2022.
  29. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
  30. Pytorch-ood: A library for out-of-distribution detection based on pytorch. In Workshop at the Proc. Int. Conf. on Computer Vision and Pattern Recognition CVPR, 2022.
  31. Understanding attention and generalization in graph neural networks. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2019.
  32. Verified uncertainty calibration. In Proc. Adv. in Neural Information Processing Systems NeurIPS, 2019.
  33. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2017.
  34. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Proc. Adv. in Neural Information Processing Systems NeurIPS, 2018.
  35. Distance encoding: Design provably more powerful neural networks for graph representation learning. In Proc. Adv. in Neural Information Processing Systems NeurIPS, 2020.
  36. Energy-based out-of-distribution detection. In Proc. Adv. in Neural Information Processing Systems NeurIPS, 2020.
  37. Revisiting the calibration of modern neural networks. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2021.
  38. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020. URL www.graphlearning.io.
  39. Attending to graph transformers. CoRR, abs/2302.04181, 2023.
  40. Obtaining well calibrated probabilities using bayesian binning. In Proc. Conf. on Adv. of Artificial Intelligence (AAAI), 2015.
  41. Learning to extrapolate: A transductive approach. In Proc. Int. Conf. on Learning Representations (ICLR), 2023.
  42. Predicting out-of-domain generalization with local manifold smoothness. CoRR, abs/2207.02093, 2022.
  43. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In Proc. Adv. in Neural Information Processing Systems NeurIPS, 2019.
  44. Recipe for a General, Powerful, Scalable Graph Transformer. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2022.
  45. Do bayesian neural networks need to be fully stochastic? In AISTATS, 2023.
  46. Single model uncertainty estimation via stochastic data centering. In Proc. Adv. in Neural Information Processing Systems (NeurIPS), 2022.
  47. Understanding over-squashing and bottlenecks on graphs via curvature. In Proc. Int. Conf. on Learning Representations ICLR, 2022.
  48. A closer look at scoring functions and generalization prediction. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.  1–5. IEEE, 2023a.
  49. A closer look at model adaptation using feature distortion and simplicity bias. In Proc. Int. Conf. on Learning Representations (ICLR), 2023b.
  50. Graph attention networks. In ICLR, 2018.
  51. Vim: Out-of-distribution with virtual-logit matching. In Proc. Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2022a.
  52. Equivariant and stable positional encoding for more powerful graph neural networks. In Proc. Int. Conf. on Learning Representations (ICLR), 2022b.
  53. Be confident! towards trustworthy graph neural networks via confidence calibration. In Proc. Adv. in Neural Information Processing Systems NeurIPS, 2021.
  54. A Fine-Grained Analysis on Distribution Shift. In Proc. Int. Conf. on Learning Representations (ICLR), 2022.
  55. How powerful are graph neural networks? In ICLR, 2019.
  56. Optimization of graph neural networks: Implicit acceleration by skip connections and more depth. In Proc. Int. Conf. on Machine Learning (ICML), 2021.
  57. Groupinn: Grouping-based interpretable neural network for classification of limited, noisy brain data. In Proc. Int. Conf. on Knowledge Discovery & Data Mining, KDD, 2019.
  58. Graph R-CNN for scene graph generation. In Proc. Euro. Conf. on Computer Vision (ECCV), 2018.
  59. From local structures to size generalization in graph neural networks. In International Conference on Machine Learning, pp.  11975–11986. PMLR, 2021.
  60. Link prediction based on graph neural networks. In Proc. Adv. in Neural Information Processing Systems NeurIPS, 2018.
  61. From stars to subgraphs: Uplifting any GNN with local structure awareness. In Proc. Int. Conf. on Learning Representations (ICLR), 2022.
  62. A survey on deep graph generation: Methods and applications. In Learning on Graphs Conference (LoG), 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now