In-n-Out: Calibrating Graph Neural Networks for Link Prediction (2403.04605v2)

Abstract: Deep neural networks are notoriously miscalibrated, i.e., their outputs do not reflect the true probability of the event we aim to predict. While networks for tabular or image data are usually overconfident, recent works have shown that graph neural networks (GNNs) show the opposite behavior for node-level classification. But what happens when we are predicting links? We show that, in this case, GNNs often exhibit a mixed behavior. More specifically, they may be overconfident in negative predictions while being underconfident in positive ones. Based on this observation, we propose IN-N-OUT, the first-ever method to calibrate GNNs for link prediction. IN-N-OUT is based on two simple intuitions: i) attributing true/false labels to an edge while respecting a GNNs prediction should cause but small fluctuations in that edge's embedding; and, conversely, ii) if we label that same edge contradicting our GNN, embeddings should change more substantially. An extensive experimental campaign shows that IN-N-OUT significantly improves the calibration of GNNs in link prediction, consistently outperforming the baselines available -- which are not designed for this specific task.

• The paper presents IN-N-OUT, a novel calibration technique that adjusts GNN logits to better match predicted confidences with empirical link outcomes.
• It employs a temperature-scaling approach that modulates predictions based on embedding variability from adding or removing links.
• Extensive experiments across standard datasets show that IN-N-OUT reduces calibration errors compared to conventional methods like Isotonic Regression and Temperature Scaling.

Calibrating Graph Neural Networks for Enhanced Link Prediction Accuracy

Introduction to \texttt{IN-N-OUT}

Graph Neural Networks (GNNs) have established themselves as pivotal instruments in understanding complex relational data, impacting various sectors by providing insights into network structured problems. However, a common critique against these models is their miscalibration - the disparity between their predicted confidences and the actual likelihood of predictions. Such a disjunction poses significant risks, especially in decision-sensitive environments.

Amplifying on this concern for GNNs focused on link prediction tasks, this paper introduces \texttt{IN-N-OUT}, a novel approach for the post-hoc calibration of GNNs. \texttt{IN-N-OUT} particularly addresses the calibration challenge for link prediction by leveraging two key insights about the behavior of GNN predictions and the structural properties of graph embeddings.

Key Insights and Methodology

Primarily, the research underscores the mixed behavior of GNN predictions for link prediction tasks, observed across various datasets and models. GNNs tend to display overconfidence in negative predictions while showing underconfidence in positive ones. To mitigate this miscalibration, the paper presents \texttt{IN-N-OUT}, derived from two main premises:

1. The introduction or absence of a true link should minimally perturb the embedding of the said link if the GNN's prediction aligns with reality.
2. Conversely, if a link's presence contradicts the GNN's prediction, its embedding should exhibit significant variance.

Grounded in these premises, \texttt{IN-N-OUT} employs a temperature-scaling approach that modulates the GNN logits based on the degree of discrepancy between calculated embeddings upon the hypothetical addition or removal of a link.

Empirical Validation and Results

An extensive experimental campaign across multiple standard datasets and GNN architectures showcases \texttt{IN-N-OUT}'s superior performance in enhancing the calibration of GNNs for link prediction. The approach consistently outperforms conventional calibration methods like Isotonic Regression, Temperature Scaling, and others. Specifically, \texttt{IN-N-OUT} achieved the lowest calibration errors in the majority of tested scenarios, indicating a closer alignment between predicted probabilities and empirical outcomes.

Practical and Theoretical Implications

This paper not only illuminates the calibration issues inherent in GNNs for link prediction but also provides a robust tool to mitigate such challenges, thereby enhancing the reliability of GNN predictions in sensitive applications. On a theoretical level, the methodology propels a deeper understanding of the interaction between graph embeddings and GNN predictions, potentially guiding future explorations in improving GNN architectures.

Towards Future Developments in AI

While focusing on link prediction, the implications of this paper stretch beyond, hinting at the potential for similar calibration strategies across other graph-related tasks. The interplay between graph structure, embeddings, and predictive certainty uncovered here lays a foundational stone for future advancements in calibration methods, fostering the development of more reliable and interpretable GNNs across varied applications.

Conclusion

\texttt{IN-N-OUT} presents a significant step forward in calibrating GNNs for link prediction, addressing the nuanced predictive behaviors of these models. Through meticulous experimentation and insightful methodology, this work not only advances our understanding of GNN calibration but also sets the groundwork for further innovations in the field of graph neural networks. As we push the boundaries of what AI can achieve, ensuring the reliability of our models remains paramount, with \texttt{IN-N-OUT} marking a notable advancement towards this goal.