Understanding Negative Sampling in Graph Representation Learning (2005.09863v2)

Abstract: Graph representation learning has been extensively studied in recent years. Despite its potential in generating continuous embeddings for various networks, both the effectiveness and efficiency to infer high-quality representations toward large corpus of nodes are still challenging. Sampling is a critical point to achieve the performance goals. Prior arts usually focus on sampling positive node pairs, while the strategy for negative sampling is left insufficiently explored. To bridge the gap, we systematically analyze the role of negative sampling from the perspectives of both objective and risk, theoretically demonstrating that negative sampling is as important as positive sampling in determining the optimization objective and the resulted variance. To the best of our knowledge, we are the first to derive the theory and quantify that the negative sampling distribution should be positively but sub-linearly correlated to their positive sampling distribution. With the guidance of the theory, we propose MCNS, approximating the positive distribution with self-contrast approximation and accelerating negative sampling by Metropolis-Hastings. We evaluate our method on 5 datasets that cover extensive downstream graph learning tasks, including link prediction, node classification and personalized recommendation, on a total of 19 experimental settings. These relatively comprehensive experimental results demonstrate its robustness and superiorities.

• The paper establishes the importance of negative sampling in graph representation learning and proposes a novel theoretical distribution.
• The paper proposes the MCNS method, which efficiently samples negative instances guided by the new theoretical distribution.
• Experiments show MCNS consistently outperforms baseline methods in graph tasks, demonstrating improved performance and robustness.

Understanding Negative Sampling in Graph Representation Learning

The exploration of graph representation learning has garnered significant attention in recent years, where sampling plays a crucial role. The research presented in "Understanding Negative Sampling in Graph Representation Learning" offers a comprehensive analysis of negative sampling, specifically addressing its importance alongside positive sampling, which is often the focus of the field. This paper explores the theoretical underpinnings of negative sampling, presenting a new framework and methodology that notably impact practical applications like link prediction, node classification, and recommendation systems.

Theoretical Foundation and Contributions

The authors begin by establishing the theoretical significance of negative sampling through an examination of its influence on the optimization objective and variance. They demonstrate that negative sampling is essential for accurate representation learning, showing it is as vital as positive sampling. The paper derives a novel theoretical distribution for effective negative sampling, expressed as $p_n(u|v) \propto p_d(u|v)^\alpha$, with $0 < \alpha < 1$. This finding challenges existing notions, suggesting that negative sampling distributions should be positively, but sub-linearly, correlated with positive sampling distributions. This perspective provides a structured approach to negative sampling, moving beyond heuristic methods previously employed.

Methodology: MCNS Framework

Guided by the theoretical insights, the authors propose Markov Chain Monte Carlo Negative Sampling (MCNS), which operationalizes these theoretical insights. MCNS employs a self-contrast approximation to estimate the unknown positive distribution by leveraging the current graph embeddings. This approximation addresses computational inefficiencies inherent in previous methods. A key innovation in MCNS is its use of the Metropolis-Hastings algorithm to sample from the complex space efficiently, skipping the burn-in period by leveraging the natural assumption of similarity between adjacent nodes.

Experimental Evaluation

The paper evaluates MCNS across three graph learning tasks using various datasets: MovieLens, Amazon, Alibaba for recommendation; Arxiv for link prediction; and BlogCatalog for node classification. These experiments cover a diverse array of settings, demonstrating that MCNS outperforms eight other negative sampling strategies consistently across a total of 19 experimental setups. Notably, MCNS surpasses hard-sampling and GAN-based methods, presenting improvements in metrics such as MRR and Hits@30. The robustness of the MCNS approach is evident as it integrates seamlessly with different encoders like DeepWalk, GCN, and GraphSAGE.

Practical Implications and Future Directions

The findings from this research have profound implications for the development of graph-based machine learning applications. By presenting a theoretically grounded approach to negative sampling, this paper paves the way for more stable and reliable graph representation learning models. Practically, this means more accurate recommendations, classifications, and predictions in real-world applications, from social networks to e-commerce platforms.

Future research could extend this work by exploring the dynamic adaptation of the negative sampling distribution over the course of model training, further enhancing the efficiency and effectiveness of representation learning. Additionally, integrating this negative sampling strategy with emerging graph neural network frameworks could further enhance performance in large-scale and complex graph datasets.

In summary, "Understanding Negative Sampling in Graph Representation Learning" provides a solid theoretical and practical framework for advancing the sampling methodologies crucial to graph representation learning, offering both clarity and innovation to the field.