Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels (1902.11038v2)

Abstract: Graph Convolutional Networks(GCNs) play a crucial role in graph learning tasks, however, learning graph embedding with few supervised signals is still a difficult problem. In this paper, we propose a novel training algorithm for Graph Convolutional Network, called Multi-Stage Self-Supervised(M3S) Training Algorithm, combined with self-supervised learning approach, focusing on improving the generalization performance of GCNs on graphs with few labeled nodes. Firstly, a Multi-Stage Training Framework is provided as the basis of M3S training method. Then we leverage DeepCluster technique, a popular form of self-supervised learning, and design corresponding aligning mechanism on the embedding space to refine the Multi-Stage Training Framework, resulting in M3S Training Algorithm. Finally, extensive experimental results verify the superior performance of our algorithm on graphs with few labeled nodes under different label rates compared with other state-of-the-art approaches.

• The paper presents the M3S algorithm that integrates multi-stage training with DeepCluster to refine label propagation in GCNs.
• It reveals the 'Layer Effect', showing that deeper network architectures enhance information flow when labeled data is sparse.
• Experimental results on Cora, CiteSeer, and PubMed demonstrate significantly improved classification accuracy under low label conditions.

Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes

The paper presents a novel approach to improving the performance of Graph Convolutional Networks (GCNs) in scenarios where only a few labeled nodes are available. The primary contribution of the research is the Multi-Stage Self-Supervised (M3S) Training Algorithm—a combination of a newly proposed Multi-Stage Training Framework and a self-supervised learning technique called DeepCluster. Unlike traditional GCNs, which depend heavily on a large amount of labeled data, this approach aims to refine the learning process even when labeled data is sparse.

Key Findings and Methodology

1. Layer Effect on Sparse Data: One of the core observations of the paper is the "Layer Effect", which states that GCNs require more layers to maintain their performance as the number of labeled nodes reduces. This implies that deeper network architectures are beneficial in scenarios with sparse labels to facilitate effective propagation of label information.
2. Symmetric Laplacian Smoothing: The inherent property of GCNs—Symmetric Laplacian Smoothing—limits the number of convolution layers, thus affecting performance when labeled data is limited due to inefficient label propagation. The authors leverage this understanding to propose enhancements via self-supervised learning.
3. Multi-Stage Training Framework: This framework extends traditional self-training by iteratively adding the most confidently predicted nodes with virtual labels across multiple stages, thereby enhancing label information dissemination. This addresses challenges with sparse labels by dynamically increasing the labeled dataset.
4. DeepCluster and Aligning Mechanism: DeepCluster is utilized to create pseudo-labels by clustering nodes in the embedding space. An aligning mechanism further refines these pseudo-labels by associating them with corresponding classes based on embedding distances. This permits more accurate virtual label assignments.
5. Integration into M3S Training Algorithm: By integrating DeepCluster's self-checking with the Multi-Stage Training Framework, the M3S Algorithm iteratively refines the labeled set, leading to superior performance on datasets with sparse labels. The integration ensures robust propagation of label signals by effectively using pseudo labels.

Experimental Findings

The proposed algorithm demonstrates superior classification accuracy across three standard datasets—Cora, CiteSeer, and PubMed—compared to other state-of-the-art methods. Notably, the advantages are most pronounced in conditions with low label rates, confirming the effectiveness of combining self-supervised learning with multi-stage training to handle sparse labeled data scenarios.

Implications and Future Developments

The proposed M3S algorithm significantly improves the capability of GCNs to operate well in weakly supervised settings. The introduction of self-supervised mechanisms opens avenues for further research into extending these concepts to other domains such as image or sentence classification where labeled data scarcity is a typical issue. Additionally, investigating other self-supervised methods that could potentially integrate with this framework would be valuable.

The paper implicitly encourages broader exploration of the synergy between sparse label learning frameworks and self-supervised techniques that could help generalize this approach to various data structures and machine learning tasks. Future research could focus on refining the algorithm's elements, exploring alternate clustering techniques or aligning mechanisms, and identifying optimal configurations that further maximize performance on generalized tasks with minimal labeled input.