Graph Neural Network leveraging Higher-order Class Label Connectivity for Heterophilous Graphs

Abstract: Node classification in graph neural networks (GNNs) has been widely applied in various fields of graph analysis. GNNs achieve high-accuracy node classification in homophilous graphs, where nodes with the same class label tend to be connected. However, their performance remains limited in heterophilous graphs, where nodes with different class labels are more likely to be connected. In particular, current GNNs derived from graph convolutional networks cannot capture higher-order class label connectivity, which is frequently observed in real-world heterophilous graphs. To address this issue, we propose a novel classifier, Label Context Classifier (LCC), designed to capture higher-order class label connectivity in directed graphs. LCC estimates the class label of a target node by leveraging label context embeddings that are generated through four distinct types of walks. In addition, our approach allows the integration of LCC and any GNN by adaptively learning their importance. Experimental results demonstrate that GNNs integrated with LCC outperform SOTA methods and the label context embeddings improve the node classification performance in heterophilous directed graphs.

• The paper introduces the LCC framework that leverages higher-order label connectivity using four distinct label walk types to capture multi-hop dependencies.
• It integrates LCC with traditional GNNs via an adaptive ensemble mechanism, achieving significant test accuracy improvements on heterophilous graph datasets.
• Empirical evaluations on diverse datasets validate that modeling directed label walks enhances the discriminative power of GNNs over standard approaches.

Higher-Order Class Label Connectivity in GNNs for Heterophilous Graphs

Introduction and Motivation

Graph Neural Networks (GNNs) have established themselves as the primary paradigm for node classification in graph-structured data, exhibiting high efficacy in homophilous settings where edges are likely to connect nodes of the same class. However, GNN performance sharply declines on heterophilous graphs—a significant limitation given that many real-world networks (e.g., social, biological, and web graphs) exhibit strong heterophily. In such graphs, the relational inductive bias of homophily-centric aggregation is deleterious, causing feature smoothing and label leakage that obfuscate informative patterns.

The examined paper addresses a critical gap: existing GNN architectures fail to exploit higher-order class label connectivity—how class labels are connected beyond simple adjacency (i.e., through multi-hop, directed paths). While prior work leveraged first-order compatibility matrices or hand-designed meta-paths, these approaches are insufficient to capture the intricate, multi-directional dependencies manifest in real heterophilous graphs.

The Label Context Classifier (LCC): Architecture and Label Walks

To remedy these deficiencies, the authors introduce the Label Context Classifier (LCC), a model that operationalizes higher-order class label connectivity by harnessing "label walks". Label walks consist of sequences of class labels traversed according to specific graph motifs (forward, backward, sibling, and guardian walks).

LCC operates in three phases:

1. Extraction of Four Types of Label Walks:
   • Forward walks traverse outgoing directed edges to model downstream label dependencies.
   • Backward walks traverse incoming edges to model upstream label dependencies.
   • Sibling walks sample nodes sharing a common parent (capturing structural equivalence among "siblings").
   • Guardian walks sample nodes sharing a common child ("guardians"), sensitive to convergent connections.
2. Generation of Label Context Embeddings: For each walk, a context embedding is learned by modeling the prediction of context labels given a target node, in the style of the Skip-gram model (word2vec). This is achieved via a simple two-layer MLP, where the internal layer encodes the embedding based on the observed label context in walks.
3. Classification Using Label Context Embeddings: The concatenated vector of a node's attributes and the four walk-type context embeddings serves as input to the final classifier, allowing the model to leverage heterogeneous connectivity channels.

The schematic architecture is depicted below.

Figure 2: LCC framework—four label walk extraction, context embedding generation, and classifier training leveraging the concatenated embeddings.

Empirical Observations of Higher-Order Label Connectivity

Analysis of real-world datasets, such as the Texas web graph, reveals distinct patterns of higher-order label connectivity: for example, "Student" nodes tend to be connected through multi-hop paths to "Department" or "Staff" nodes with strong statistical regularities. Particularly in directed graphs, these dependencies are not symmetric and necessitate direction-aware traversal. Simple first-order models, or models that disregard directionality and higher-order dependencies, are fundamentally incapable of capturing such phenomena.

The following illustrations reflect 2nd-order ("Course" → "Staff" → "Department") and specialized walk-based connectivity (sibling, guardian) that cannot be recovered by neighborhood aggregation alone.

Figure 4: Empirical class label connectivity in the Texas dataset via (a) sibling and (b) guardian walks—demonstrating significant structure absent from first-order adjacency.

Integration of LCC and Existing GNNs

LCC is neither a replacement for classical GNNs nor redundant; it is explicitly designed to complement neighborhood aggregation. The integration strategy consists of training both LCC and a given GNN separately, then constructing a weighted output ensemble, where the weights are adaptively set based on validation performance. The combined prediction is a convex sum of the two models' outputs, and the weighting coefficients are computed as a softmax function over the inverse validation losses, adjusted via a temperature hyperparameter.

This model-agnostic design allows LCC to improve the representational bias of a wide array of GNNs (including those tailored for heterophily, e.g., H2GCN, LINKX, GloGNN), harnessing structural label information orthogonal to message-passing.

The high-level integration mechanism is shown below.

Figure 1: LCC-GNN integration pipeline—final classification via weighted averaging of LCC and GNN predictions, with weights determined by validation losses.

Experimental Evaluation and Contradictory Findings

The benchmark evaluation encompasses seven established heterophilous directed graph datasets, including small-scale (Cornell, Texas, Wisconsin) and large-scale graphs (Chameleon, Squirrel, Roman-Empire, Amazon-Ratings). LCC, both standalone and when integrated with state-of-the-art GNNs, consistently improves test accuracy, often by substantial margins:

• For instance, H2GCN+LCC yields improvements of up to +6.51% over H2GCN and up to +12.24% over LCC-only on selected datasets.
• On Chameleon, Squirrel, Roman-Empire, and Amazon-Ratings, gains over baseline MLP exceed 10%.
• There are limited cases where the ensemble does not outperform the stronger individual component, but this behavior directly correlates with extreme weighting determined during integration, suggesting that additional adaptive weighting schemes might further enhance robustness.

The paper makes a bold and empirically validated claim: standard GNNs and recent heterophily-specialized GNNs still fail to exploit higher-order class label connectivity, and this deficiency materially limits their classification accuracy, often allowing simple MLPs to outperform them in challenging regimes.

Analysis of Label Walk Types and Hyperparameters

Ablation experiments show that the contribution of different walk types (forward, backward, sibling, guardian) is dataset-dependent, but guardian and sibling walks provide particularly strong improvements on datasets with pronounced convergent or structurally-equivalent class association (e.g., Roman-Empire, Amazon-Ratings). Furthermore, optimal walk length varies by graph, with higher-order walks providing greater benefit in large, low-homophily datasets, and shorter walks sufficing in more compact graphs.

Importantly, the effectiveness of LCC is robust to hyperparameter selection, and its computational cost, though higher than simple MLPs, is manageable on modern hardware for the tested benchmark graphs.

Implications and Future Directions

The core insight—that directed, higher-order label connectivity is a missing inductive signal in current GNN frameworks—has substantial implications for both theoretical development and practical engineering of graph representation learning systems. The design of LCC demonstrates that label sequence modeling can be orthogonal to node feature aggregation, and that ensemble mechanisms can leverage this for significant accuracy improvement.

Practical Implications:

• LCC can be incorporated atop existing pipelines with minimal retraining or architectural change.
• Structural label signal, captured through context embeddings, is crucial for graph tasks in heterophilous, directionally constrained domains (e.g., hyperlink graphs, biological interaction networks, recommender systems).

Theoretical Implications and Open Directions:

• Deep label walk modeling opens avenues for generalizing motif-based traversal schemes, potentially beyond the four fundamental types here.
• Replacement of shallow embedding models with more expressive transformers or contrastive techniques may further improve label context modeling.
• The weighting/combination mechanism could be node- or subgraph-adaptive, moving beyond fixed dataset-level weights.

Conclusion

This work identifies and exploits a fundamental, underleveraged signal in node classification for heterophilous directed graphs: higher-order class label connectivity. Through the LCC framework—leveraging directed label walks and walk-derived context embeddings—and a principled integration with conventional GNNs, the paper achieves consistent, substantial gains over state-of-the-art baselines. These results underscore the necessity of revisiting the architectural priors of GNNs in non-homophilous domains, and highlight the value of higher-order, label-driven contexts for advancing the discriminative power of graph neural networks.