Learn When and Where to Connect: Adaptive Virtual Nodes for Dynamic Message Passing on Graphs

Abstract: While Virtual Nodes (VNs) are often utilized in Message Passing Neural Networks (MPNNs) to facilitate effective message passing, existing VN-based methods have limitations, such as constraining all nodes to connect to the same number of VNs, fixing the connections before applying MPNNs, and connecting a node to a VN independently of the other nodes that connect to the same VN. We propose MAVN, an end-to-end differentiable MPNN framework that allows non-constrained connections between nodes and VNs and dynamically introduces VNs on demand in response to evolving node representations across layers. Specifically, MAVN learns to adaptively determine when (at which layer) and where (to which nodes) to introduce and connect VNs based on the relative importance of connections. From a pool of candidate VNs, MAVN selects the necessary VNs in each layer, where each selected VN is connected to a nonempty subset of nodes, guided by a dual-perspective scoring mechanism that jointly captures the nodes' preferences for VNs and the VNs' preferences for nodes. We theoretically prove that for any node-VN connectivity pattern, there exists a set of MAVN's parameters that can simulate the pattern. Experiments on nine real-world datasets demonstrate that MAVN consistently improves the performance of backbone MPNNs, achieving up to 46.5% improvement over the backbones and outperforms the baselines.

• The paper develops MAVN, which dynamically selects virtual nodes at each layer to address under-reaching, over-squashing, and over-smoothing in message passing.
• It introduces a dual-perspective scoring mechanism that adapts VN-node connections based on evolving representations, achieving superior expressiveness compared to fixed VN methods.
• Experimental results show up to 46.5% performance improvements on diverse benchmarks while maintaining linear computational complexity.

Adaptive Virtual Nodes for Dynamic Message Passing on Graphs

Introduction and Motivation

Message Passing Neural Networks (MPNNs) provide a foundational approach for graph representation learning by aggregating information from local neighborhoods to iteratively refine node representations. Despite their algorithmic and empirical utility, standard MPNNs face well-documented limitations: under-reaching (inability to capture long-range dependencies with insufficient depth), over-smoothing (induced indistinguishability in node representations with excessive depth), and over-squashing (compression of information from extensive receptive fields into limited-dimensional embeddings). Remedial architectural modifications have centered on rewiring the graph structure or introducing auxiliary structures such as Virtual Nodes (VNs).

While previous VN-based strategies (e.g., IPR-MPNN, N$^{2}$) ameliorate some algorithmic bottlenecks, they are encumbered by inflexible design choices: every node connects to the same number of VNs, connectivity patterns are predetermined and layer-invariant, and selection is performed independently, neglecting critical interaction effects. The paper "Learn When and Where to Connect: Adaptive Virtual Nodes for Dynamic Message Passing on Graphs" (2606.03068) introduces MAVN (Message passing with Adaptive Virtual Nodes), which dynamically determines both the timing (when) and topology (where) of VN attachment, governed by the node and graph's intermediate representations at each layer.

Figure 1: Existing VN-based methods connect all nodes to fixed VNs per layer (top); MAVN allows nodes and VNs to form variable, adaptive, layer-dependent connections (bottom).

The MAVN Architecture

Dynamic, Layer-wise VN Selection and Connection Formation

MAVN constructs a pool of candidate VNs and, at each layer, dynamically selects a variable subset for introduction to the computational graph. A scoring mechanism, based on learned functions of the evolving node and VN representations, determines not only which VNs are instantiated but also which nodes are connected to which VNs. Importantly, this is achieved via dual-perspective scoring: connections are evaluated by jointly considering the preferences of both nodes (for particular VNs) and VNs (for specific nodes), realized through a calibrated relevance score with adaptive regularization based on the connectivity pattern.

Figure 2: Blockwise overview of MAVN. At each layer, a subset of VNs is selected and connected to a nonempty subset of nodes; MAVN updates node and VN features and may also introduce connections between VNs.

This schema contrasts sharply with prior VN-based frameworks: connection number and topology are not globally fixed but reflect the local demands of the representation space at each depth. MAVN's design ensures that the effective graph structure, used for message passing, can flexibly adapt to mitigate under-reaching and over-squashing in a data- and context-sensitive manner.

Expressiveness and Universality

A formal result establishes that MAVN is strictly more expressive than fixed VN-based methods: for any connectivity pattern between nodes and VNs, including those constructed by baseline approaches, there exists a MAVN parameterization producing the same message passing topology in a single layer. This follows from the universal approximation capability of MLPs deployed in the scoring mechanism and the non-constraining nature of the selection process.

Pathological Limitations: Empirical Analyses

Mitigation of Under-reaching and Over-squashing

Synthetic benchmarks elucidate MAVN's ability to bypass canonical MPNN failures. On tree-structured graphs, where information flow to distal leaves is bottlenecked, MAVN introduces auxiliary VNs as strategic information conduits, yielding perfect label propagation where GCN baselines fail.

Figure 3: Example from synthetic tree – purple nodes are VNs adaptively introduced to shortcut message passing and overcome under-reaching.

Over-squashing and Effective Resistance

Assessment of total effective resistance—a topological metric correlated with over-squashing—shows a systematic reduction in MAVN-augmented graphs. The addition of VNs and associated edges increases path connectivity, broadening the available message passing channels without incurring quadratic complexity typical of uniform rewiring.

Figure 4: MAVN consistently lowers the log-average total effective resistance, indicating alleviation of over-squashing.

Over-smoothing: Dirichlet Energy Diagnostics

Contrary to many MPNN extensions that inadvertently induce over-smoothing, MAVN avoids exponential decay in the Dirichlet energy of node representations with depth, confirming the layer-dependent interactions between nodes and VNs preserve representational diversity.

Figure 5: Dirichlet energy analysis reveals no exponential decay with depth, i.e., no over-smoothing under MAVN.

Experimental Results

MAVN is evaluated on nine datasets (including LRGB and heterophilic graphs) with multiple backbone MPNNs. Across all benchmarks, MAVN consistently improves backbone performance—by up to 46.5%—and exceeds or matches state-of-the-art methods for both node and graph-level tasks. The efficacy is observed with both architectural variants (GCN, GAT, GraphSAGE, GINE, GatedGCN) and evaluation setups (inductive and transductive).

Additionally, ablation studies demonstrate that the principal gain derives from (a) allowing VN introduction and connection decisions to be layer-wise and adaptive, (b) using dual-perspective scoring, and (c) integrating preference regularization that emphasizes relative over absolute relevance scores. Notably, full connectivity or restricting VN dynamism sharply reduces performance, especially in graphs where local topology and long-range interactions are highly variable.

Computational Complexity and Practicality

Despite the expressive flexibility, MAVN maintains linear complexity in the number of nodes ($\mathcal{O}(M|\mathcal{V}|)$ with $M \ll |\mathcal{V}|$) at practical settings, ensuring scalability. The computational overhead from VN selection and connection scoring does not offset the efficiency gains relative to quadratic-complexity rewiring approaches, and empirical runtime and memory profiling confirm sublinear scaling with increasing VN budget.

Implications, Limitations, and Future Directions

Theoretically, MAVN generalizes static, uniform VN design and stochastic rewiring by providing a data-driven, context-adaptive augmentation of the computational graph at each layer. Practically, this allows backbone-agnostic enhancement—MAVN can wrap any MPNN—and demonstrates robust utility on synthetic and real-world benchmarks. The framework's design introduces a template for future adaptive graph augmentation: VNs here are interpreted as dynamically induced cluster centers or role actors, and extending the approach to hierarchical graph pooling, coarsening, or dynamic graph abstraction is a natural next step.

One limitation is the need for well-chosen VN budget $M$, although the model's score-driven selection mechanism mitigates the sensitivity. For very large-scale graphs, memory efficiency and distributed parallelization will be essential areas for engineering adaptation.

Conclusion

MAVN presents an expressive, adaptive framework for MPNNs, fundamentally improving the dynamic construction of message passing paths via flexible, data-driven virtual node selection and connection mechanisms. The formal expressivity result and systematic empirical improvements across diverse tasks establish MAVN as a robust architectural advance, providing a foundation for future explorations in adaptive graph learning, graph coarsening, and hierarchical representation.

(2606.03068)