Feature-Message Passing gMLP

• Feature-message passing gMLP is a graph neural network paradigm that decouples message propagation from neural updates, enabling efficient multi-hop feature integration.
• It leverages cycle message passing and adaptive gating to encode high-order topological features in spatio-temporal graphs.
• Empirical findings demonstrate state-of-the-art accuracy and significant training acceleration compared to traditional GNN frameworks.

Feature-message passing gMLP architectures build upon the core idea of decoupling the message propagation and neural update in graph neural networks (GNNs). This separation enables scalable, adaptive, and expressive learning in large or topologically rich graphs. Recent work extends these principles to spatio-temporal and topological domains, notably with the introduction of Cycle Message Passing (Cy2Mixer), which uses parallel gMLP blocks for temporal, spatial, and cycle-aware feature mixing. This approach enhances both scalability and topological sensitivity for a variety of graph-based learning tasks (Lee et al., 2024, Zhang et al., 2021).

1. Background and Motivation

Classical neural message passing in GNNs alternately aggregates the hidden states of each node's neighbors and applies neural updates. This recursive expansion is computationally demanding and lacks flexibility—the message sets are fixed for all nodes and layers, and scaling to large graphs is prohibitive. To address these issues, GMLP (Graph Multi-layer Perceptron) frameworks completely decouple feature propagation from neural computation by pre-computing multi-hop, feature-only messages and then learning to mix them adaptively with small, shared neural models (Zhang et al., 2021).

In spatio-temporal graphs such as traffic or interaction networks, purely spatial or temporal models are insufficient; rich cycles and higher-order connectivity patterns carry crucial information. However, standard message passing, even with multi-head attention, cannot explicitly align with the graph's deeper topological invariants. The Cy2Mixer architecture introduces cycle message passing blocks, informed by the cycle basis (first homology) of the graph, to address this gap (Lee et al., 2024).

2. Feature-Message Passing: Mathematical Structure

Given a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ with node features $\mathbf{x}_v\in\mathbb{R}^d$ for $v\in\mathcal{V}$, and integer $T$ denoting the maximal hop, feature-message passing initializes

$$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$

For $t=1,2,\dots,T$,

$$\mathbf{m}_v^{(t)} = \mathsf{graph\_aggregator}\left(\{\mathbf{m}_u^{(t-1)}: u\in\mathcal{N}(v)\}\right),$$

where $\mathcal{N}(v)$ is the (possibly self-looped) neighbor set. This produces a multiscale message set: $$\mathcal{M}_v = \left\{\mathbf{m}_v^{(0)},\,\mathbf{m}_v^{(1)},\,\dots,\,\mathbf{m}_v^{(T)}\right\}.$$ Messages may be aggregated across hops via concatenation, pooling, gating,

$$w_t = \sigma(\mathbf{s}^\top \mathbf{m}_v^{(t)}),\quad \mathbf{c}_v = \sum_{t=0}^T w_t\,\mathbf{m}_v^{(t)},$$

or adaptive attention (see (Zhang et al., 2021) for detailed formulations).

The neural update step then forms the final representation: $\mathbf{x}_v\in\mathbb{R}^d$0 where $\mathbf{x}_v\in\mathbb{R}^d$1 is typically an MLP. This entire process allows the expensive graph propagation ($\mathbf{x}_v\in\mathbb{R}^d$2 for $\mathbf{x}_v\in\mathbb{R}^d$3 edges) to be pre-computed, with subsequent epochs reduced to cheap MLP forward/backward passes ($\mathbf{x}_v\in\mathbb{R}^d$4) (Zhang et al., 2021).

3. gMLP-Based Gating and Unified Cy2Mixer Blocks

In Cy2Mixer, the gMLP structure is generalized to spatio-temporal tensors $\mathbf{x}_v\in\mathbb{R}^d$5, where $\mathbf{x}_v\in\mathbb{R}^d$6 is history length, $\mathbf{x}_v\in\mathbb{R}^d$7 node count, and $\mathbf{x}_v\in\mathbb{R}^d$8 hidden dimension. Each encoder layer applies: $\mathbf{x}_v\in\mathbb{R}^d$9 and splits $v\in\mathcal{V}$0. The spatial gating unit modulates one half by a function $v\in\mathcal{V}$1 of the other: $v\in\mathcal{V}$2 The function $v\in\mathcal{V}$3 is replaced depending on which stream the block implements:

• Temporal: $v\in\mathcal{V}$4
• Spatial: $v\in\mathcal{V}$5 MPNN on $v\in\mathcal{V}$6
• Cycle: $v\in\mathcal{V}$7 MPNN on $v\in\mathcal{V}$8

Each output is projected,

$v\in\mathcal{V}$9

and the three outputs are concatenated then fused: $T$0 A residual connection and normalization complete the layer. This entirely attention-free architecture is shown to match or exceed transformer-based models on benchmarks (Lee et al., 2024).

4. Cycle (Clique) Message-Passing and Topological Encoding

The cycle message passing block leverages a clique adjacency $T$1 formed by considering cycles in a basis of $T$2: the $T$3 entry of $T$4 is $T$5 if $T$6 and $T$7 co-occur in a fundamental cycle. Theorem 3.1 from (Lee et al., 2024) proves that the cycle basis of the temporal lifted graph $T$8 is isomorphic to that of $T$9, so all necessary topological cycles are captured at each time slice by $$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$0: $$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$1 where $$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$2 runs over a cycle basis of $$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$3.

The cycle block then executes

$$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$4

enriching feature mixing with 1-dimensional homology information.

5. Representational Expressiveness and Block Comparison

Standard (spatial) message passing with adjacency $$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$5 is provably limited: MPNNs with only $$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$6 cannot distinguish graphs that share the same universal cover but differ by cycle attachments, corresponding to failure modes of the Weisfeiler-Lehman test in cycle-rich graphs. The addition of the cycle block permits the network to distinguish between nodes in otherwise identical local unfoldings but distinct cycle contexts (Lee et al., 2024).

This yields greater representational capacity, both mathematically and empirically. The cycle message passing block provides direct access to the graph's nontrivial homology classes, making cycles “visible” to the deep model.

6. Variants, Efficiency, and Empirical Results

The gMLP framework encompasses several variants:

• GMLP-GU: Uses only $$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$7-hop features; equivalent to methods such as SGC given normalized adjacency.
• GMLP-GMU: Sum, gated, or global gate aggregation of multiscale messages.
• Full GMLP: Self-guided attention over hops, resulting in state-of-the-art accuracy and robust learning to depth (Zhang et al., 2021).

Key empirical findings demonstrate:

• On ogbn-products, GMLP achieves $$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$876.5% accuracy, outperforming SIGN and APPNP.
• Training acceleration up to $$\mathbf{m}_v^{(0)} = \mathbf{x}_v \quad \forall\,v.$$9 versus APPNP, and $t=1,2,\dots,T$0 versus GAT, owing to pre-computation.
• Near-linear distributed scaling over 16 GPUs.
• Robustness to depth without over-smoothing; nodes learn retention weights over $t=1,2,\dots,T$1 hops.

For Cy2Mixer, benchmarks confirm that the cycle-aware block yields state-of-the-art performance across spatio-temporal graph datasets, with practical source code made available (Lee et al., 2024).

7. Unification and Extension via gMLP Skeleton

Cy2Mixer demonstrates that one can generalize transformer-like architectures by replacing multi-head attention with parallel gMLP gating units, each using a different token-mixing operator. Any structural operator—including higher-order cliques or k-hop neighborhoods—can be implemented by choosing the mapping $t=1,2,\dots,T$2 appropriately. The architectural principle is modular, feed-forward, and free of attention bottlenecks, yet can attain expressiveness comparable to or exceeding self-attention models on real-world benchmarks.

The gMLP skeleton thus unifies feature-message passing, spatial and cycle-aware aggregation, and temporal convolution within a single, efficient, and highly expressive framework, opening new directions for structural learning in large-scale, topologically intricate graphs (Lee et al., 2024, Zhang et al., 2021).