Higher-Order Equivariant Message Passing

• Higher-order equivariant message passing is a framework that extends traditional graph neural networks by aggregating information over multi-way relationships to capture richer structural patterns.
• It leverages advanced mathematical formalisms such as CW-complexes and permutation-equivariant tensor maps to maintain symmetry under group actions.
• Empirical studies demonstrate improved expressivity and generalization in applications like molecular simulation, geometric analysis, and topological learning.

Higher-order equivariant message passing refers to neural architectures that propagate information not only along edges (pairs of objects) but also across higher-dimensional or multi-way relationships—such as triangles, rings, general subgraphs, or topological cells—while preserving equivariance under group actions (e.g., E(n) for geometric data, permutation groups for combinatorial arrangements). This class of models aims to capture interactions and structural motifs that cannot be encoded by conventional pairwise (first-order) message passing, delivering higher expressivity and improved generalization in geometric, chemical, and topological learning tasks.

1. Mathematical Formalisms for Higher-order Equivariance

Higher-order equivariant message passing generalizes classical graph neural network (GNN) updates to richer combinatorial objects:

• CW-complex and Simplicial Complex Formalism: EMPCN (Kovač et al., 2024) employs a general CW-complex representation where each cell (0-cell: node, 1-cell: edge, 2-cell: polygon/ring, k-cell: higher structures) admits features. Message passing involves aggregating over boundary (𝓑), upper adjacency (𝓝_↑), and—optionally—coboundary relations. In Clifford-simplicial models (Liu et al., 2024), similar concepts are formalized for abstract simplicial complexes using grade-structured Clifford algebra features.
• Permutation-Equivariant Tensors (P-tensors): (Sun et al., 2023) formalizes features as tensors attached to subgraphs, transforming according to the action of Sₙ (symmetries of vertex arrangements). Linear equivariant maps between k-order and ℓ-order P-tensors are generated from a basis indexed by set partitions, systematically constructing all possible equivariant linear message updates between subgraphs.
• Tensor and Geometric Algebraic Encodings: Clifford algebra-based models (Tran et al., 2024, Liu et al., 2024) extend the feature spaces to multivectors (scalars, vectors, bivectors, etc.) enabling natural treatment of higher-order geometrical objects (areas, volumes, etc.) under O(n) or E(n) symmetry. This approach is particularly well-suited for geometric deep learning settings.

2. Core Message Passing Equations and Invariant Construction

Higher-order message updates extend the aggregation paradigm from pairs to sets or cells of arbitrary size while carefully constructing message functions to enforce equivariance:

• General Cell-wise Updates (EMPCN):
  • For a k-cell σ, aggregate messages from boundary 𝓑(σ) and from upper adjacency 𝓝_↑(σ):

  $m_𝓑(σ) = \bigoplus_{\tau\in𝓑(σ)} \mathrm{Mes}_𝓑(h_σ, h_τ, \mathrm{Inv}(σ, τ))$

  $m_↑(σ) = \bigoplus_{\tau\in𝓝_↑(σ)} \bigoplus_{δ\in𝓒(σ)\cap𝓒(τ)} \mathrm{Mes}_↑(h_σ, h_τ, h_δ, \mathrm{Inv}(σ, τ, δ))$

  $h'_σ = \mathrm{Update}(h_σ, m_𝓑(σ), m_↑(σ))$ - All features and invariant quantities (e.g., distances, areas, volumes, angles) are derived from the geometry of σ ∪ τ or higher unions via barycentric subdivision or convex hulls, ensuring E(n)-invariance.

• P-tensor Linear Maps:
  • General message from k- to ℓ-order tensor is constructed as:

  $$M_{k\to\ell}(T^{(k)}) = \sum_{P} W_{P}\;\Phi_{P}(T^{(k)})$$

  where $\Phi_{P}$ are canonical contraction/broadcasting patterns defined by set partitions.

• Clifford Algebra and Multivector Updates:
  • Node features $\mathbf{h}_i^l$ in $\mathrm{Cl}(\mathbb{R}^n)$ are updated via summation of d-order messages over all d-tuples in the neighborhood:

  $$\mathbf{m}_i^{(d)} = \sum_{A\subseteq\mathcal{N}^k(i),\,|A|=d} \phi_m^{(d)}(\mathbf{h}_i^l, \sum_{j\in A} \mathbf{h}_j^l)$$

  $$\mathbf{h}_i^{l+1} = \phi_h(\mathbf{h}_i^l, \mathbf{m}_i^{(1)}, \ldots, \mathbf{m}_i^{(D)})$$

3. Computational Design and Scalability Strategies

A dominant challenge in higher-order message passing is the rapid combinatorial growth of adjacency and message terms, especially as the order increases ($\mathcal{O}(|V|^{k+1})$ in a k-cell complex):

• Decoupled Message Passing (EMPCN): (Kovač et al., 2024) introduces a "decoupled" scheme that restricts dense message passing to the node-level (EGNN), while introducing a sparse auxiliary structure connecting higher-order cells only to their constituent vertices. This design keeps total complexity at $m_↑(σ) = \bigoplus_{\tau\in𝓝_↑(σ)} \bigoplus_{δ\in𝓒(σ)\cap𝓒(τ)} \mathrm{Mes}_↑(h_σ, h_τ, h_δ, \mathrm{Inv}(σ, τ, δ))$0 when #cells ≲ |V|, i.e., comparable to baseline (pairwise-only) GNNs.
• Hybrid P-tensor Architectures: (Sun et al., 2023) recommends restricting maximum tensor order $m_↑(σ) = \bigoplus_{\tau\in𝓝_↑(σ)} \bigoplus_{δ\in𝓒(σ)\cap𝓒(τ)} \mathrm{Mes}_↑(h_σ, h_τ, h_δ, \mathrm{Inv}(σ, τ, δ))$1 to 3 or 4, using Bell number truncation for basis size, and sharing parameters across isomorphic subgraphs to mitigate the parameter and compute blowup.
• Algebraic Truncation and Feature Control: Clifford and Cartesian tensor frameworks often truncate to low-grade (grade ≤ 2 or 3) and enforce sparse couplings in the message construction, focusing on geometrically and physically relevant orders.

4. Empirical Performance and Expressivity Gains

Higher-order equivariant message passing demonstrably increases both task-specific accuracy and the ability to distinguish complex structural patterns:

• Expressivity Enhancements: CW-complex and P-tensor approaches can distinguish molecular motifs (rings, cycles, functional groups) and topological signatures invisible to first-order models. Simplicial and hypergraph frameworks (e.g. MultiSet) enable capturing of network homophily and higher-order connectivity (Telyatnikov et al., 2023, Batatia, 2023).
• Benchmark Results: On N-body systems, QM9, and motion capture tasks, EMPCNs yield substantial improvements in MSE and MAE (e.g., CMU 30-step prediction: EGNN MSE $m_↑(σ) = \bigoplus_{\tau\in𝓝_↑(σ)} \bigoplus_{δ\in𝓒(σ)\cap𝓒(τ)} \mathrm{Mes}_↑(h_σ, h_τ, h_δ, \mathrm{Inv}(σ, τ, δ))$2 vs. EMPCN $m_↑(σ) = \bigoplus_{\tau\in𝓝_↑(σ)} \bigoplus_{δ\in𝓒(σ)\cap𝓒(τ)} \mathrm{Mes}_↑(h_σ, h_τ, h_δ, \mathrm{Inv}(σ, τ, δ))$3) (Kovač et al., 2024). P-tensor GNNs outperform conventional baselines on molecular regression and structural node/graph classification (Sun et al., 2023).
• Generalization and Data Efficiency: Decoupled higher-order models show sharper generalization in low-data and shallow-depth regimes (e.g., ablation on QM9 shows clear advantage in few-shot settings) (Kovač et al., 2024).

5. Group Actions, Equivariance Guarantees, and Theoretical Insights

Preservation of symmetry (equivariance) is fundamental for robust geometric learning:

• Group Actions: E(n)-equivariance is enforced by designing all message functions around invariant or equivariant geometric quantities. For CW-complexes, this is achieved by expressing message arguments exclusively as E(n)-invariant scalars. In Clifford and tensorial models, feature updates and the algebraic operations themselves (e.g., geometric product, grade projection, contractions) commute with group actions.
• Universality: Clifford-based and ACE/MACE constructions show that higher-order (body-order) message passing is universal: any continuous invariant (or equivariant) functional on point configurations can be approximated to arbitrary precision by an appropriate sum-aggregation GNN layer with sufficient order (Tran et al., 2024, Batatia, 2023).
• Theoretical Hierarchy: Expressivity hierarchy is strict: higher-order messages allow for separation of graph instances (e.g., particular cycles, motifs) that are indistinguishable to first-order GNNs. In the permutation-symmetric setting, structural message-passing achieves full graph identification at sufficient depth and width (Vignac et al., 2020).

6. Practical Methodological Recommendations

Implementation and architecture recommendations, summarized from state-of-the-art approaches:

• Order and Layer Count: Use layer counts in the 4–7 range for typical geometric and molecular benchmarks, restricting highest-order features and contractions to those needed by the downstream task.
• Feature Split: When employing a decoupled scheme, allocate 70–80% of channels to 0-cell (node) features, the remainder to higher-order cells (rings, polygons) (Kovač et al., 2024).
• Invariant Selection: For efficiency, restrict geometric invariants to those relevant for the physical or structural motif of interest (e.g., rings in aromatic systems, dihedral angles in polymers).
• Dropout and Regularization: Moderate dropout (0.1–0.2) in message/MLP layers is effective for generalization.
• Substructure Selection: For large-scale graphs, manually or heuristically select a subset of higher-order templates to limit the total number of higher-order cells.

7. Applications, Impact, and Outlook

Higher-order equivariant message passing has proven critical in advanced applications:

• Atomistic Simulation and Quantum Chemistry: Many-body message passing (up to f-orbital blocks) enables accurate modeling of electronic Hamiltonians and interatomic potentials, with sub-meV accuracy on matrix elements even for extended materials (Qian et al., 20 Aug 2025, Zaverkin et al., 2024).
• Dynamical Systems and Physical Simulation: Hierarchical and higher-order models exhibit improved fidelity on molecular dynamics, human motion capture, and PDE simulation on meshes (Kovač et al., 2024, Han et al., 2022, Park et al., 2023).
• Combinatorial and Topological Learning: P-tensor and structural message-passing frameworks generalize across discrete structures, hypergraphs, and matrix recovery tasks, with universality for equivariant learning over structured data (Sun et al., 2023, Telyatnikov et al., 2023, Vignac et al., 2020).

The convergence of algebraic, topological, and geometric insights in higher-order equivariant message passing yields architectures of unprecedented expressivity—connecting advanced mathematical invariants with scalable deep learning, and setting a foundation for further developments in geometric deep learning, multi-body physics, and beyond.