---
title: Node Embedding from Neural Hamiltonian Orbits
url: https://www.emergentmind.com/topics/node-embedding-from-neural-hamiltonian-orbits
type: topic
---

# Node Embedding from Neural Hamiltonian Orbits

Node embedding from neural Hamiltonian orbits refers to a family of graph representation learning techniques leveraging principles from Hamiltonian dynamics. These approaches address the limitations of classical Graph Neural Networks (GNNs) in handling diverse graph geometries and the over-smoothing phenomena in deep architectures by constructing node embeddings as trajectories (orbits) evolving under learnable Hamiltonian systems. The resulting embeddings are capable of automatically inferring latent geometry, preserving information under deep propagation, and providing robustness to perturbations [2303.01030][2309.04885][2305.18965].

## 1. Hamiltonian Formulation for Node Embedding

Each node $k$ in the graph is associated with “position” vector $q_k \in \mathbb{R}^r$ and “momentum” vector $p_k \in \mathbb{R}^r$, which are assembled as $q = (q_1, ..., q_{|V|}) \in \mathbb{R}^{r|V|}$ and $p = (p_1, ..., p_{|V|}) \in \mathbb{R}^{r|V|}$. A learnable Hamiltonian function $H(q, p; \theta)$, typically parameterized as a small neural network (e.g., a two-layer GCN or MLP), assigns an “energy” to each state in the phase space $T^*Q_G \cong \mathbb{R}^{2r|V|}$.

- In HDG [2303.01030], the Hamiltonian is realized as 
  $$
  H(q, p; \theta) = \left\| \mathrm{GCN}_{\theta_2}\left(\tanh\left(\mathrm{GCN}_{\theta_1}([q, p], A)\right)\right) \right\|_F,
  $$
  where $A$ is the adjacency matrix and $[q, p]$ denotes feature concatenation.

- In HamGNN [2305.18965], $H(q, p)$ can adopt several forms, including a metric-based quadratic form for manifold learning,
  $$
  H(q, p) = \frac{1}{2} p^T g^{-1}_{\mathrm{net}}(q) p,
  $$
  or a general MLP on $[q, p]$.

- In SAH-GNN [2309.04885], the total energy decomposes into kinetic and potential terms,
  $$
  H(q, p) = \frac{1}{2} p^T p + \mathrm{MLP}_\theta(q),
  $$
  with a key innovation being the use of a learnable symplectic structure.

No explicit regularizer is necessary as the Hamiltonian flow guarantees energy conservation by construction.

## 2. Continuous-time Hamiltonian Dynamics

Node features evolve according to canonical Hamiltonian ODEs:
$$
\dot{q}_k = \frac{\partial H}{\partial p_k}, \qquad \dot{p}_k = -\frac{\partial H}{\partial q_k},
$$
defining a symplectic orbit on $T^*Q_G$. The time evolution preserves $H$ exactly (the energy-conservation law), ensuring stable long-range propagation and well-separated node representations.

- In the most general framework (HamGNN), the system can recover any local geometry, with the exponential map (shortest-path geodesics) being a special case for quadratic $H$.
- In SAH-GNN, the symplectic structure itself is learned on a Riemannian manifold, generalizing standard Hamiltonian evolution and adapting phase space geometry to the task [2309.04885].

## 3. Discretization and GNN Integration

Discretization of the continuous flow is essential for implementation and message passing. The standard procedure is:

- Set integration time $T > 0$, step-size $\tau = T / N$, and select a symplectic integrator.
- Most approaches adopt symplectic (semi-implicit) Euler:
  $$
  \begin{aligned}
  q^{(n+1)} &= q^{(n)} + \tau \, \frac{\partial H}{\partial p}(q^{(n)}, p^{(n)}), \\
  p^{(n+1)} &= p^{(n)} - \tau \, \frac{\partial H}{\partial q}(q^{(n+1)}, p^{(n)}).
  \end{aligned}
  $$
- Higher-order symplectic schemes (e.g., Leapfrog/Störmer–Verlet) can be used for improved numerical stability [2309.04885].
- Each integration step is analogous to a GNN layer; stacking $N$ iterations yields an $N$-layer network.

Between integration steps, standard message passing is performed:
$$
q_{n+1} \leftarrow (\hat{A} + I) \, q_{n+1},
$$
where $\hat{A}$ is a normalized adjacency.

After the final time step, the embedding is $q(T)$, sometimes refined by an additional readout network.

## 4. End-to-End Training and Algorithms

All model parameters, including the Hamiltonian network, momentum network, and (when applicable) the symplectic matrix, are learned via backpropagation through the discretized ODE steps. Automatic differentiation computes required gradients $\partial H / \partial \theta$, $\partial H / \partial (q, p)$.

- **Objective for node classification:** cross-entropy on labeled nodes with softmax readout of $q_i(T)$.
- **Objective for link prediction:** cross-entropy or margin ranking loss on embedding pairs.
- **Symplectic matrix learning (SAH-GNN):** Riemannian gradient descent projects updates onto the tangent space of the symplectic Stiefel manifold, followed by Cayley retraction to guarantee manifold constraints.
- **Energy regularization (optional in SAH-GNN):** 
  $$
  \mathcal{L}_E = \frac{1}{N} \sum_i \left| H_\theta(z_i(t_1)) - H_\theta(z_i(t_0)) \right|^2,
  $$
  although exact integrators and learned symplectic structure yield near-perfect conservation.

Pseudocode for generic implementation appears in [2303.01030], [2305.18965], and [2309.04885].

## 5. Geometry Adaptation and Robustness

A principal advantage is the ability to learn and adapt to latent graph geometry. The Hamiltonian $H$ is agnostic to manifold type; it discerns local or mixed curvature (Euclidean, hyperbolic, pseudo-Riemannian) during training, without pre-prescribed structure [2303.01030][2305.18965].

The conservation of $H$ precludes energy "collapse," directly addressing over-smoothing: node features remain diverse even after deep propagation (empirically verified up to 32–64 layers, with GCN/HGCN collapsing by contrast [2303.01030], [2305.18965]).

Empirical studies under adversarial perturbations (SPEIT, TDGIA, Nettack) show only ≈5% performance drop for Hamiltonian models, outperforming attention-based and diffusion GNNs in resilience [2303.01030].

## 6. Experimental Results and Analysis

### Benchmarking

Key datasets: Cora, Citeseer, Pubmed (higher hyperbolicity), Disease, Airport (tree-like, low hyperbolicity), and various mixed-geometry graphs formed by union.

### Baselines

  - **Euclidean GNNs:** GCN, SAGE, SGC, GAT
  - **Hyperbolic/Mixed GNNs:** HGNN, HGCN, HGAT, LGCN, $\kappa$-GCN, Q-GCN, GIL
  - **Neural ODE/PDE:** GRAND, GraphCON

### Metrics

  - Node classification accuracy.
  - Link prediction ROC–AUC.

### Results

| Model           | Disease (%) | Airport (%) | Cora (%) | Mixed Geo (best) |
|-----------------|------------|-------------|----------|------------------|
| HDG / HamGNN    | 91.3–91.5  | 94.5–95.5   | 82+      | >5% over best baseline in 3/4 cases |
| Best baseline   | 90.8       | 91.5        | 82.8     | —                |

  - Hamiltonian-based models outperform all baselines on tree-like and mixed-geometry graphs, and on link-prediction tasks except one marginal case [2303.01030][2305.18965][2309.04885].
  - All variants of $H$ (metric-based, unconstrained, convex, relaxations) attain comparable results; the metric-based/geodesic variant is both simple and effective.
  - Ablations show $\geq$10% performance gap over plain ODEs without Hamiltonian structure; choice of ODE solver (Euler, RK4, Dopri5) is not critical.
  - Inference time is competitive: HDG is $\sim$5.8 ms/sample vs GCN 2.96 ms, HGCN 6.1 ms, GraphCON 4.3 ms.

### Stability

Energy curves remain nearly constant over long training, confirming theoretical guarantees. When stacking many layers, Hamiltonian models maintain accuracy, while GCN/HGCN degrade rapidly [2303.01030][2305.18965].

## 7. Extensions, Limitations, and Future Work

- **Symplectic learning:** SAH-GNN [2309.04885] extends with a learnable shape for the symplectic structure, optimized on the symplectic Stiefel manifold, enabling adaption to arbitrary graph data.
- **ODE integration:** Future directions include exact symplectic integrators for stricter energy conservation and generalization to full (non-diagonal) metrics or low-rank structures.
- **Applicability:** Current methods focus on node representations; further development is needed for graph-level classification, dynamic or heterophilic graphs, and theoretical analyses of generalization properties [2305.18965].
- **Implementation:** Open-source code is provided for HamGNN [2305.18965].

A plausible implication is that Hamiltonian-inspired node embedding frameworks offer a unified, geometrically flexible, and robust solution to representation learning on complex graphs, generalizing and improving upon traditional GNNs without significant increases in inference time or hyperparameter sensitivity.

---

**References**  
[2303.01030] Node Embedding from Hamiltonian Information Propagation in Graph Neural Networks  
[2309.04885] Symplectic Structure-Aware Hamiltonian (Graph) Embeddings  
[2305.18965] Node Embedding from Neural Hamiltonian Orbits in Graph Neural Networks

Source: https://www.emergentmind.com/topics/node-embedding-from-neural-hamiltonian-orbits