Hamiltonian Generative Networks

• Hamiltonian Generative Networks are deep models that infer an abstract phase space and neural Hamiltonian directly from high-dimensional observations.
• They integrate an inference network, Hamiltonian network, and decoder within a temporal VAE framework to enable reversible rollouts and energy conservation.
• Recent advancements focus on improving latent Hamiltonian fidelity, rollout stability, and adapting the model for non-conservative dynamics with port-Hamiltonian modifications.

Hamiltonian Generative Networks (HGNs) are deep generative models that learn an abstract phase space and a neural Hamiltonian directly from high-dimensional observations such as images, thereby coupling representation learning with continuous-time latent dynamics. Introduced as “the first approach capable of consistently learning Hamiltonian dynamics from high-dimensional observations (such as images) without restrictive domain assumptions,” HGN combines an inference network, a Hamiltonian network, and a decoder in a temporal variational framework, and supports sampling, forward and backward rollouts, and changes in rollout speed through the learned latent dynamics (Toth et al., 2019). Subsequent work has centered on whether the learned latent variables are genuinely Hamiltonian, how rollout stability and interpretability can be improved, and how continuous-time Hamiltonian video models behave when evaluated outside their training timestep regime (Higgins et al., 2021, Laird et al., 8 Jul 2026).

1. Origins and formal motivation

HGN is grounded in the Hamiltonian formalism, which models continuous-time evolution in phase space using canonical coordinates $q$ and $p$. In the original formulation, the latent state is $s=(q,p)\in\mathbb{R}^{2n}$, and the dynamics are defined by Hamilton’s equations,

$$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$

This formalism brings time reversibility, smooth interpolation in time, and conserved quantities into a deep generative model (Toth et al., 2019).

The motivating premise in HGN is that high-dimensional observations lie on a low-dimensional manifold and that observed trajectories correspond to flow along that manifold under Hamiltonian dynamics. The paper describes this as the “Hamiltonian manifold hypothesis.” In contrast to earlier Hamiltonian Neural Networks, which required access to low-dimensional state variables and their derivatives, HGN aims to infer both the abstract phase space and the Hamiltonian from pixels, with no need for ground-truth state or its derivatives (Toth et al., 2019).

This design places HGN at the intersection of physics-inspired representation learning, video prediction, and generative sequence modeling. A central claim of the original work is not that the latent coordinates must coincide with a known physical coordinate system, but that the learned phase space can be abstract while still supporting structured, reversible, and energy-conserving latent evolution. This suggests that Hamiltonian structure functions as an inductive bias for dynamical consistency rather than solely as an explicit system-identification mechanism.

2. Core architecture and objective

The original HGN consists of three main components: an inference network, a Hamiltonian network, and a decoder (Toth et al., 2019). The inference network takes an image sequence $(x_0,\ldots,x_T)$ and infers a posterior over the initial abstract state,

$q_\phi(s_0 \mid x_0,\ldots,x_T),$

typically parameterized as a diagonal Gaussian. The latent initial state is $s_0=(q_0,p_0)$.

The Hamiltonian network parameterizes a scalar Hamiltonian $\mathcal{H}_\gamma(q,p)$. Given the current abstract state, the model computes latent updates through Hamilton’s equations and produces a latent rollout using a symplectic leapfrog integrator. The decoder maps only the abstract position $q_t$ back to an observation $\hat{x}_t=d_\theta(q_t)$. The stated reason is that momentum does not directly influence instantaneous appearance in many physical systems (Toth et al., 2019).

Training uses a temporal variational autoencoder objective,

$p$0

with a unit Gaussian prior over $p$1. In the more explicit notation given in the same source,

$p$2

The leapfrog scheme is used for stability, volume preservation, and accurate energy conservation (Toth et al., 2019).

Operationally, HGN is trained end-to-end: the encoder infers an initial latent phase-space state from observed frames, the Hamiltonian network induces a latent trajectory, and the decoder reconstructs the corresponding image sequence. This is a generative model of trajectories rather than merely a one-step predictor.

3. Rollouts, reversibility, and density modeling

A defining property of HGN is that the latent dynamics are reversible. Because Hamiltonian evolution is invertible when integrated with an appropriate scheme, the model can perform rollouts both forward and backward in time. In practice, the original paper states that backward rollout is achieved by negating the integration time-step $p$3 (Toth et al., 2019).

The same continuous-time structure enables smooth interpolation and speed control. Once trained, HGN can “speed up or slow down the learned dynamics” by changing the integration step size, and it can generate plausible trajectories from latent samples. The paper reports faithful reversible rollouts on unseen test trajectories and emphasizes that symplectic integration preserves the learned Hamiltonian over long rollouts (Toth et al., 2019).

The original work also introduced Neural Hamiltonian Flow (NHF), obtained by a simple architectural modification that turns HGN into a normalizing flow model. The key observation is that Hamiltonian flows are invertible and volume-preserving. In NHF, the flow is implemented through learned Hamiltonian updates,

$p$4

and the determinant of the Jacobian is $p$5 to $p$6, which removes the need for explicit Jacobian determinant evaluation in density computation. The resulting likelihood expression is written as

$p$7

For density modeling over observed variables with latent momentum, the paper gives the variational lower bound

$p$8

Within the HGN literature, NHF represents a distinct use of Hamiltonian structure: not sequence reconstruction from images, but expressive density estimation with reversible, volume-preserving flows (Toth et al., 2019).

4. Evaluation methodology and HGN++

A major issue identified after the original HGN is that image reconstruction quality does not reliably measure whether the latent dynamics are Hamiltonian. The paper “SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision” argues that image-space metrics such as MSE correlate poorly with physical correctness, because a model can reconstruct frames well while learning incorrect or trivial latent dynamics (Higgins et al., 2021).

To address this, the paper introduces three related measures: $p$9 for variance explained, $s=(q,p)\in\mathbb{R}^{2n}$0 for symplecticity error, and SyMetric as a binary indicator of Hamiltonian fidelity. The symplecticity condition is based on the Jacobian $s=(q,p)\in\mathbb{R}^{2n}$1 of a learned map from latent state to ground-truth phase space, with the canonical symplectic form

$s=(q,p)\in\mathbb{R}^{2n}$2

and the condition

$s=(q,p)\in\mathbb{R}^{2n}$3

The paper defines

$s=(q,p)\in\mathbb{R}^{2n}$4

where $s=(q,p)\in\mathbb{R}^{2n}$5, and then combines this with $s=(q,p)\in\mathbb{R}^{2n}$6 through

$s=(q,p)\in\mathbb{R}^{2n}$7

with empirically chosen thresholds $s=(q,p)\in\mathbb{R}^{2n}$8 and $s=(q,p)\in\mathbb{R}^{2n}$9 (Higgins et al., 2021).

Using these measures, the authors performed an architecture and hyperparameter search leading to HGN++. The reported modifications include a 1D vector phase space instead of a 3D tensor latent, direct inference of latent positions and momenta, encoder and decoder with leaky ReLU activations and $$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$0 kernels, a Hamiltonian network using Swish activations and an MLP architecture, a shift from GECO-based training to a $$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$1-VAE objective, longer rollout horizons of 60 steps, prediction-based training instead of reconstruction, training for both forward and backward rollouts, and leapfrog integration with tuned step size and layers for stability (Higgins et al., 2021).

Empirically, HGN++ is reported to be “stable for significantly longer rollouts on a diverse range of 13 datasets,” and on a subset of those datasets it produces “rollouts of essentially infinite length both forward and backwards in time with no degradation in quality.” The same work states that HGN++ can discover an interpretable phase space with physically meaningful latents on some datasets and achieves near-perfect recovery of Hamiltonian dynamics on several benchmark tasks (Higgins et al., 2021). A common misconception in this area is therefore that good reconstruction alone demonstrates successful Hamiltonian learning; the evaluation literature explicitly rejects that equivalence.

5. Temporal generalization beyond conservative dynamics

Standard HGN is inherently conservative, which is advantageous for reversible physical systems but restrictive in the presence of friction, external forcing, or dissipation. A later study on temporal generalization in Hamiltonian video dynamics models observes that HGNs learn a continuous-time energy function over latent phase space and are, in principle, independent of observation frame rate, allowing evaluation at arbitrary step sizes. However, the same study reports that “in practice, their temporal generalization breaks down in non-conservative settings” (Laird et al., 8 Jul 2026).

To model externally forced and dissipative environments, that work adopts a port-Hamiltonian generalization,

$$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$2

where $$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$3 is a learned damping coefficient and $$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$4 is a learned mapping from action to phase space. The Hamiltonian is written as

$$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$5

with $$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$6 and $$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$7 learned via neural networks (Laird et al., 8 Jul 2026).

The analysis identifies two failure modes when evaluation uses timesteps outside the training regime. The first is latent magnitude growth driven by an unconstrained action-force map, producing amplitude blowup and visible artifacts. The second is global truncation error accumulation, causing phase drift even when the rollout remains bounded. The paper recommends two targeted fixes: spectral normalization of the action-force map $$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$8, capping its norm at $$\frac{dq}{dt} = \frac{\partial \mathcal{H}}{\partial p}, \qquad \frac{dp}{dt} = -\frac{\partial \mathcal{H}}{\partial q}.$$9, and inference-time integration substepping, in which each coarse evaluation step is subdivided into $(x_0,\ldots,x_T)$0 leapfrog substeps of size $(x_0,\ldots,x_T)$1 while preserving the total action impulse (Laird et al., 8 Jul 2026).

This line of work clarifies an important boundary condition for HGN-style models. Continuous-time Hamiltonian structure does enable interpolation and extrapolation in timestep, but it does not, by itself, guarantee robust temporal generalization in non-conservative environments. The failure is not presented as a refutation of HGN; rather, it is a statement that conservative Hamiltonian inductive bias must be augmented when the target dynamics are externally forced or dissipative.

6. Related Hamiltonian generative models and terminological boundaries

HGN has influenced a broader family of Hamiltonian generative models, but these models address different problems and should not be conflated. In “Hamiltonian GAN,” the temporal component of a GAN-based video generator is replaced with a Hamiltonian Neural Network motion model, and a learned configuration space map $(x_0,\ldots,x_T)$2 is introduced so that the configuration space need not be specified a priori. The model is trained with a cyclic-coordinate loss,

$(x_0,\ldots,x_T)$3

which is intended to encourage a minimal representation of the configuration space and improve interpretability (Allen-Blanchette, 2023). This differs from HGN’s temporal VAE formulation even though both use Hamiltonian latent dynamics.

In “Continuous Methods: Hamiltonian Domain Translation,” Hamiltonian structure is used to reformulate Cycle-GAN-style unsupervised domain translation as a continuous-time invertible flow in a shared latent space. The method learns an encoder and decoder for both domains and a Hamiltonian latent flow

$(x_0,\ldots,x_T)$4

with reverse mapping obtained by integrating backward in time. The paper emphasizes invertibility by construction, continuity, and the removal of explicit cycle-consistency loss (Menier et al., 2022). This is closely related in spirit to HGN’s use of Hamiltonian latent evolution, but the task is domain translation rather than sequential world modeling.

A different contrast is provided by Symplectic Generative Networks. That work defines a latent-to-data mapping as the solution $(x_0,\ldots,x_T)$5 of Hamilton’s equations over a symplectic latent space and proves invertibility and volume preservation, with exact likelihood evaluation and no Jacobian determinant term. It explicitly distinguishes itself from HGNs by stating that HGNs focus on learning time-evolution of physical systems and “do not use a symplectic flow as a latent-to-data generative mapping” (Aich et al., 28 May 2025). The distinction is structural: HGN is centered on latent trajectory generation from observations, whereas SGN is formulated as an invertible generative model.

Finally, the acronym “HGN” is not unique to Hamiltonian Generative Networks. In “Rapid training of Hamiltonian graph networks without gradient descent,” HGN denotes Hamiltonian Graph Networks rather than Hamiltonian Generative Networks, and the focus is permutation-invariant modeling of N-body mass-spring systems with random feature-based parameter construction and least-squares fitting (Rahma et al., 6 Jun 2025). This terminological overlap has created ambiguity in parts of the literature. A plausible implication is that references to “HGN” require contextual disambiguation, especially in recent work spanning graph networks, video models, and invertible generative systems.