Energy-Preserving Variational Autoencoder

• Energy-Preserving VAE is a model family that integrates energetic constraints into the variational autoencoder framework using geometric, metabolic, or architectural approaches.
• Quasi-symplectic formulations employ Hamiltonian dynamics to achieve low-variance pathwise gradients with constant Jacobian corrections for stable latent inference.
• Poisson and Hopfield variants leverage metabolic cost scaling and shared energy landscapes to balance reconstruction fidelity, sparsity, and hardware-friendly training.

Searching arXiv for the cited papers and closely related work to ground the article. Computation in variational autoencoders can be called “energy-preserving” in more than one technical sense. In one line of work, the term denotes variational inference schemes whose latent dynamics inherit the near energy conservation and volume-preserving structure of symplectic or quasi-symplectic integrators, as in the Quasi-symplectic Langevin Variational Autoencoder (Wang et al., 2020). In another, it denotes a VAE whose encoder and decoder are two uses of the same Hopfield energy landscape, so that inference and generation share a single Lyapunov-consistent energy-based architecture trained by equilibrium propagation (Meersch et al., 2023). A third, distinct formulation ties information processing to an explicit metabolic proxy by using Poisson latent variables, so that the Kullback–Leibler term becomes linearly scaled by prior firing rates and thereby induces an emergent energy cost; this yields an energy-aware Poisson variational autoencoder that can be interpreted as an “Energy-Preserving VAE” in the sense of preserving an accuracy–energy budget (Vafaii et al., 13 Feb 2026). Across these formulations, the common theme is not a single universal definition of energy conservation, but the insertion of energetically meaningful structure into the VAE objective, latent geometry, or training dynamics.

1. Meanings of “energy-preserving” in VAE research

The phrase has at least three non-equivalent uses in the literature represented here. In the quasi-symplectic Langevin formulation, “energy-preserving” refers to the geometric behavior of deterministic Hamiltonian substeps inside the variational flow: these substeps are volume-preserving and nearly conserve the Hamiltonian $H(z,p)$, with exact symplectic structure recovered when the damping parameter vanishes (Wang et al., 2020). In this setting, preservation concerns phase-space geometry and numerical stability.

In the Hopfield formulation, “energy” refers instead to the Lyapunov energy of a Continuous Hopfield Network,

$$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$

with $W=W^\top$ and zero diagonal. The encoder and decoder are defined by the same or same-form energy function, and free-phase inference follows gradient descent on that scalar energy until equilibrium. Here, “preserving” does not mean conserving energy; it denotes architectural reuse of a single energy-based structure across encoding and decoding, together with Lyapunov-consistent dynamics (Meersch et al., 2023).

In the Poisson variational formulation, the relevant notion is neither Hamiltonian conservation nor Hopfield Lyapunov descent, but the appearance of an emergent metabolic cost inside the $\beta$-weighted ELBO. For factorized Poisson latents with residual parameterization, the KL term becomes

$$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$

so the abstract coding-rate penalty is directly scaled by prior firing rates. This couples information rate to firing rate and supports an energy-aware interpretation of inference and representation (Vafaii et al., 13 Feb 2026).

A plausible implication is that “Energy-Preserving VAE” is best treated as a family resemblance term rather than as a single standardized model class. The three formulations share the aim of making energetic structure operational in VAEs, but they do so through different mathematical objects: Hamiltonians, Hopfield energies, and Poisson firing-rate scales.

2. Geometric energy preservation through quasi-symplectic Langevin inference

The Quasi-symplectic Langevin Variational Autoencoder augments the latent state with momentum and defines

$$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$

The variational posterior is then enriched by applying $L$ differentiable kinetic Langevin steps to an initial latent sample and auxiliary momentum, producing a tighter and lower-variance ELBO than a simple amortized posterior can provide (Wang et al., 2020).

The central technical device is a quasi-symplectic splitting scheme. Using the paper’s notation $(\Phi,K)$ for $(z,p)$, step size $t$, and damping $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$0, the update is

$$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$1

Rewritten in $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$2, this yields exponential damping around a kick–drift–kick structure, akin to a BAOAB-like splitting (Wang et al., 2020).

Two theoretical properties are central. First, the quasi-symplectic method degenerates to symplectic when $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$3. In that limit the map is volume-preserving, with Jacobian determinant $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$4, and the deterministic substeps exhibit near energy conservation. Second, the Jacobian of each quasi-symplectic Langevin step is constant: $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$5 Therefore each step contributes a constant log-determinant term $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$6, so the change-of-variables correction in the ELBO is trivial and does not require Hessians (Wang et al., 2020).

This constant-Jacobian property distinguishes the method from naïve overdamped Langevin normalizing flows, where the log-determinant involves the Hessian of $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$7. The quasi-symplectic construction retains gradient-informed inference while avoiding expensive second-order derivatives. In the paper’s deterministic implementation, $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$8, so the practical method is a differentiable flow with low-variance pathwise gradients (Wang et al., 2020).

The resulting ELBO estimator uses the transformed position–momentum pair and adds a simple $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$9 correction. Rao–Blackwellization is then applied to further reduce variance. This suggests that, in this formulation, “energy preservation” is inseparable from tractable geometry: the same structure that stabilizes Hamiltonian motion also yields manageable density corrections during training.

3. Metabolic energy preservation through Poisson latent variables

The Poisson formulation begins from the $W=W^\top$0-weighted ELBO

$W=W^\top$1

with factorized Poisson latent posterior and prior,

$W=W^\top$2

and residual parameterization

$W=W^\top$3

The closed-form KL is

$W=W^\top$4

which under the residual parameterization becomes

$W=W^\top$5

Because the prior rates $W=W^\top$6 are learnable, minimizing free energy pushes baseline rates downward unless reconstruction demands otherwise. The paper identifies the additive term embedded in $W=W^\top$7 as an emergent metabolic cost, and interprets the Poisson KL as a coding rate directly multiplied by a biophysical rate scale (Vafaii et al., 13 Feb 2026).

This coupling is sharpened by the paper’s bits-per-spike definition,

$W=W^\top$8

and by the energy budget proxy

$W=W^\top$9

Here the average posterior firing rate serves as the energetic proxy, and $\beta$0 functions as a Lagrange multiplier mediating the trade-off between reconstruction fidelity and rate-dependent cost (Vafaii et al., 13 Feb 2026).

The empirical behavior is specific to the Poisson family. On whitened $\beta$1 van Hateren grayscale patches, with linear encoder/decoder, Gaussian reconstruction, learnable priors, and Poisson relaxation for training, the paper sweeps

$\beta$2

and

$\beta$3

Across all $\beta$4, average activity, measured by MC as the mean of sampled $\beta$5 over the validation set, decreases monotonically with $\beta$6 by nearly two orders of magnitude, while sparsity, measured by PZ as the proportion of zeros in $\beta$7, increases monotonically with $\beta$8. For $\beta$9, PZ rises from approximately $$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$0 at low $$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$1 to approximately $$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$2 at high $$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$3 (Vafaii et al., 13 Feb 2026).

The comparison model, Grelu-VAE, uses Gaussian latent posterior and prior with ReLU applied to samples before decoding. Its Gaussian KL,

$$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$4

is translation-invariant with respect to the prior mean and independent of absolute activity scale. Empirically, Grelu-VAE activity remains flat across $$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$5, and sparsity plateaus near $$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$6, with increasing $$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$7 mainly harming reconstruction (Vafaii et al., 13 Feb 2026). The paper’s conclusion is therefore narrow but strong: non-negativity alone does not create a metabolic term; Poisson statistics do.

4. Energy-based architectural preservation in Hopfield VAEs

The Hopfield Variational Autoencoder uses Continuous Hopfield Networks as both encoder and decoder, trained with equilibrium propagation rather than backpropagation. The state evolves according to

$$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$8

and, in discrete time,

$$\mathrm{KL}(q\Vert p)=\sum_{j=1}^K \lambda_p^{(j)}\, f\!\big(\delta\lambda^{(j)}\big),\qquad f(y)=y\log y-y+1,$$9

In the free phase, this implies

$$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$0

so the Hopfield energy is non-increasing during inference (Meersch et al., 2023).

The VAE interpretation is imposed by partitioning the state differently during encoding and decoding. For the encoder, one clamps the input $$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$1, relaxes to equilibrium, and reads the latent mean $$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$2 and log-variance $$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$3. The latent sample is obtained by the standard reparameterization

$$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$4

For the decoder, one clamps $$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$5, relaxes to an equilibrium output $$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$6, and interprets the reconstruction term as

$$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$7

under Gaussian output-noise assumptions (Meersch et al., 2023).

The ELBO retains the conventional $$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$8-VAE form,

$$U(z)\equiv -\log p_\theta(x,z),\qquad K(p)\equiv \tfrac{1}{2}p^\top M^{-1}p,\qquad H(z,p)\equiv U(z)+K(p).$$9

with standard normal prior and Gaussian posterior. The KL is

$L$0

Equilibrium propagation then supplies local learning rules by comparing weakly nudged equilibria at $L$1 and $L$2 (Meersch et al., 2023).

For the decoder, the weakly clamped phase adds a nudging term proportional to $L$3, and the EP weight update becomes

$L$4

For the encoder, the reconstruction gradient is first pulled back to $L$5 via the EP input-gradient identity, then converted to output nudges on $L$6 and $L$7,

$L$8

$L$9

and the same EP difference-of-equilibria rule is applied (Meersch et al., 2023).

The most distinctive variant is the tied dense Hopfield VAE, or TDH-VAE, in which a single symmetric weight matrix $(\Phi,K)$0 is reused for both encoder and decoder. To match dimensionalities, this tied model fixes $(\Phi,K)$1, eliminating the $(\Phi,K)$2 block. The paper explicitly argues that this design could effectively halve the required chip size for VAE implementations on dedicated analog hardware (Meersch et al., 2023).

The paper evaluates four models on MNIST using Fréchet Inception Distance: F-VAE at $(\Phi,K)$3, H-VAE at $(\Phi,K)$4, DH-VAE at $(\Phi,K)$5, and TDH-VAE at $(\Phi,K)$6. These results establish that equilibrium propagation can train a Hopfield VAE and that a single shared symmetric network is feasible, although the untied dense Hopfield model performs better than the tied one in the reported experiments (Meersch et al., 2023).

5. Mechanisms, objectives, and contrasts

The three formulations differ most sharply in what the ELBO’s regularization term means and how energetic structure enters training.

Formulation	Energy object	Main consequence
QS-LVAE	Hamiltonian $(\Phi,K)$7	Near energy conservation and tractable constant Jacobian
Poisson VAE	Prior-rate-scaled Poisson KL	Coding rate coupled to firing-rate cost
Hopfield VAE	Lyapunov energy $(\Phi,K)$8	Shared encoder–decoder energy landscape and local EP learning

In QS-LVAE, the objective remains the standard ELBO, but the variational family is transformed by a quasi-symplectic kinetic flow. The main gain is posterior expressiveness and low-variance pathwise optimization with a constant Jacobian correction $(\Phi,K)$9 (Wang et al., 2020). In this sense, energy preservation concerns inference geometry rather than explicit energy budgeting.

In the Poisson VAE, the objective itself becomes energy-aware because the KL term is proportional to prior firing rates. This has two sparse-coding consequences. First, near $(z,p)$0,

$(z,p)$1

so higher baseline rates amplify the curvature penalizing positive deviations. Second, with Gaussian likelihood and linear decoder $(z,p)$2, the reconstruction term decomposes as

$(z,p)$3

For Poisson latents, $(z,p)$4, so the variance term adds a linear penalty in $(z,p)$5, weighted by dictionary norms. The paper therefore interprets the model as a spiking form of sparse coding (Vafaii et al., 13 Feb 2026).

In the Hopfield VAE, by contrast, the regularization is still the standard Gaussian KL; the energy-based novelty lies in the architecture and learning rule. Encoding and decoding are not separate directed maps but two equilibrium computations in the same or same-form symmetric network, and the gradients are estimated by weakly clamped differences rather than by backpropagation (Meersch et al., 2023).

A common misconception is that any non-negative or rectified latent representation makes a VAE “energy-aware.” The Poisson comparison to Grelu-VAE directly contradicts this: ReLU rectification alone does not produce a prior-rate linear term in the KL, and does not make $(z,p)$6 a meaningful control knob for metabolic cost (Vafaii et al., 13 Feb 2026). Another misconception is that “energy-preserving” always means exact conservation. That is true only in the Hamiltonian limit of the quasi-symplectic construction; Hopfield free-phase dynamics explicitly decrease energy, and Poisson models optimize an energy-aware objective rather than conserve a dynamical invariant (Wang et al., 2020, Meersch et al., 2023, Vafaii et al., 13 Feb 2026).

6. Implementations, empirical regimes, and limitations

The three lines of work occupy different implementation regimes. QS-LVAE is a flow-augmented VAE with auxiliary momentum, a quasi-symplectic kinetic update, and pathwise ELBO optimization. On MNIST with Bernoulli decoder, $(z,p)$7-layer CNN encoder and decoder, latent dimension $(z,p)$8, friction $(z,p)$9, step $t$0, and Adamax learning rate $t$1, the method matches HVAE closely. For $t$2, the reported NLL values are $t$3 for LVAE versus $t$4 for HVAE; ELBO values are $t$5 versus $t$6; FID values are $t$7 versus $t$8; and IS values are $t$9 versus $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$00 (Wang et al., 2020). On a $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$01D medical CT cochlea dataset, QS-LVAE improves over standard VAE in Avg. ELBO, Avg. NLL, and early stopping epochs (Wang et al., 2020).

The Poisson VAE is implemented with Poisson latents under residual parameterization, linear encoder and decoder, Gaussian reconstruction loss with closed-form moments, and learnable Poisson prior rates. Training uses the Exponential Arrival Time Poisson relaxation with a cubic indicator approximation and temperature $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$02, Adamax optimizer, learning rate $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$03, batch size $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$04, $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$05 warmup epochs, cosine annealing, and gradient clipping with max norm $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$06 (Vafaii et al., 13 Feb 2026). The reported metrics are MC, PZ, $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$07, and the closed-form reconstruction loss.

The Hopfield VAE is implemented as a Continuous Hopfield Network with bounded nonlinearity $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$08, no biases, symmetric weights, zero diagonal, and equilibrium propagation training. Digital simulation is expensive because each sample typically requires an encoder free phase, two encoder nudged phases, a decoder free phase, and two decoder nudged phases, each involving repeated dense matrix–vector multiplications. The intended advantage is therefore strongest on analog hardware, where local updates and physical relaxation can reduce memory movement and exploit in-memory computation (Meersch et al., 2023).

The limitations also differ. The Poisson approach depends on a count-based latent geometry; the paper states that non-negativity and discreteness are essential to the metabolic coupling, while deep decoders would weaken the closed-form sparse-coding interpretation (Vafaii et al., 13 Feb 2026). The quasi-symplectic approach can suffer from discretization error when $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$09 is too large, and poor mass matrices can slow exploration; with $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$10, exploration relies on the encoder and deterministic flow (Wang et al., 2020). The Hopfield approach is sensitive to relaxation length, nudging amplitude, and initialization, and its image quality remains modest relative to modern VAEs; the paper identifies larger state spaces, convolutional inductive biases, and better nonlinearities as important future directions (Meersch et al., 2023).

Taken together, these limitations clarify the scope of the field. “Energy-preserving” does not designate a settled benchmark architecture, but a research program spanning three directions: geometric stabilization of latent inference, explicit rate-dependent energy accounting, and energy-based symmetric hardware-friendly architectures.

7. Significance and research outlook

The three formulations collectively reposition the VAE as a model in which energetic structure can be inserted at different levels of abstraction. QS-LVAE shows that energy-preserving numerical integrators can be imported into variational inference to obtain richer posteriors, low-variance pathwise gradients, and tractable Jacobians without Hessians (Wang et al., 2020). The Hopfield VAE shows that a VAE can be trained without backpropagation by using local equilibrium-based updates in a symmetric network that serves as both encoder and decoder, with clear implications for analog hardware efficiency (Meersch et al., 2023). The Poisson VAE shows that under count-valued latent statistics, the coding-rate KL can be made proportional to firing-rate scale, yielding a direct trade-off between fidelity and a metabolic proxy and recovering a spiking form of sparse coding (Vafaii et al., 13 Feb 2026).

A plausible implication is that future work may continue to separate into these strands rather than converge immediately to a single canonical model. One strand concerns geometry-preserving latent flows and tighter ELBOs; another concerns biologically or neuromorphically grounded energy budgets; a third concerns training algorithms and architectures whose physical implementation is itself energy-efficient. The Poisson paper explicitly notes extensions to super-Poisson variability such as the negative binomial as promising, while the Hopfield paper emphasizes larger associative architectures and tied models with learnable $$E(s) = \frac{1}{2}\|s\|^2 - \frac{1}{2}\rho(s)^\top W \rho(s),$$11 (Vafaii et al., 13 Feb 2026, Meersch et al., 2023). The quasi-symplectic work points toward mass-matrix adaptation, Riemannian preconditioning, importance weighting, and hybridization with normalizing flows (Wang et al., 2020).

The broader significance of the topic lies in reframing variational autoencoding away from purely likelihood-centered optimization. Whether through conserved Hamiltonians, Lyapunov-consistent equilibria, or rate-scaled coding costs, these models treat energy not as an irrelevant implementation detail but as a structural variable of inference, learning, or deployment.