Thermodynamically Optimal Regularization under Information-Geometric Constraints

Abstract: Modern machine learning relies on a collection of empirically successful but theoretically heterogeneous regularization techniques, such as weight decay, dropout, and exponential moving averages. At the same time, the rapidly increasing energetic cost of training large models raises the question of whether learning algorithms approach any fundamental efficiency bound. In this work, we propose a unifying theoretical framework connecting thermodynamic optimality, information geometry, and regularization. Under three explicit assumptions -- (A1) that optimality requires an intrinsic, parametrization-invariant measure of information, (A2) that belief states are modeled by maximum-entropy distributions under known constraints, and (A3) that optimal processes are quasi-static -- we prove a conditional optimality theorem. Specifically, the Fisher--Rao metric is the unique admissible geometry on belief space, and thermodynamically optimal regularization corresponds to minimizing squared Fisher--Rao distance to a reference state. We derive the induced geometries for Gaussian and circular belief models, yielding hyperbolic and von Mises manifolds, respectively, and show that classical regularization schemes are structurally incapable of guaranteeing thermodynamic optimality. We introduce a notion of thermodynamic efficiency of learning and propose experimentally testable predictions. This work provides a principled geometric and thermodynamic foundation for regularization in machine learning.

• The paper develops a conditional theory linking thermodynamics, information geometry, and machine learning regularization, characterizing optimal regularization by Fisher–Rao distance.
• The framework is based on three explicit assumptions: an intrinsic information measure, maximum-entropy belief states, and quasi-static thermodynamic processes.
• Standard Euclidean regularization, such as weight decay, is found to be structurally incapable of satisfying the optimality criterion defined by the Fisher–Rao metric.

Overview

"Thermodynamically Optimal Regularization under Information-Geometric Constraints" (2601.17330) by Laurent Caraffa develops a conditional theory linking three traditionally separate threads: the Landauer bound on irreversible computation, Čencov's uniqueness theorem for the Fisher–Rao metric, and the practice of regularization in machine learning. The central claim is that, given three explicit assumptions, thermodynamically optimal regularization is uniquely characterized by minimizing squared Fisher–Rao distance to a reference belief state — and that standard Euclidean regularization schemes such as weight decay are structurally incapable of satisfying this optimality criterion. The paper's strength lies in making its assumptions explicit and its conclusions conditional; it does not claim that real training runs approach these bounds.

Assumptions and scope

The framework rests on three assumptions, which the author explicitly identifies as modeling choices rather than theorems:

• A1 (Intrinsic information measure): fundamental quantities such as minimal energy cost must be parametrization-invariant.
• A2 (Maximum-entropy belief states): a learner's belief states are maximum-entropy distributions subject to known constraints.
• A3 (Quasi-static processes): thermodynamic optimality corresponds to infinitesimally slow processes, defining a dissipation lower bound even though actual learning deviates from this regime.

The conditional character of all results follows directly from these premises. In particular, A3 concedes from the outset that the derived bounds apply to an idealized limit; the paper acknowledges that quantifying deviation in non-equilibrium regimes remains open.

Main theorem

The central result states that under A1–A3:

1. The metric on belief space must be Fisher–Rao, by Čencov's theorem (the unique Riemannian metric, up to scale, invariant under sufficient statistics and Markov morphisms).
2. The optimal regularization functional is $\mathcal{L}_{\mathrm{reg}} = d_F^2(q, q^*)$, since quasi-static paths minimize total erased information $\int_\gamma ds_F$ via geodesics.
3. The minimal dissipated energy is $$E_{\min} = k_B T \ln 2 \cdot D_{\mathrm{KL}}(q \| q^*)$$, combining the Landauer bound with the local expansion $$D_{\mathrm{KL}}(p \| p + dp) = \tfrac{1}{2} d_F^2(p, p+dp) + O(\|dp\|^3)$$.

The proof is short and modular: each step traces back to one assumption plus a classical result. This transparency is a virtue — a reader who rejects any single assumption knows exactly which conclusion fails.

Induced geometries

Two worked examples instantiate the framework. For univariate Gaussian beliefs $\mathcal{N}(\mu, \tau^{-1})$, the Fisher–Rao metric $ds^2 = \tau\, d\mu^2 + \tfrac{1}{2\tau^2} d\tau^2$ has constant negative curvature and is isometric to the hyperbolic half-plane $\mathbb{H}^2$. For circular beliefs, maximum entropy under mean-direction and concentration constraints yields the von Mises family, whose Fisher–Rao geometry captures temporal coherence and phase synchronization. These derivations give concrete manifolds on which the abstract optimality principle can be evaluated.

Structural suboptimality of Euclidean regularization

The paper's sharpest claim concerns weight decay and ridge-style penalties. Because the Euclidean metric is not reparametrization-invariant, it violates A1 outright, so Euclidean regularization cannot guarantee thermodynamic optimality. More quantitatively, for Gaussian beliefs with fixed variance $\sigma^2$, the ratio of squared Euclidean to squared Fisher–Rao distance equals $\sigma^2$ — a quantity that can be arbitrarily large or small depending on uncertainty. The implication is that Euclidean regularization can be arbitrarily suboptimal in high-uncertainty regimes, where the geometry of belief space diverges most strongly from flat parameter space. This is a strong structural claim, but note it is a statement about the idealized bound, not about generalization performance per se.

Thermodynamic efficiency and diagnostics

The paper defines thermodynamic efficiency as $$\eta = E_{\text{Landauer}} / E_{\text{actual}} \in (0, 1]$$, with $\int_\gamma ds_F$0 marking thermodynamic optimality, and decomposes inefficiency into hardware, algorithmic, and dissipative factors. The framework addresses only the third factor. A diagnostic quantity, the crystallization index $\int_\gamma ds_F$1 (spatial precision times temporal coherence), is proposed to characterize exploration, criticality, and over-constrained learning regimes.

Testable predictions

Three predictions are offered: (1) Fisher–Rao regularization achieves efficiency at least that of Euclidean regularization, with equality only in degenerate cases; (2) the performance gap grows with belief-space curvature, proxied by variability in precision $\int_\gamma ds_F$2; (3) the crystallization index predicts collapse and overfitting in self-supervised learning. Proposed protocols compare identical models trained with Euclidean versus Fisher–Rao penalties, measuring information divergence, energy proxies, and generalization. Notably, no experimental results are reported in this paper — the empirical program remains entirely prospective.

Limitations and open questions

The paper is candid about its boundaries. Real learning is far from quasi-static, and non-equilibrium corrections to the Landauer-based bound are unquantified here. The maximum-entropy model of belief states may poorly approximate representations in deep networks, and extension to more expressive families is left open. Additionally, computing exact Fisher–Rao distances or geodesics at scale is not addressed; tractable approximations for large models remain future work. Whether the predicted efficiency gaps are measurable against hardware-level energy overheads is also unresolved, since the framework bounds only the dissipative factor in the efficiency decomposition.

Conclusion

This paper offers a clean conditional result: if learning optimality requires intrinsic information measures, maximum-entropy beliefs, and quasi-static processes, then Fisher–Rao regularization is uniquely optimal and Euclidean penalties are structurally excluded. Its value lies less in immediate practical prescriptions than in converting regularization from a set of heuristics into a constrained optimization problem with a physical lower bound. The decisive test — whether the proposed efficiency and curvature-dependent performance gaps materialize empirically — remains to be run.