Dissipative Learning: A Framework for Viable Adaptive Systems

Abstract: We propose a perspective in which learning is an intrinsically dissipative process. Forgetting and regularization are not heuristic add-ons but structural requirements for adaptive systems. Drawing on information theory, thermodynamics, and information geometry, we introduce the BEDS (Bayesian Emergent Dissipative Structures) framework, modeling learning as the evolution of compressed belief states under dissipation constraints. A central contribution is the Conditional Optimality Theorem, showing that Fisher-Rao regularization measuring change via information divergence rather than Euclidean distance is the unique thermodynamically optimal regularization strategy, achieving minimal dissipation. Euclidean regularization is shown to be structurally suboptimal. The framework unifies existing methods (Ridge, SIGReg, EMA, SAC) as special cases of a single governing equation. Within this view, overfitting corresponds to over-crystallization, while catastrophic forgetting reflects insufficient dissipation control. The framework distinguishes BEDS-crystallizable problems, where beliefs converge to stable equilibria, from BEDS-maintainable problems, which require continual adaptation. It extends naturally to continual and multi-agent systems, where viability, stability under adaptation and finite resources replaces asymptotic optimality as the primary criterion. Overall, this work reframes learning as maintaining viable belief states under dissipation constraints, providing a principled lens on forgetting, regularization, and stability.

• The paper proposes the BEDS framework (Bayesian Emergent Dissipative Structures) for integrating regularization, forgetting, and stability as thermodynamic principles into adaptive systems.
• The framework derives the Energy-Precision Bound, which shows maintaining precision against a dissipation rate requires thermodynamic energy, under assumptions related to intrinsic measure, maximum entropy, and quasi-static processes.
• The work unifies various learning techniques, such as Ridge regression, EMA, and Transformer attention, as special cases or approximations of a fundamental equation.

Motivation and scope

The paper, authored by Laurent Caraffa (2601.17933), addresses a persistent gap in machine learning theory: regularization techniques such as weight decay, dropout, batch normalization, and exponential moving averages (EMA) work empirically but lack a unifying theoretical justification. The author proposes that these techniques are not optional add-ons but structural necessities for adaptive systems operating under finite resources. The central claim is that learning is an intrinsically dissipative process, and that forgetting, regularization, and stability should be understood as first-class thermodynamic principles rather than heuristics.

The work is explicitly positioned as theoretical rather than algorithmic. It introduces the BEDS framework (Bayesian Emergent Dissipative Structures), derives results conditional on three explicit assumptions, and states plainly what it does not claim: it does not assert that real learning algorithms operate in the quasi-static regime or follow Fisher–Rao geodesics, does not propose an implementable training procedure, and provides no empirical benchmarks or hardware-level energy modeling. The intended role of the framework is as a reference model and diagnostic tool.

Physical and informational foundations

The framework rests on two bodies of established physics. First, Prigogine's theory of dissipative structures identifies four conditions under which ordered patterns persist far from equilibrium: openness, distance from equilibrium, nonlinearity, and continuous entropy export. The paper maps each condition onto neural network training—data mini-batches provide flux, structured weights differ from random initialization, activations and attention supply nonlinearity, and regularization exports entropy.

Second, Landauer's principle supplies the quantitative link between information and energy: erasing one bit dissipates at least $k_B T \ln 2$ joules (approximately $2.87 \times 10^{-21}$ J at 300 K). From this the paper constructs its "regularization–forgetting–energy" chain: regularization erases information about training data; erasure carries an irreducible Landauer cost; therefore regularization has a thermodynamic price. A concrete illustration is given for a LeJEPA-style ViT-Large model (~300M parameters) with a ~20 GB memory budget, of which roughly 3.6 GB—the EMA teacher (1.2 GB, momentum $m = 0.996$, i.e., 0.4% forgetting per step) plus Adam optimizer states (2.4 GB, $\beta_1 = 0.9$, i.e., 10% decay per step)—is dedicated to implementing controlled forgetting. The author notes this is an engineering cost, not the thermodynamic minimum, which would be negligible by comparison.

Assumptions and the Conditional Optimality Theorem

The framework's rigor depends on three explicit assumptions:

• A1 (Intrinsic measure): optimality requires a parametrization-invariant information metric. By Čencov's theorem, this uniquely selects the Fisher–Rao metric.
• A2 (Maximum entropy): belief states are maximum-entropy distributions subject to known constraints, yielding Gaussians for spatial beliefs and von Mises distributions for temporal beliefs.
• A3 (Quasi-static processes): optimal processes approach the quasi-static limit, where geodesics minimize dissipation.

The central result is the Conditional Optimality Theorem: under A1–A3, thermodynamically optimal regularization is uniquely characterized by minimizing squared Fisher–Rao distance to a reference belief state, with geodesic trajectories. Two corollaries follow directly. First, the Energy-Precision Bound: maintaining precision against dissipation rate $\gamma$ requires $P_{\min} \geq \gamma k_B T / 2$, derived via rate-distortion arguments combined with Landauer's principle. Second, Euclidean regularization (e.g., Ridge) violates A1 and is structurally suboptimal; for Gaussian beliefs the ratio of squared Euclidean to Fisher–Rao distances equals $1/\tau = \sigma^2$, so Euclidean regularization can be arbitrarily suboptimal when precision varies across parameters.

The conditional character of these claims deserves emphasis: they hold if and only if A1–A3 are accepted. The proof of uniqueness is short and relies on standard results (Čencov's theorem, the local KL–Fisher correspondence); the substantive content lies in the assumptions rather than novel mathematics.

The BEDS state space and dynamics

Under A1–A3, belief states are parameterized by four coordinates $(\mu, \tau, \phi, \kappa)$ on the product manifold $H^2 \times M$: position $\mu$ and precision $2.87 \times 10^{-21}$0 (spatial, hyperbolic geometry), phase $2.87 \times 10^{-21}$1 and coherence $2.87 \times 10^{-21}$2 (temporal, von Mises manifold). In the absence of observations, dissipation drives $2.87 \times 10^{-21}$3 and $2.87 \times 10^{-21}$4 to decay exponentially. The paper defines a crystallization index $2.87 \times 10^{-21}$5 measuring overall "solidity": low values favor exploration, high values indicate rigidity.

This parameterization yields reinterpretations of familiar pathologies: overfitting becomes over-crystallization ($2.87 \times 10^{-21}$6 prematurely), mode collapse becomes premature crystallization ($2.87 \times 10^{-21}$7), representation collapse corresponds to falling toward the maximum-entropy center, and catastrophic forgetting reflects insufficient dissipation structure in the prior hierarchy.

Unification of existing methods

A key contribution is showing that disparate methods emerge as special cases or approximations of one fundamental equation:

$2.87 \times 10^{-21}$8

Specifically, Ridge regression approximates Fisher–Rao regularization only under constant precision; SIGReg directly regularizes toward $2.87 \times 10^{-21}$9, the maximum-entropy target; EMA controls temporal coherence via $m = 0.996$0; SAC operates in a low-$m = 0.996$1 exploration regime with temperature $m = 0.996$2. The paper further argues that Transformer attention implements Bayesian belief updates with effective coherence scaling as $m = 0.996$3, and that diffusion models instantiate the full BEDS cycle—forward diffusion as pure dissipation ($m = 0.996$4), reverse denoising as reconstruction along learned score functions. Gaussian splatting representations are identified as native BEDS encodings, since each primitive carries explicit $m = 0.996$5 coordinates; the paper cites independent work on Wasserstein–Fisher–Rao gradient flows for splat models as mathematical support for this identification.

Two extensions deserve note. A hierarchy theorem shows that with geometrically decreasing dissipation rates $m = 0.996$6 across levels, total maintenance energy converges: $m = 0.996$7, implying abstract (crystallized) knowledge is energetically cheap to maintain. And a six-class problem taxonomy follows from the product structure: each component ($m = 0.996$8, $m = 0.996$9) may be either crystallizable or maintainable, yielding classes from C-full (stationary supervised learning) through M-full (continual RL, world models). The taxonomy is proven under A1–A3 and is intended to guide algorithm selection a priori.

Predictions and epistemic status

The paper distinguishes carefully between proven results (conditional on A1–A3), conjectures, and speculation. Proven items include the Conditional Optimality Theorem, the Energy-Precision Bound, Euclidean suboptimality, the $\beta_1 = 0.9$0 geometry, and the six-class taxonomy. Conjectured items include a claimed thermodynamic efficiency advantage for self-supervised learning over supervised learning ($\beta_1 = 0.9$1), the multi-agent Markov random field formulation, and a GNC-dissipation coupling prediction. Explicitly speculative material includes the "GLP Conjecture," which posits structural parallels among Gödel incompleteness, Landauer irreversibility, and Prigogine dissipation—a pattern-recognition argument the author flags as possibly unprovable in its current form.

Qualitative predictions include: forgetting is a structural necessity for long-lived systems; failure modes are preceded by detectable changes in variability and coherence; precision, stability, and plasticity are fundamentally coupled trade-offs; and viability—remaining stable, adaptable, and resource-bounded—replaces asymptotic optimality as the primary criterion for continual and distributed systems.

Limitations and open questions

The paper is candid about restrictions. A3 (quasi-static) is the most restrictive assumption: SGD noise exceeds thermal noise by roughly $\beta_1 = 0.9$2 times, learning rate schedules violate equilibrium assumptions, and batch processing is irreversible. The quasi-static regime thus provides lower bounds, not descriptions of actual dynamics. A2 is limited because deep networks may learn representations far from maximum-entropy exponential families. A1 is physically motivated but practically implemented only through coordinate-dependent approximations.

Practical implementation of exact Fisher–Rao regularization is infeasible at scale (the Fisher matrix is $\beta_1 = 0.9$3 with $\beta_1 = 0.9$4 in the billions), and the paper surveys tractable approximations—diagonal Fisher, K-FAC, low-rank-plus-diagonal, empirical Fisher—noting that even crude proxies like SIGReg appear to capture essential geometric structure. Open questions stated explicitly include: quantifying how far SGD operates from quasi-static equilibrium; efficient Fisher–Rao regularization at scale; conditions under which the product structure between $\beta_1 = 0.9$5 and $\beta_1 = 0.9$6 breaks down; and whether self-supervised learning directly on Gaussian primitives can outperform learned latent embeddings.

Conclusion

This paper offers a conditional theoretical framework in which regularization, forgetting, and stability derive from thermodynamic constraints rather than empirical tuning. Its principal formal contributions—the Conditional Optimality Theorem, the Energy-Precision Bound, and the six-class problem taxonomy—are proven only relative to explicit assumptions whose applicability to real stochastic gradient dynamics remains unverified. The framework's value, as the author frames it, is diagnostic and interpretive: it provides a common language in which dropout, weight decay, EMA, SAC, attention, and diffusion can be compared as mechanisms regulating information dissipation. Whether this language yields quantitative predictions that survive contact with large-scale experiments is the question the work leaves open.