Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective

Abstract: Autoregressive generative models -- including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba -- all generate sequences by sampling each output from a deterministic summary of the past, producing genuinely non-Markovian observed processes. We develop a general theoretical framework based on stochastic thermodynamics for this class of architectures and introduce the entropy production, which can be efficiently estimated from sampled trajectories without exponential sampling cost despite the non-Markovian nature of the observed dynamics. As a proof-of-concept experiment for a LLM, we evaluate the token-level and sentence-level entropy production for a pre-trained Transformer-based model, GPT-2. We also demonstrate the framework in the linear Gaussian case, where the model reduces to the Kalman innovation representation and the entropy production admits an analytical expression. We further show that the entropy production decomposes exactly into non-negative per-step contributions in terms of retrospective inference, and each of those terms further splits into information-theoretically meaningful terms: a compression loss and a model mismatch. Our results establish a bridge between stochastic thermodynamics and modern generative models, and provide a starting point for quantifying irreversibility in a broad class of highly non-Markovian processes such as LLMs.

• The paper introduces a unified stochastic thermodynamics framework for autoregressive models with deterministic latent memory to quantify irreversibility.
• The approach enables efficient Monte Carlo estimation of entropy production at token and block levels across diverse architectures including Transformers and RNNs.
• Analytical results for linear models and GPT-2 experiments validate the framework, offering insights into non-Markovian dynamics and information flow.

Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective

Introduction and Theoretical Framework

This work introduces a formal theory of stochastic thermodynamics for autoregressive generative models with deterministic latent memory, encompassing a broad spectrum of architectures, including Transformers, RNNs, state-space models (SSMs), Mamba, and the classical Kalman filter. These architectures generate sequences by recursively sampling each output conditioned on a deterministic function of the observed history, thereby realizing genuinely non-Markovian observable dynamics. The key insight is that, despite the high-order non-Markovianity, the explicit separation between deterministic latent updates and stochastic emissions enables tractable computation of entropy production in the sense of stochastic thermodynamics.

The theoretical setup treats the observed variables $\{y_t\}_{t=1}^T$ and their deterministic latent states $\{h_t\}_{t=0}^T$, with $y_{t+1} \sim p_t(y_{t+1} \mid h_t)$ and $h_t = \Phi_t(y_{1:t})$. The emission kernel $p_t$ and latent update $\Phi_t$ may be general, encompassing non-recursive (Transformer) and recursive (RNN, SSM, Kalman) instantiations. An essential restriction is that the size of $h_t$ remains fixed, preventing a trivial encoding of the entire history. This framework establishes a unifying foundation for analyzing irreversibility and entropy flow in modern sequence models—a necessity given the widespread deployment of large autoregressive models in both AI and control systems.

Figure 1: Model depiction for the general non-recursive autoregressive case—forward direction (a) and protocol-reversed backward process (b) with explicit dependency graph.

Entropy Production in Non-Markovian Autoregressive Processes

Entropy production, $\mathcal{S}_y$, is defined as the Kullback-Leibler divergence between the forward path probability $P_\to(y_{1:T})$ and the backward path probability $P_\leftarrow(y_{T:1})$, where the latter is constructed by applying the forward model components in strict reverse time order. The formulation leverages the fact that in all included models, the emission kernels are explicit once the latent state trajectory is specified, and the full latent trajectory is uniquely determined by the observed trajectory via deterministic functions.

The entropy production is thus:

$\{h_t\}_{t=0}^T$0

with the stochastic entropy production for a particular realization $\{h_t\}_{t=0}^T$1, satisfying the integral fluctuation theorem, $\{h_t\}_{t=0}^T$2. Importantly, no assumptions of Markovianity are needed for direct and unbiased estimation of $\{h_t\}_{t=0}^T$3.

Figure 2: Specialization to the recursive case where joint (latent, observation) dynamics are Markovian but observed dynamics remain non-Markovian.

Monte Carlo Estimation and Computational Tractability

Contrasted with typical non-Markovian processes, where trajectory-based entropy production estimation suffers exponential sampling overhead, the structure of autoregressive generative models enables efficient Monte Carlo computation. For any sampled forward trajectory $\{h_t\}_{t=0}^T$4, the latent state sequence is exactly recovered; both forward and backward likelihoods can be computed through paired forward passes (original and reversed/modified sequence). For practical architectures:

• Transformer-based models require $\{h_t\}_{t=0}^T$5 per-pass computation due to full self-attention.
• Recursive models (RNN, SSM, Mamba, Kalman) require $\{h_t\}_{t=0}^T$6 per pass.

The Monte Carlo variance is standard—the estimator converges at rate $\{h_t\}_{t=0}^T$7 samples without any worsening due to the model’s non-Markovianity.

Temporal Coarse-Graining: Block-Level Entropy Production

Token-level path reversal is typically dominated by syntactic artifacts that obscure physically or semantically relevant irreversibility. The framework therefore introduces block-wise reversal, where sequences are divided into contiguous blocks (preferably sentences or episodes), and coarse-grained entropy production is evaluated by reversing block order but preserving order within blocks. This block-level entropy production is more interpretable, especially for LLMs where block boundaries are aligned with semantic or discourse structure.

Empirical Validation: GPT-2 Experiments

Proof-of-concept experiments are conducted with GPT-2 (117M parameters). For each sampled or fixed sequence, both token-level and block-level entropy production are evaluated.

Figure 3: Distributions of per-token stochastic entropy production for GPT-2-generated sequences. The token-level reversal displays large values, reflecting syntactic irreversibility.

A substantial gap is observed between token-reversal and block-reversal entropy production—block-reversal values are an order of magnitude lower, confirming that the syntactic destruction dominates token-level measurements. In fixed-text experiments (where input sequences are fixed and not generated by GPT-2), causal narratives yield higher block-level entropy production than non-causal (order-insensitive) collections, indicating sensitivity to underlying event dependencies rather than superficial sequence ordering.

Figure 4: Per-token entropy production for 30 causal and 30 non-causal externally generated texts. Block-level reversal reveals significant separation, highlighting semantic irreversibility.

Monte Carlo convergence and satisfaction of the fluctuation theorem are numerically demonstrated.

Figure 5: Convergence of sample mean entropy production estimate with increasing sample size, confirming statistical efficiency and estimator convergence.

Analytical Results for Linear Gaussian Models

The framework is specialized to the linear Gaussian (Kalman innovation) case, allowing derivation of exact, closed-form entropy production expressions. Key findings include:

• For scalar observation processes, stationary Gaussianity ensures time-reversibility and vanishing asymptotic entropy production.
• For multidimensional systems, entropy production can grow linearly with trajectory length if cross-covariances break symmetry (i.e., genuine time-irreversibility).

Analytical predictions are validated by Monte Carlo sampling, confirming estimator consistency and formula correctness.

Figure 6: Direct comparison of analytical and sampled entropy production for scalar and multivariate linear Gaussian processes.

Information-Theoretic Decompositions

Entropy production admits exact, non-negative per-step decompositions $\{h_t\}_{t=0}^T$8 via retrospective Bayesian inference, with each $\{h_t\}_{t=0}^T$9 further decomposing into:

• Compression loss $y_{t+1} \sim p_t(y_{t+1} \mid h_t)$0: quantifies residual future information lost by deterministic compression into latent state.
• Model mismatch $y_{t+1} \sim p_t(y_{t+1} \mid h_t)$1: quantifies KL divergence between the backward emission model and true retrospective posterior.

A formal analogy is drawn to ELBO decompositions in variational inference, though the origin here is time-reversal and thermodynamics rather than likelihood bounding.

A refined lower bound is obtained, showing total entropy production is bounded from below by the difference in predictive information (mutual information) captured by the forward and backward latent representations.

Implications, Open Problems, and Future Directions

The theoretical connection between stochastic thermodynamics and autoregressive machine learning models invites further investigation of finite-time trade-offs (accuracy, speed, irreversibility), with analogues to thermodynamic uncertainty and speed-precision relations. Application to more semantically complex models and real-world data will necessitate additional semantic-level coarse-graining and better disentanglement between causal, temporal, and discourse-induced irreversibility. These developments may ultimately inform the measurement of physicality, causality, and agency in AI systems.

Conclusion

A comprehensive and computationally tractable thermodynamic theory for a broad class of deterministic-memory, autoregressive generative models is established. The work unifies diverse sequence models under a single formalism, elucidates the precise structure of irreversibility in observed processes, and demonstrates the empirical and analytic utility of entropy production as a diagnostic for non-Markovianity and information flow. This bridges modern AI practice with foundational principles in statistical physics and information theory, setting the stage for both practical probing of large generative models and theoretical advancement in stochastic thermodynamics.

(2604.07867)