Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

Abstract: Autoregressive models (ARMs) currently constitute the dominant paradigm for LLMs. Energy-based models (EBMs) represent another class of models, which have historically been less prevalent in LLM development, yet naturally characterize the optimal policy in post-training alignment. In this paper, we provide a unified view of these two model classes. Taking the chain rule of probability as a starting point, we establish an explicit bijection between ARMs and EBMs in function space, which we show to correspond to a special case of the soft Bellman equation in maximum entropy reinforcement learning. Building upon this bijection, we derive the equivalence between supervised learning of ARMs and EBMs. Furthermore, we analyze the distillation of EBMs into ARMs by providing theoretical error bounds. Our results provide insights into the ability of ARMs to plan ahead, despite being based on the next-token prediction paradigm.

• The paper establishes a precise bijection between autoregressive and energy-based models using the chain rule, showing that teacher forcing recovers globally optimal, lookahead-aware sequence modeling.
• It integrates maximum entropy reinforcement learning and dynamic programming to bridge local next-token prediction with a soft value function that captures global planning.
• It quantifies the approximation error in finite-capacity models via KL divergence bounds, with empirical results confirming the theoretical unification between ARMs and EBMs.

Autoregressive LLMs as Energy-Based Models: A Unified Theoretical View

Introduction and Motivation

This essay examines "Autoregressive LLMs are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction" (2512.15605), which bridges the theoretical frameworks underpinning Autoregressive Models (ARMs) and Energy-Based Models (EBMs) in language modeling. While ARMs (e.g., next-token prediction transformers) dominate LLM implementations due to efficient training and parallelizable sampling, EBMs possess inherent global planning and lookahead capabilities but suffer from intractable partition functions. The paper establishes a precise bijection between these two model classes, grounded in the chain rule of probability and the soft Bellman equation from maximum entropy reinforcement learning (MaxEnt RL). This yields novel theoretical and empirical insights into the approximation power of ARMs, their limitations and capabilities around lookahead, and provides a formal justification for the teacher forcing paradigm.

Unification of ARMs and EBMs

The central result is the proof of an explicit bijection between globally normalized sequence models (EBMs) and locally normalized, next-token models (ARMs) in function space. For EBMs, the predictive distribution over a sequence is defined via a global "reward" $R(X, Y)$ and a log-partition function that is intractable for large spaces. In contrast, ARMs construct the sequence probability by chaining local next-token conditional distributions parameterized by a function $q(S_t, y_t)$.

The authors show that, via the chain rule, for any EBM defined by reward $R$ (decomposable into per-token terms), there exists a corresponding ARM with a local scoring function $q$ that augments the immediate reward with a "soft value" term encoding the log-sum-exp over possible future continuations—a direct analogue to the soft Bellman equation. Conversely, ARM parameters $q$ can be mapped back to EBM reward functions efficiently. This mapping is bijective in function space, i.e., under tabular (universal) parametrization, ARM and EBM families are theoretically equivalent in expressive power.

Figure 1: Empirical validation of the equivalence of minima—Training losses and logits convergence for EBM and ARM parameterized by Transformers.

Maximum Entropy RL and Dynamic Programming Perspective

The unification is shown to correspond to concepts well-studied in MaxEnt RL. Specifically, ARMs are interpretable as MDP policies over context-token spaces, with soft Bellman backups yielding the local-to-global reward transformation. The mapping $q = \mathcal{B}(r)$ is explicitly constructed in the acyclic, deterministic, finite-horizon case relevant for LLMs. This clarifies the equivalence between value-based (reward function/RL) and policy-based (conditional distributions) learning for LLMs, mapping the problem of distilling an EBM into an ARM directly onto MaxEnt RL policy optimization.

Further, the connection to entropy-regularized dynamic programming on a prefix-tree DAG is made precise—enabling the use of DP for soft-maximizing total sequence reward.

Theoretical Implications: Equivalence of Minima and Optimality of Teacher Forcing

A key theoretical result is that, when learning from input-output pairs by minimizing negative log-likelihood with sufficient model capacity (function space), ARMs and EBMs not only enjoy the same global optimum but their optimal parameters are in bijection (Proposition 2). Hence, teacher forcing—the standard ARM objective—is in principle sufficient to recover globally consistent, lookahead-aware sequence models, supposing optimization and model class capacity are not limiting.

Empirical results confirm that, for over-parameterized toy models trained to optimality, EBM and ARM Transformer models achieve identical training losses, and post-hoc application of the bijection maps their logits with near-zero error (Figure 1).

Figure 2: Loss convergence and logits distance for different Transformer sizes, illustrating convergence towards the same minima.

Quantifying the ARM Approximation to EBMs Under Finite Capacity

In practice, ARMs are parameterized by neural architectures such as causal transformers, with limited capacity to exactly realize the soft value function for every context. The paper derives a bound on the KL divergence between the EBM solution and an approximate ARM, showing that the error is proportional to $2T$ times the maximum logit difference between the two models (Proposition 3). Furthermore, recent universal approximation theorems guarantee that causal transformers can approximate the required mapping arbitrarily well, at the cost of scaling with vocabulary size.

Figure 3: Empirical validation of the KL divergence upper bound by the maximal infinity-norm distance between ARM and mapped EBM logits.

Addressing Lookahead and "Planning" in ARMs

A notable claim is the refutation of structural myopia in ARMs: global lookahead and credit assignment are possible when optimization and function approximation are ideal, because the required soft value is implicitly encoded in the optimal ARM logits. Failures to plan are attributed not to the next-token objective per se, but to finite capacity, optimization error, or poor inductive bias.

Empirical results demonstrate that as model capacity increases (Transformer depth, embedding size), the ARM logits align more closely with the mapped EBM logits, and both training loss and logits distance decrease accordingly (Figure 2, Figure 4).

Figure 4: Loss and logits convergence for the high $T > V$ regime, illustrating consistent alignment between ARMs and EBMs as training proceeds.

Limitations and Open Directions

The paper acknowledges practical limitations: while ARMs and EBMs are equivalent in function space, the architectural demands of implementing the soft value function within typical causally-masked models pose significant challenges, especially in long-horizon, large-vocabulary settings. Furthermore, non-causal architectures (as in EBMs) may be practically more efficient when global rewards are non-local, but ARMs are typically limited to causal/fixed context representations. The relative difficulty of optimization for $q$ vs.\ $R$, the role of latent variable augmentations or memory mechanisms, and improved architectures for capturing lookahead efficiently remain open areas.

Implications and Prospects

This theoretical unification reframes the choice between ARM and EBM sequence modeling for LLMs. It formalizes the power and limitations of next-token prediction and clarifies the circumstances under which teacher forcing suffices for global sequence-level objectives. It lays the groundwork for hybrid architectures, informs RLHF-style alignment by framing it as EBM-to-ARM distillation, and motivates further work on architecture and optimization to close the practical expressivity gap.

Conclusion

This work provides a rigorous framework connecting ARMs and EBMs for autoregressive language modeling, establishes their theoretical equivalence in function space, and demonstrates that next-token prediction can in principle implement globally normalized, value-aware sequence models. Practical implications include justification of teacher forcing, clarification of planning capabilities, and precise quantification of ARM approximation quality under function approximation. The results inform future research in language modeling architectures, RL-based alignment, and the broader synergy between generative modeling, probabilistic inference, and reinforcement learning paradigms.