Autoregressive LLMs Overview

• AR-LLMs are generative models that produce text one token at a time using an autoregressive (left-to-right) approach, underpinning many modern language models.
• They utilize causal attention masks in transformer architectures to ensure each token is generated based on all previous tokens, reinforcing semantic information flow.
• Practical insights include inherent limitations such as high latency and inability to revise outputs, prompting research into hybrid models for improved efficiency.

Autoregressive LLMs (AR-LLMs) are a fundamental class of generative models in which text or other sequences are produced one token at a time, with each new token conditioned on previously generated tokens in a strict left-to-right order. This architecture underpins modern general-purpose LLMs, including GPT-family models, and serves as the backbone of state-of-the-art systems for open-ended text generation, chain-of-thought reasoning, and prompt-based learning. The autoregressive paradigm is defined by sequential, causal probability factorization and is closely linked to semantic information flow, computational limitations, and specific usability affordances.

1. Formal Definition and General Properties

AR-LLMs operate via a sequential, autoregressive process where the probability of an output sequence $Y = (y_1, ..., y_T)$ given input $X$ (or initial prompt) is factorized as: $$p(Y \mid X) = \prod_{t=1}^{T} p(y_t \mid y_{<t}, X)$$ This left-to-right conditional structure is realized on the model side via causal attention masks in transformer architectures. More generally, AR-LLMs can be formalized as time-varying vector autoregression (TV-VAR) processes: $$\mathbf{u}_t = \arg\mathrm{softmax}\left(\frac{1}{\Xi}\tilde{\mathbf{u}}_{1:N}^T \left(\sum_{j=1}^{t-1} \mathbf{A}_{tj} \mathbf{u}_j\right)\right), \qquad t = n+1, ..., T$$ where $\mathbf{u}_t$ are token embeddings, $\mathbf{A}_{tj}$ are time-varying autoregressive coefficient matrices, and $\Xi$ is a temperature parameter (Bai, 3 Nov 2025). The general probabilistic view is that the model emits a next-token distribution at each step based on previously generated outputs and the semantic context.

2. Theoretical Analysis—Semantic Information Flow, Optimality, and Limits

Recent theoretical work situates AR-LLMs within the semantic information theory framework, emphasizing that LLMs are best characterized at the token (semantic symbol) level rather than bitwise encodings. The directed information flow in AR-LLMs,

$$I(X_{1:n} \to Y_{1:n}) = \sum_{t=1}^n I(X_{1:t}; Y_t \mid Y_{1:t-1})$$

captures the causality imposed by autoregressive processing (Bai, 3 Nov 2025). Pretraining efficiency and generalization error can be related to directed rate-distortion and rate-reward functions: $$R_{\mathrm{pre}}(D) = \frac{1}{T} \inf_{\Phi : \frac{1}{T} \sum_{t=n+1}^T D_{\mathrm{KL}}(P_t^\star \| Q_t^\Phi) < D} I(S_{1:n} \to U_{n+1:T}; \Phi)$$ where $P_t^\star$ denotes target next-token distributions, $Q_t^\Phi$ model predictions, and $S_{1:n}$ source semantic embeddings.

Theoretical bounds derived from Rademacher complexity and Talagrand’s inequality link AR-LLM generalization error directly to the logit values and attention feature space, informing optimal quantization and architecture design.

Granger causality is interpreted as matching the AR-LLM's conditional predictive structure to that implicit in the data, establishing theoretical alignment between autoregressive training and causal sequence generation (Bai, 3 Nov 2025).

3. Architectural Realizations and Attention Mechanisms

The transformer implementation of AR-LLMs uses causal attention masks to ensure each token only attends to itself and prior tokens: $$\mathbf{M}_{\text{AR}} = \begin{bmatrix} 1 & -\infty & \cdots \ 1 & 1 & -\infty \ \vdots & \vdots & \ddots \end{bmatrix}$$ General attention is given by: $$\text{masked-attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d}} \odot \mathbf{M}\right)\mathbf{V}$$ where the mask $\mathbf{M}$ enforces autoregressive constraints.

The AR paradigm is not unique to transformers; it extends to architectures such as state-space sequence models (e.g., Mamba), which implement linear TV-VAR processes and can be situated within the same information-theoretic analysis, albeit with reduced non-linear expressive power.

4. Computational Complexity, Learnability, and Structural Bottlenecks

Despite the Turing-completeness of AR-LLMs in theory, formal results demonstrate intrinsic computational bottlenecks for non-sequential or structure-rich tasks. Specifically, for classes such as NP-hard reasoning, graph editing, or code tasks involving highly non-local dependencies or rewriting, AR-LLMs face superpolynomial requirements for context and generation steps, making them impractical for such domains (Yang et al., 7 Oct 2025).

Key limitations arise because the AR process is fundamentally "non-erasable": tokens cannot be revised or deleted post hoc, precluding efficient backtracking, structural editing, or dynamic context infilling. These limitations manifest in:

• Poor generalization on multi-step compositional or parity tasks (e.g., Dyck-$k$ parenthesis languages, combinatorial puzzles).
• High memory and inference costs for code generation or formal reasoning where edits and revisions are essential.
• Empirical deficits relative to paradigms permitting more flexible, non-sequential generation.

Recent work quantifies these bottlenecks, demonstrating with formal theorems that for hard problems, AR-LLMs cannot efficiently simulate even parallel algorithms unless equipped with edit or remask operations (Yang et al., 7 Oct 2025).

5. Efficiency, Throughput, and Architectural Alternatives

AR-LLMs are limited in throughput due to their single-token, strictly sequential generation. Even with optimizations such as key/value (KV) caching, each token requires a serial forward pass. Studies indicate this latency persists regardless of available computational resources (Deschenaux et al., 28 Oct 2024, Wang et al., 8 Aug 2025).

In code generation and interactive applications, this leads to high latency and restricts scalability (Li et al., 14 Sep 2025). For longer-context reasoning, AR-LLMs exhibit rapid degradation in retrieval/accuracy as context length increases—typically showing <10% accuracy at 4K-token contexts on benchmarks like RepoQA, where alternative methods maintain significantly higher performance.

Hybrid and non-autoregressive architectures have been explored to overcome these limits:

• Diffusion LLMs (dLLMs): Use parallel denoising steps to fill or update multiple tokens iteratively, allowing parallelization and improved throughput. Methods such as discrete diffusion forcing (D2F) enable blockwise AR/diffusion hybrids that match or surpass AR-LLMs in speed, maintaining comparable output quality (Wang et al., 8 Aug 2025).
• Self-Distillation Through Time (SDTT): Compresses diffusion generation to enable 32–64 token parallel updates, achieving up to 8× faster throughput compared to AR-LLMs with KV-caching, while matching NLU benchmark performance (Deschenaux et al., 28 Oct 2024).
• Blockwise and inter-block generation: Partition sequences into blocks to enable partial parallelism, further narrowing the AR–dLLM gap (Wang et al., 8 Aug 2025).

A summary of efficiency and performance trade-offs:

Model Type	Sequentiality	Token Parallelism	Inference Speed	Quality (text/code)
AR-LLM	Token-by-token	1	Baseline	Strong/SOTA (short text)
Diffusion	Multi-token/step	$k>1$	2–50× AR (blockwise)	Competitive; closing gap

6. Practical Usability, Prompting, and Cognitive Affordances

AR-LLMs enable general-purpose, free-form natural language prompting—a modality shown to provide high task customizability and transparency with minimal user-level complexity (Li et al., 17 May 2024). Unlike task-specific adaptation channels, AR-LLMs allow users to define arbitrary tasks via prompt engineering, capturing a broad array of linguistic, reasoning, and planning capacities:

• Cognitive behaviors: Chain-of-thought reasoning, multi-step planning, and feedback learning can be evoked directly via sequence prompts.
• Zero/few-shot setup: No retraining is needed; prompt design suffices for new downstream tasks.
• Deployability: AR-LLMs are readily repurposed as generalist agents and in multi-agent frameworks, enabling collaborative and iterative problem solving.

Limitations arise in tasks requiring non-sequential synthesis, in-place text editing, or large-scale codebase understanding where diffusion or edit-based paradigms provide improved alignment with real-world user processes (Li et al., 14 Sep 2025).

7. Outlook: Future Directions and Hybrid Paradigms

The structural bottlenecks of AR-LLMs for high-throughput, structure-aware, or edit-intensive applications have driven the development of hybrid, diffusion, and edit-based generation models. Hybrid systems, combining AR and diffusion principles, demonstrate complementary performance, with AR components managing structure/planning and diffusion managing flexible local refinement (Li et al., 14 Sep 2025, Wang et al., 8 Aug 2025).

Formal theory suggests that the most capable future LLMs will extend beyond pure autoregressive processing, incorporating revisable, structure-aware, or edit-capable generation processes to achieve polynomial-space learning and robust out-of-distribution generalization in mathematics, science, and programming (Yang et al., 7 Oct 2025).

The semantic information theory perspective provides a rigorous, architecture-agnostic foundation for evaluating and comparing generative architectures and training objectives, facilitating more targeted advances in LLM design (Bai, 3 Nov 2025).

---

References:

• (Yang et al., 7 Oct 2025) On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond
• (Li et al., 14 Sep 2025) Beyond Autoregression: An Empirical Study of Diffusion LLMs for Code Generation
• (Bai, 3 Nov 2025) Forget BIT, It is All about TOKEN: Towards Semantic Information Theory for LLMs
• (Deschenaux et al., 28 Oct 2024) Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
• (Li et al., 17 May 2024) Rethinking ChatGPT's Success: Usability and Cognitive Behaviors Enabled by Auto-regressive LLMs' Prompting
• (Wang et al., 8 Aug 2025) Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing