Autoregressive Diffusion Models (ADMs)

• Autoregressive Diffusion Models (ADMs) are generative models that fuse autoregressive factorization with diffusion-based denoising to efficiently generate discrete sequences.
• They enable diverse applications including text synthesis, image generation, graph creation, speech modeling, and multivariate forecasting through adaptable parallel processing.
• ADMs achieve faster inference and lossless compression while improving conditional modeling by integrating token reordering, adaptive correction, and hybrid noising strategies.

Autoregressive Diffusion Models (ADMs), often written as Autoregressive Diffusion Models (ARDMs) in the discrete-sequence literature, denote a family of generative models that combine autoregressive factorization with diffusion-style denoising, noising, or training objectives. In the foundational formulation, an ARDM learns to reveal discrete variables in an arbitrary order and subsumes both order-agnostic autoregressive models and absorbing discrete diffusion under mild assumptions (Hoogeboom et al., 2021). Subsequent work extended the label to patchwise image models, node-by-node graph generation, chunkwise video restoration, tokenwise speech synthesis, causal motion generation, and multivariate forecasting, while other papers use the term more loosely for hybrids that blur or reinterpret the boundary between autoregression and diffusion (Fathi et al., 8 Apr 2025).

1. Foundational definition and conceptual scope

The original ARDM formulation considers a discrete datapoint $x \in \{1,\dots,K\}^D$ together with a random permutation $\sigma$ over coordinates. Rather than fixing a left-to-right order, the model predicts hidden variables from a revealed prefix under a diffusion-style objective that samples only one timestep per datapoint. Its lower bound is written as

$$\log p(x)\ge \mathbb{E}_{t\sim\mathcal U(1,\ldots,D)}[D\cdot \mathcal L_t],$$

with

$$\mathcal L_t = \frac{1}{D-t+1} \mathbb E_{\sigma\sim\mathcal U(S_D)} \sum_{k\in \sigma(\ge t)} \log p(x_k\mid x_{\sigma(<t)}).$$

This construction removes the need for causal masking, because order is represented by the permutation mask rather than by a triangular attention pattern, and it supports parallel generation that can be adapted to a generation budget (Hoogeboom et al., 2021).

Within this formulation, order-agnostic autoregressive models appear as a flat special case, while absorbing discrete diffusion emerges in the continuous-time limit. The same paper shows that ARDMs require significantly fewer steps than discrete diffusion models to attain the same performance and are uniquely suited to lossless compression, including single-data-point compression, because they provide direct discrete likelihoods together with adaptable parallel generation (Hoogeboom et al., 2021).

Later work generalized the unification claim. “Unifying Autoregressive and Diffusion-Based Sequence Generation” introduces token-position-specific hyperschedules,

$$\tau_t^i \in \{0,1,\dots,T\}, \qquad T=\tau_0^i \ge \tau_1^i \ge \cdots \ge \tau_T^i = 0,$$

and interprets GPT-style autoregression as a quenched extreme with window width $\omega=1$, while standard discrete diffusion corresponds to the flat schedule $\tau_t^i=T-t$ for all positions (Fathi et al., 8 Apr 2025). In that view, ADMs are not a single architecture but a continuum of tokenwise uncertainty schedules. The same work adds hybrid noising processes and an Adaptive Correction Sampler so that already unmasked tokens can be revised rather than merely filled in once (Fathi et al., 8 Apr 2025).

2. Generative factorizations and objectives

Across the literature, the autoregressive unit in an ADM may be a token, patch, chunk, frame, or forecast step. TimeGrad is an early continuous-valued example for multivariate forecasting. It factorizes the predictive distribution over future times,

$$q_{\mathcal X}(\mathbf{x}_{t_0:T}^0 \mid \mathbf{x}_{1:t_0-1}^0, \mathbf{c}_{1:T}) = \prod_{t=t_0}^{T} q_{\mathcal X}(\mathbf{x}_{t}^0 \mid \mathbf{x}_{1:t-1}^0, \mathbf{c}_{1:T}),$$

and models each conditional factor by a denoising diffusion process conditioned on an RNN hidden state. Sampling proceeds by turning white noise into a sample at each future step through a Markov chain, yielding a state-of-the-art multivariate probabilistic forecasting method on real-world datasets with thousands of correlated dimensions (Rasul et al., 2021).

Autoregressive video diffusion adopts an analogous factorization over latent chunks. AVIS writes a video latent sequence as $z=[z^1,z^2,\dots,z^N]$ and models

$p(z)=\prod_{n=1}^{N}p(z^{n}\mid z^{<n}).$

Each chunk is restored conditioned on earlier chunks by a flow-matching backbone, and reverse diffusion is initialized from a measurement-consistent estimate rather than pure noise (Kwon et al., 20 May 2026). The autoregressive step is therefore chunkwise rather than tokenwise, but the diffusion subproblem remains iterative.

Patchwise image ADM theory introduces yet another factorization. “Capturing Conditional Dependence via Auto-regressive Diffusion Models” splits $\sigma$0 into patches $\sigma$1 and models

$\sigma$2

so that each patch is generated by a conditional reverse diffusion process given previously generated patches (Huang et al., 30 Apr 2025). Closely related image work on “Condition Errors Refinement in Autoregressive Image Generation with Diffusion Loss” instead makes the condition itself autoregressive,

$\sigma$3

and argues that the evolving condition can be refined through patch denoising rather than reused as a static conditioning signal (Zhou et al., 2 Feb 2026).

3. Architectural realizations across modalities

The architectural diversity of ADMs is substantial. In image generation, D-AR recasts diffusion as vanilla next-token prediction by learning a sequential diffusion tokenizer that maps an image to a discrete sequence $\sigma$4 and then trains

$\sigma$5

The crucial design choice is that token groups correspond to different diffusion conditioning stages in pixel space, so the autoregressive order becomes naturally coarse-to-fine. The method therefore uses a standard LLaMA-style decoder with ordinary causal masking, while the tokenizer decoder supplies the diffusion semantics, consistent previews, and zero-shot layout-controlled synthesis (Gao et al., 29 May 2025).

In graph generation, GraphArm replaces one-shot Gaussian diffusion on dequantized adjacency matrices with a discrete node-absorbing diffusion process defined directly on graphs. A learned diffusion ordering network samples a topology-dependent node absorption order,

$\sigma$6

and the reverse denoising network reconstructs one node and all of its edges to already denoised nodes at a time (Kong et al., 2023). This design makes the process discrete, graph-aware, and naturally compatible with constrained generation.

Speech ARDMs occupy a different point in the design space. “Direct Preference Optimization for Speech Autoregressive Diffusion Models” characterizes speech ARDMs as models that generate continuous latent speech tokens autoregressively, with each next token produced by diffusion rather than by direct categorical prediction. DiTAR, the model fine-tuned in that work, separates computation for generation history from denoising of the next token and thereby avoids the technical complexity of discrete speech tokenization (Liu et al., 23 Sep 2025).

Motion generation yields a further specialization. CMDM combines a Motion-Language-Aligned Causal VAE with a Causal Diffusion Transformer and trains with causal diffusion forcing, where each frame receives its own diffusion timestep. The result is an autoregressive motion generator in a causal latent space that supports text-to-motion generation, streaming synthesis, and long-horizon motion generation (Yu et al., 26 Feb 2026).

4. Inference, scalability, and computational trade-offs

A persistent theme in ADM research is that sequencing the generative process changes inference economics. The original ARDM paper formalizes budget-adaptive parallel generation through a transition-cost matrix and dynamic programming, allowing the same trained model to trade quality against the number of network calls at test time (Hoogeboom et al., 2021). This is one of the earliest explicit claims that diffusion-style models need not have a fixed inference horizon.

In video inverse problems, the advantage becomes operationally concrete. AVIS restores videos chunk by chunk, so the first chunk can be displayed without waiting for the full sequence. On $\sigma$7 video super-resolution, the reported initial latency drops from 114 s for LVTINO and 167 s for VISION-XL to 4 s for both AVIS and AVIS Flash; throughput increases from 0.71 to 1.18 FPS for AVIS and to 5.91 FPS for AVIS Flash on a single RTX 4090 GPU, while an NVIDIA H100 reaches 1.85 s latency and 10.18 FPS (Kwon et al., 20 May 2026). The acceleration comes from streaming restoration, measurement-consistent initialization, and, in AVIS Flash, enforcing measurement consistency only on the first chunk.

Long-rollout autoregressive video diffusion introduces a different bottleneck: KV-cache growth. “Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention” identifies redundant cached keys across frames, slowly evolving queries and keys, and selective cross-attention over long prompts, then proposes TempCache, AnnCA, and AnnSA as a training-free attention framework. The combined system yields up to $\sigma$8–$\sigma$9 end-to-end speedups, nearly constant peak GPU memory usage over long rollouts, and 10.7–10.8$$\log p(x)\ge \mathbb{E}_{t\sim\mathcal U(1,\ldots,D)}[D\cdot \mathcal L_t],$$0 speedup on Rolling-Forcing while preserving near-baseline visual quality (Samuel et al., 2 Feb 2026).

In motion generation, CMDM’s Frame-Wise Sampling Schedule with causal uncertainty overlaps denoising across adjacent frames rather than fully denoising one frame before starting the next. The reported throughput reaches up to 125 fps for CMDM with FSS, versus about 28 fps for standard autoregressive rollout, with about 30 ms per subsequent token compared with about 150–360 ms for AR diffusion baselines (Yu et al., 26 Feb 2026). In time-series forecasting, ARMD likewise reports much faster training and inference than prior diffusion-based TSF models; on ETTm1, the reported training/inference times are 11.872 s / 31.650 s for ARMD, against 149.974 s / 1183.593 s for Diffusion-TS and 449.792 s / 1749.896 s for TimeGrad (Gao et al., 2024).

5. Conditional dependence, guidance, and alignment

A major theoretical argument for ADMs is that they can model conditional dependence structures more faithfully than vanilla diffusion. “Capturing Conditional Dependence via Auto-regressive Diffusion Models” proves conditional KL bounds for patchwise AR diffusion and exhibits a contrasting lower bound showing that a model may have small joint KL while its conditional distribution $$\log p(x)\ge \mathbb{E}_{t\sim\mathcal U(1,\ldots,D)}[D\cdot \mathcal L_t],$$1 remains arbitrarily inaccurate (Huang et al., 30 Apr 2025). Empirically, the same work reports that AR diffusion captures structured relations such as sun–shadow geometry and odd/even or arithmetic dependencies in composite MNIST, but does not outperform DDPM when there is no obvious cross-patch conditional dependence. This directly counters the misconception that autoregressive factorization is universally beneficial.

Guidance in discrete ARDMs is also tractable in ways that differ from continuous diffusion. “Discriminator Guidance for Autoregressive Diffusion Models” defines discriminator ratios over partial samples and shows that, with an optimal discriminator, the pretrained conditional can be corrected exactly to the true data conditional distribution. For realistic sub-optimal discriminators, the paper derives Sequential Monte Carlo variants—BSDG and FADG—with ESS-based resampling (Kelvinius et al., 2023). On molecular graph generation, BSDG and FADG substantially improve validity and molecular stability over the pretrained ARDM, reaching around 98% validity and about 88% molecular stability on QM9 (Kelvinius et al., 2023).

Preference alignment extends the same principle to continuous-token speech ARDMs. ARDM-DPO optimizes full ARDM sampling trajectories rather than isolated outputs and reports large gains on expressiveness and robustness for DiTAR. On LibriTTS-based expressiveness alignment, F0 variance rises from 14.2 to 29.2, WER improves from 5.17 to 3.73, speaker similarity changes only slightly from 0.770 to 0.765, and KL divergence to the reference model remains 0.010 (Liu et al., 23 Sep 2025). Related image theory on diffusion loss models argues that patch denoising optimization refines the condition, yielding an exponential decay bound

$$\log p(x)\ge \mathbb{E}_{t\sim\mathcal U(1,\ldots,D)}[D\cdot \mathcal L_t],$$2

and proposes OT-based condition refinement as a Wasserstein Gradient Flow to mitigate condition inconsistency (Zhou et al., 2 Feb 2026).

6. Empirical landscape, neighboring paradigms, and terminological disputes

Empirically, ADM-style methods have been competitive across disparate domains, but with domain-specific strengths. The original ARDM benchmarks report 1.43 bpc on text8 for OA-ARDM and strong CIFAR-10 likelihoods together with 2.71 bpd per-image compression for ARDM-Upscale 4 on CIFAR-10 (Hoogeboom et al., 2021). GraphArm achieves better or comparable graph generation quality than prior state of the art while being up to 100$$\log p(x)\ge \mathbb{E}_{t\sim\mathcal U(1,\ldots,D)}[D\cdot \mathcal L_t],$$3 faster than GDSS and around 30$$\log p(x)\ge \mathbb{E}_{t\sim\mathcal U(1,\ldots,D)}[D\cdot \mathcal L_t],$$4 faster than DiGress on some datasets (Kong et al., 2023). TimeGrad reports new state-of-the-art multivariate probabilistic forecasting results on all but the smallest benchmark among the datasets evaluated (Rasul et al., 2021). ARMD reports the best performance in 12 out of 14 diffusion-based forecasting settings and, on ETTm1, a 47.7% MSE reduction and 30.1% MAE reduction versus the second-best model D3VAE (Gao et al., 2024). In image generation, D-AR achieves 2.09 FID on ImageNet $$\log p(x)\ge \mathbb{E}_{t\sim\mathcal U(1,\ldots,D)}[D\cdot \mathcal L_t],$$5 using a 775M Llama backbone with 256 discrete tokens (Gao et al., 29 May 2025). In music information content estimation, ADM-based diffusion ODE models outperform GIVT on negative log-likelihood across JAM, SAL, VOC, and SYN, and improve both pitch surprisal alignment and segment-boundary detection at appropriate noise levels (Bjare et al., 7 Aug 2025).

The term “ADM” nevertheless remains contested. “Visual Autoregressive Models Beat Diffusion Models on Inference Time Scaling” explicitly states that it is not really proposing a new “Autoregressive Diffusion Model (ADM)” in the classical sense; instead, it argues that discrete visual autoregressive models are architecturally better matched to search, beam pruning, and prefix caching than continuous diffusion models (Riise et al., 19 Oct 2025). By contrast, “Beyond Autoregression” argues for discrete diffusion as a superior inductive bias for reasoning and planning, reporting 91.5% and 100% accuracy on Countdown and Sudoku, respectively, versus 45.8% and 20.7% for autoregressive models (Ye et al., 2024). A3 reaches the opposite conclusion for flexible language generation, claiming that generalized autoregression can recover any-order, bidirectional generation without diffusion-style training (Du et al., 19 Jan 2026), while “Diffusion in Diffusion” adds a global revisional diffusion stage to semi-autoregressive block diffusion models rather than choosing a pure AR or pure diffusion position (Ma et al., 20 Jan 2026). This suggests that “ADM” now names a design region rather than a single settled architecture: in some papers it denotes discrete order-agnostic diffusion-like autoregression, in others next-token diffusion in continuous latent space, and in still others an AR–diffusion hybrid whose main contribution is to reallocate where sequencing and denoising occur.