Fast-ARDiff: Accelerating AR Diffusion Models

• Fast-ARDiff is a framework for accelerating autoregressive diffusion models in vision through adaptive and hybrid inference methods.
• It leverages adjacency-adaptive draft trees and entropy-guided speculative decoding to dynamically optimize sampling across complex image regions.
• Empirical studies report 3–5× speedups with maintained generative quality, enabling scalable, real-time high-fidelity image synthesis.

Fast-ARDiff encompasses several distinct but thematically convergent lines of research on fast inference for autoregressive diffusion models, specifically in the context of vision and hybrid AR-diffusion generative modeling. "Fast-ARDiff" refers both to model-specific acceleration frameworks—such as those utilizing adjacency-adaptive dynamical draft trees or entropy-informed speculative decoding—and to the broader class of parallel and hybrid AR-diffusion algorithms that fundamentally accelerate sequential autoregressive sampling to approach or surpass diffusion-level efficiency.

1. Autoregressive Diffusion: Motivation and Inference Bottlenecks

Autoregressive (AR) models, widely adopted in high-fidelity image synthesis (e.g., Emu3, Anole, Lumina-mGPT), operate by tokenizing images into sequences $s_{1:T}$ and decoding them via the chain rule: $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$. For images of size $576\times 576$, tokenization typically yields $T\sim 2,000$ sequential steps, resulting in inference times of tens of seconds per image and orders-of-magnitude slower sampling than diffusion models, which require only $\sim20$–$50$ denoising steps. Naïve sequential AR inference is thus prohibitive for real-time or large-scale deployment (Lei et al., 26 Dec 2025, Zou et al., 9 Dec 2025, Zou et al., 30 Mar 2026).

Hybrid AR-diffusion models further exacerbate the bottleneck, as they require both stepwise AR latent prediction and iterative diffusion-based refinement, leading to a cumulative latency $$T_\text{AR+Diff} = O(N) t_\text{AR} + O(T) t_\text{Diff}$$ for $N$ AR and $T$ diffusion steps (Zou et al., 9 Dec 2025).

2. Fast-ARDiff via Adjacency-Adaptive Dynamical Draft Trees

A primary paradigm within Fast-ARDiff employs Adjacency-Adaptive Dynamical Draft Trees (ADT-Tree) for spatially and contextually adaptive speculative decoding (Lei et al., 26 Dec 2025). Traditional speculative decoding, originally successful in LLMs, involves generating $L$ proposed tokens in parallel with a lightweight draft model, then verifying their acceptability with a heavy target model. However, visual AR domains face spatial variability in prediction difficulty: smooth image regions yield long acceptance prefixes, whereas complex textures yield short prefixes, invalidating static draft parameters.

ADT-Tree addresses this by dynamically adapting tree depth $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$0 and width $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$1 per image position $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$2. Key features include:

• Horizontal adjacency initialization: Each token's draft-tree parameters inherit from its left neighbor.
• Bisectional dynamic adaptation: Parameters are incremented or decremented based on observed acceptance rates, $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$3, with step sizes $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$4, $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$5, enforcing bounds $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$6.
• Empirical results: On MS-COCO 2017 and PartiPrompts, ADT-Tree yields $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$7 and $p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$8 acceleration over vanilla AR, with indistinguishable quality in HPSv2, CLIP, and FID metrics. The method is orthogonal to relaxed acceptance schemes such as LANTERN, allowing stacking of speedups (Lei et al., 26 Dec 2025).

Method	Speedup	$p(I) = \prod_{t=1}^T p(s_t | s_{1:t-1})$9 (Accepted prefix)	$576\times 576$0 (Draft depth)	HPSv2	CLIP
Anole (vanilla)	1.00×	1.00	1.00	0.2309	0.3086
EAGLE-2	1.62×	2.91	5.00	0.2338	0.3078
LANTERN	3.03×	4.25	5.00	0.2188	0.2955
ADT-Tree	2.21×	3.40	3.86	0.2331	0.3081
ADT-Tree + LANTERN	3.13×	4.86	5.15	0.2191	0.2965

The ADT-Tree mechanism thus adapts compute allocation in response to local generative uncertainty, substantially improving AR sample-time efficiency (Lei et al., 26 Dec 2025).

3. Entropy-Guided Speculative Decoding and Hybrid AR-Diffusion

Another strand of Fast-ARDiff research exploits entropy alignment between draft and target AR models to maximize speculative prefix acceptance and eliminate redundancy in hybrid AR-diffusion frameworks (Zou et al., 9 Dec 2025, Zou et al., 30 Mar 2026). The principal observations are:

• Draft overconfidence (low attention entropy) leads to high target rejection rates. Training the draft with a feature-level entropy loss, $576\times 576$1, aligns it with the target's entropy profile, increasing acceptance rates (Zou et al., 9 Dec 2025, Zou et al., 30 Mar 2026).
• Shallow-feature entropy pre-filtering allows early rejection of speculative prefixes with low entropy, aborting redundant target model calls.

In AR-diffusion hybrids (e.g., MAR, TransDiff, NextStep-1), entropy from the AR attention maps is also propagated to optimize diffusion-step schedule and distillation, producing a truly unified acceleration framework. The overall loss combines AR-regression, entropy, and diffusion distillation (trajectory and distribution matching) terms, annealed by a scheduler over training (Zou et al., 9 Dec 2025).

Empirical results show state-of-the-art accelerations: $576\times 576$2 lossless speedup for TransDiff on ImageNet 256$576\times 576$3256, and up to $576\times 576$4 for text-conditional NextStep-1, without significant FID or CLIP degradation. Aggressive speculative parameters or suboptimal entropy thresholds can, however, compromise semantic coherence.

4. Drift-AR and Single-Step Decoding: Entropy as Unified Uncertainty

Recent advances reinterpret per-position prediction entropy as a unifying signal for both AR speculative decoding and diffusion-based vision decoding (Zou et al., 30 Mar 2026). Drift-AR employs entropy-informed speculative decoding to manage acceptance lengths during AR sampling, and then uses the same entropy map to parameterize the variance of a single-step anti-symmetric drift field in the diffusion decoder:

• At each spatial position, the draft AR feature $576\times 576$5 yields an entropy $576\times 576$6; variance is set via $576\times 576$7.
• The one-step drift field $576\times 576$8, trained in feature space, pushes the initial entropy-parameterized latent toward the data manifold in a single forward pass.

This single-network-function-evaluation (1-NFE) approach achieves $576\times 576$9–$T\sim 2,000$0 speedup (ImageNet 256$T\sim 2,000$1256, TransDiff, NextStep-1), often with improved FID and Inception Scores compared to original multi-step pipelines. Notably, this approach yields genuine one-pass generation while maintaining distributional expressivity (Zou et al., 30 Mar 2026).

5. ARDM: Parallel Generation with Order-Agnostic Autoregressive Diffusion

Order-agnostic AR diffusion models provide a foundational fast-ARDiff mechanism for discrete domains by leveraging absorbing diffusion and parallel generation (Hoogeboom et al., 2021). Core elements include:

• Absorbing state forward process: sequentially masking one coordinate per step in a random order, leading to an order-agnostic chain.
• Reverse process: autoregressively restoring masked coordinates, parameterized as $T\sim 2,000$2.
• Efficient training objective: Evidence Lower Bound (ELBO) over orderings, enabling fast, mask-based architectures without causal attention.
• Parallel sampling: Dynamic programming selects optimal nonuniform update blocks, trading off number of network calls (latency) for likelihood and fidelity.

On CIFAR-10, for dimension $T\sim 2,000$3, ARDM-parallel produces competitive bits/dim (2.74) in only $T\sim 2,000$4 calls, versus >1000 for full absorbing diffusion. The method also scales to high-dimensional lossless compression and audio (Hoogeboom et al., 2021).

6. Empirical Performance and Benchmarks

Fast-ARDiff methods consistently demonstrate multi-fold improvements in inference speed without notable degradation in generative quality across multiple tasks. Below is a consolidated reference table:

Model / Task	Baseline Time	Fast-ARDiff Time	Speedup	FID	IS	Notes
MAR-L (ImageNet 256x256) (Zou et al., 9 Dec 2025)	5.31 s	1.21 s	4.38×	1.77	296.0	Entropy-informed + DMD
TransDiff-L (ImageNet 256x256)	3.17 s	0.68 s	4.69×	1.66	295.1
NextStep-1 (Text–Image)	–	–	2.97×	6.68	–	GenEval 0.66, CLIP 28.88
ADT-Tree (COCO) (Lei et al., 26 Dec 2025)	–	–	3.13×	–	–	HPSv2 0.2191, CLIP 0.2965
Drift-AR (ImageNet 256x256) (Zou et al., 30 Mar 2026)	5.31 s	0.96 s	5.53×	1.76	297.4	1-NFE, single-step drift

Qualitative inspection shows preserved detail and semantic consistency, with ablations confirming the necessity of entropy parameterization and anti-symmetric drift in achieving 1-NFE synthesis (Zou et al., 30 Mar 2026).

7. Limitations, Extensions, and Open Problems

Fast-ARDiff faces several open challenges:

• Effectiveness of local adaptation (ADT-Tree) is reduced when image or data entropy is uniform (Lei et al., 26 Dec 2025).
• Success of entropy-based speculative decoding depends on accurate calibration and the validity of attention entropy as an uncertainty proxy (Zou et al., 9 Dec 2025, Zou et al., 30 Mar 2026).
• The current two-phase training for AR-drift fusion requires careful annealing; more general and automated schedulers may extend applicability.
• Extension to video, hierarchical, or multi-modal generative tasks necessitates richer per-position entropy modeling and possibly new theoretical underpinnings (Zou et al., 30 Mar 2026).
• Open research includes meta-learning of scheduler parameters, token-caching in diffusion, and exploration of alternative divergence metrics (e.g., Wasserstein) for distribution alignment (Zou et al., 9 Dec 2025, Zou et al., 30 Mar 2026).

Further, the ADT-Tree’s adaptation is confined to spatial adjacency and acceptance rates, but could be augmented with patch-level visual features or multi-scale AR tokenizations to better anticipate region-specific difficulty (Lei et al., 26 Dec 2025).

• "Fast Inference of Visual Autoregressive Model with Adjacency-Adaptive Dynamical Draft Trees" (Lei et al., 26 Dec 2025)
• "Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation" (Zou et al., 9 Dec 2025)
• "Drift-AR: Single-Step Visual Autoregressive Generation via Anti-Symmetric Drifting" (Zou et al., 30 Mar 2026)
• "Autoregressive Diffusion Models" (Hoogeboom et al., 2021)