Vision AutoRegressive Model (VAR)

• VAR is a scale-level autoregressive image generation method that factorizes images into multi-scale token maps, enabling efficient parallel prediction.
• It combines a multi-scale VQVAE tokenizer and a GPT-style Transformer decoder to generate high-fidelity images with drastically fewer autoregressive steps.
• Empirical results on ImageNet show VAR improves FID from 18.65 to 1.73 and boosts IS while achieving roughly 20x faster inference compared to diffusion models.

Searching arXiv for the primary VAR paper and closely related follow-up work to ground the article in current literature. Visual AutoRegressive modeling (VAR) is a scale-level autoregressive paradigm for image generation that redefines autoregressive learning on images as coarse-to-fine “next-scale prediction” or “next-resolution prediction,” rather than raster-scan “next-token prediction.” In VAR, an image is represented as a hierarchy of discrete token maps, and the joint distribution is factorized across scales so that all tokens within a single scale are generated in parallel. This formulation preserves spatial locality while reducing the number of autoregressive steps to the number of scales, and it has been reported to make GPT-like autoregressive models surpass diffusion transformers in image generation on ImageNet 256×256, improving FID from 18.65 to 1.73 and IS from 80.4 to 350.2, with around 20x faster inference speed (Tian et al., 2024). Within the broader taxonomy of autoregressive models in vision, VAR is a canonical instance of scale-level factorization, positioned between pixel-level and token-level approaches in its trade-off among fidelity, efficiency, and spatial coherence (Xiong et al., 2024).

1. Probabilistic formulation and representation strategy

The defining probabilistic structure of VAR is a coarse-to-fine factorization over multi-scale token maps. If an image is encoded into a hierarchy of discrete maps $\{r_1,r_2,\dots,r_K\}$, where each $r_k\in[V]^{h_k\times w_k}$, then VAR models the joint distribution as

$$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$

Crucially, within a single scale $k$, all $h_k w_k$ tokens are generated in parallel, so the autoregressive dependence is imposed across scales rather than across all spatial locations (Tian et al., 2024).

This places VAR in the “scale-level” family of visual autoregressive models. The survey literature distinguishes pixel-level, token-level, and scale-level factorization strategies, and characterizes scale-level models as balancing efficiency and spatial locality by generating coarse maps before finer details. In that account, VAR’s multi-scale token maps $R_1,\dots,R_K$ are generated as

$$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$

with tokens inside each scale generated in parallel under a block-wise causal mask (Xiong et al., 2024).

Several later works restate the same idea in closely related notation. M-VAR describes an image as a hierarchy $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$ and writes

$$p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,$$

while emphasizing that original VAR implements each conditional with a Transformer attending over the entire flattened sequence $[\mathbf{x}^1;\dots;\mathbf{x}^{s-1};\mathbf{x}^s]$ (Ren et al., 2024). CSD-VAR expresses the same factorization for VQ-VAE token maps $r_k\in[V]^{h_k\times w_k}$0, with optional conditioning $r_k\in[V]^{h_k\times w_k}$1,

$r_k\in[V]^{h_k\times w_k}$2

and further notes that preceding maps are upsampled by interpolation to keep the number of tokens fixed per scale (Nguyen et al., 18 Jul 2025).

A distinct but related formulation appears in adaptation work that presents VAR through standard autoregressive likelihood over VQ-GAN codes. There, an image $r_k\in[V]^{h_k\times w_k}$3 is tokenized into $r_k\in[V]^{h_k\times w_k}$4 discrete codes $r_k\in[V]^{h_k\times w_k}$5, and the model writes

$r_k\in[V]^{h_k\times w_k}$6

while noting that VAR groups tokens by “scale” levels and attaches both 2D positional embeddings and level embeddings (Shaikh et al., 15 Jul 2025). This suggests that the scale-level factorization can be viewed as a structured autoregressive decomposition layered atop a conventional likelihood objective.

2. Core architecture and generation pipeline

The original VAR architecture combines a multi-scale tokenizer, a decoder-only Transformer backbone, and a reconstruction path back to RGB. The tokenizer is a multi-scale VQVAE: a CNN encoder $r_k\in[V]^{h_k\times w_k}$7 produces a feature map $r_k\in[V]^{h_k\times w_k}$8, which is downsampled to each scale, quantized by a shared codebook $r_k\in[V]^{h_k\times w_k}$9, decoded back to the original latent resolution, and subtracted through a small convolution $$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$0. This residual construction ensures that each scale depends only on earlier scales. Training of the tokenizer uses

$$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$1

where $$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$2 is a perceptual loss and $$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$3 a GAN-style discriminator loss (Tian et al., 2024).

The Transformer itself follows a GPT-2–style decoder-only stack. In the class-conditional setting, it uses Adaptive LayerNorm conditioned on a class embedding prepended as a special token; it uses absolute position embeddings, and the reported implementation “does not” incorporate RoPE or SwiGLU. At generation step $$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$4, the input sequence consists of the coarser scales $$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$5 plus blank placeholders for scale $$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$6, and a single forward pass produces logits for all $$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$7 positions simultaneously. No specialized U-Net or cross-attention modules are required; the model is described as “self-attended in one monolithic transformer” (Tian et al., 2024).

The more formal architectural write-up from the circuit-complexity study decomposes VAR into three phases. In Phase 1, next-scale prediction processes token maps $$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$8 through a stack of $$p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.$$9 Transformer layers of the form

$k$0

where $k$1 is bicubic up-interpolation, $k$2 is a standard self-attention block,

$k$3

and $k$4 consists of an MLP and a layer normalization (Ke et al., 8 Jan 2025). Phase 2 upsamples each $k$5 to the final resolution and applies one or more standard $k$6-channel convolutions. Phase 3 uses a small ResNet of convolutions + ReLU, plus attention, plus up-sampling, mapping the latent feature map back to an RGB image (Ke et al., 8 Jan 2025).

Training of the VAR Transformer is standard teacher-forced cross-entropy over scales and positions: $k$7 At inference time, generation proceeds for $k$8 to $k$9: construct the input from previous scales, run one forward pass, sample all tokens at scale $h_k w_k$0 in parallel, and finally decode all scales through the VQVAE decoder to RGB. The original report characterizes the resulting complexity as only $h_k w_k$1 passes, with an example of 10 steps for $h_k w_k$2, versus $h_k w_k$3 passes in raster-scan autoregression (Tian et al., 2024).

3. Empirical performance, scaling laws, and comparison with diffusion

On ImageNet 256×256, the original VAR paper reports that the method improves an autoregressive baseline from FID 18.65 to 1.73 and IS from 80.4 to 350.2, while delivering around 20x faster inference speed (Tian et al., 2024). The detailed 256×256 comparison further lists VQGAN (AR) at FID 18.65 and IS 80.4, DiT-XL/2 at FID 2.27 and IS 278.2, VAR-d30 at FID 1.97 and IS 334.7, and VAR-d30-re at FID 1.80 and IS 356.4, with VAR requiring 10 steps rather than 250 for DiT-XL/2 (Tian et al., 2024).

The same work reports that VAR outperforms the Diffusion Transformer in image quality, inference speed, data efficiency, and scalability, and emphasizes two properties that it associates with LLMs: scaling laws and zero-shot task generalization. Scaling experiments exhibit power-law relations between performance and model size or training compute, with Pearson correlation coefficients near $h_k w_k$4 and $h_k w_k$5 in log-log space. For example,

$h_k w_k$6

and analogous relations are reported for token error rate and compute-optimal loss (Tian et al., 2024).

The survey treatment places these empirical results in a broader generative-model comparison. It notes that autoregressive models have historically traded sharper samples and stable maximum-likelihood training against slower sampling, whereas scale-wise models such as VAR materially reduce that sampling burden. In its application taxonomy, the survey lists “VAR-d30 (2.0 B params): FID=1.92, IS=323.1 on ImageNet 256” as a representative scale-wise autoregressive result, and frames scale-level factorization as a major reason why recent visual autoregressive models have become competitive with diffusion systems (Xiong et al., 2024).

Later analyses revisit why VAR can be both fast and high-fidelity. The iterative-refinement perspective argues that three design choices are central: refinement in a learned latent space, prediction as discrete classification over code indices, and partitioning by spatial frequency. In that framework, the forward process constructs a deterministic Laplacian-style latent pyramid, and the backward process reconstructs it in a small number of coarse-to-fine steps: $h_k w_k$7 This work further states that large-scale ImageNet VAR with $h_k w_k$8 scales achieves FID≈8–10 in ∼50 ms per sample on a V100, roughly 2× faster than a 250-step latent diffusion baseline at similar FID (Hong et al., 3 Oct 2025). A plausible implication is that the reported efficiency advantage is not only a consequence of fewer autoregressive stages, but also of the choice to predict discrete residual codes in latent space rather than denoise a full-resolution continuous sample.

4. Theoretical interpretation and expressive limitations

A formal complexity-theoretic analysis of VAR establishes a circuit upper bound on its expressive power. Let $h_k w_k$9 be the total token count, $R_1,\dots,R_K$0 the hidden dimension, $R_1,\dots,R_K$1 the bit precision, and $R_1,\dots,R_K$2 the number of Transformer layers. The main theorem states that there exists a $R_1,\dots,R_K$3-uniform family of depth-$R_1,\dots,R_K$4 threshold circuits $R_1,\dots,R_K$5 of size $R_1,\dots,R_K$6 such that for every input token maps $R_1,\dots,R_K$7,

$R_1,\dots,R_K$8

Equivalently, under these assumptions VAR lies in uniform $R_1,\dots,R_K$9 with polynomial size, constant depth, and polynomial precision (Ke et al., 8 Jan 2025).

The proof outline proceeds module by module. Bicubic up-interpolation, convolution, matrix multiplication in attention, exponential approximation, MLP layers, and layer normalization are each shown to admit constant-depth, polynomial-size threshold-circuit simulations. The paper summarizes these bounds as

$$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$0

with all component depths $$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$1, and concludes that composing $$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$2 layers plus convolutions and up-sampling in the VQ-VAE decoder yields

$$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$3

The authors explicitly present this as the first rigorous study to highlight limitations in the expressive power of VAR models despite their impressive performance (Ke et al., 8 Jan 2025).

The same paper draws several expressiveness consequences. Because every such VAR model can be simulated by a uniform $$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$4 circuit, it “cannot compute or approximate functions outside $$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$5,” and “increasing hidden dimension $$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$6 or width alone will not escape the constant-depth limitation.” It further states that “richer dependencies (e.g. arbitrary nesting, full hierarchical reasoning) that provably lie outside $$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$7 remain unattainable without adding depth or more powerful gates,” and proposes increasing circuit depth, introducing non-threshold nonlinearities such as gating or recurrence, or incorporating unbounded-fan-in arithmetic gates beyond majority as routes beyond the bound (Ke et al., 8 Jan 2025).

This theoretical perspective tempers a common misconception that strong empirical image-generation results imply unrestricted expressiveness. The complexity result does not contradict the practical success of VAR; rather, it identifies a precise constant-depth regime under which that success occurs. This suggests that the architectural efficiency of next-scale prediction and the formal ceiling of uniform $$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$8 are compatible descriptions of the same model class.

5. Efficiency-oriented reinterpretations and architectural variants

A substantial line of follow-up work starts from the observation that original VAR uses full-context next-scale prediction: each scale conditions on all previous scales. Markov-VAR reformulates this as a non-full-context Markov process in which each scale is treated as a Markov state, while a sliding window compresses several previous scales into a fixed-size history vector. Its central approximation is

$$p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,$$9

where $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$0 is a compressed history vector and $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$1 is a dynamic state updated by a small Transformer (Zhang et al., 28 Nov 2025). The work states that full-context attention at scale $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$2 costs

$$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$3

per attention head, with key-value cache memory $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$4 per layer, and argues that this becomes prohibitive at high resolution. On ImageNet, it reports that Markov-VAR reduces FID by 10.5% at 256×256 and decreases peak memory consumption by 83.8% at 1024×1024 relative to VAR (Zhang et al., 28 Nov 2025).

M-VAR takes a different route by decoupling scale-wise autoregression into intra-scale and inter-scale modeling. It argues that original VAR attention can be decomposed into bidirectional attention within each scale and unidirectional cross-scale attention from coarser scales into finer ones. Its empirical analysis states that intra-scale interactions account for ≈80% of total attention mass but only ≈25–30% of computation, whereas inter-scale interactions consume ≈70–75% of compute while contributing only ≈20% of attention mass. M-VAR therefore retains standard multi-head self-attention within each scale and replaces the costly inter-scale attention with Mamba, a structured state-space model, yielding $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$5 complexity instead of $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$6 for the cross-scale component (Ren et al., 2024). The largest reported model, M-VAR-d32, achieves FID 1.78 on ImageNet 256×256 and improves over prior autoregressive and diffusion baselines listed in the paper (Ren et al., 2024).

A third reinterpretation frames multi-scale autoregressive models as “Laplacian, discrete, and latent diffusion models in disguise.” In this account, the deterministic forward process constructs a Laplacian-style latent pyramid: $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$7 followed by quantization of residuals and the base latent into code indices. The backward process then predicts the base codes and residual index maps scale by scale, with each map generated in one shot by a scale-conditioned Transformer (Hong et al., 3 Oct 2025). This perspective does not change the coarse-to-fine generation rule, but it emphasizes that VAR can be interpreted as a few-step iterative-refinement process closely related to denoising diffusion, except that the forward process is deterministic and the backward process is categorical over discrete latent codes.

Taken together, these variants indicate that “VAR” denotes both a specific architecture and a broader design space. The common invariant is coarse-to-fine prediction over a hierarchy of latent maps. What changes across variants is the dependency structure among scales, the mechanism used for cross-scale communication, or the interpretation of the multiscale process itself.

6. Adaptation, reinforcement learning, and downstream applications

VAR has also been adapted beyond unconditional or class-conditional image generation. One adaptation study benchmarks non-private and differentially private fine-tuning strategies for downstream datasets such as Food-101, CUB-200, Oxford-Flowers, Cars, and Pets. It considers full-parameter fine-tuning (FFT), LoRA, and LayerNorm tuning. On non-private adaptation, the paper reports that “FFT on VAR-d20 outperforms DiffFit by ∼30–40% in FID across all five tasks,” that “LoRA on VAR matches FFT’s quality with ≲2% of parameters,” and that VAR-d20 FFT achieves FID 5.38 on Food-101 versus 10.46 for DiT-XL-2, while LoRA on VAR yields FID 6.29 on CUB-200 and 11.16 on Flowers (Shaikh et al., 15 Jul 2025). Under DP-SGD, however, the same work reports degradation: at $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$8, LoRA on Oxford-Flowers gives FID 69.92 for $$\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}$$9 augmentations and 63.24 for $$p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,$$0, and acceptable quality is said to occur only at $$p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,$$1–100 (Shaikh et al., 15 Jul 2025).

A separate reinforcement-learning line addresses what it calls “asynchronous policy conflicts” in visual autoregressive generation. Because VAR generates heterogeneous token grids $$p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,$$2 with spatial sizes $$p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,$$3, RL updates that use a single terminal reward can be dominated by later, finer stages with larger token counts. The paper formalizes an effective step-weight

$$p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,$$4

and proposes three GRPO enhancements: Value as Middle Return (VMR), Per-Action Normalization Weighting (PANW), and Mask Propagation (MP) (Sun et al., 5 Jan 2026). On text rendering, it reports that NextFlow-RL improves Word Accuracy from 0.5536 to 0.7841 and NED from 0.7816 to 0.9081 on CVTG-2K; on HPSv3 it improves the overall score from 8.43 to 10.64 (Sun et al., 5 Jan 2026).

VAR has also been extended to structured conditional generation tasks. CSD-VAR uses the scale-wise generation process for content-style decomposition, introducing a scale-aware alternating optimization strategy, an SVD-based rectification method, and augmented key-value memory. Its empirical analysis states that small scales $$p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,$$5 primarily encode style while medium scales $$p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,$$6 carry content structure, and reports that CSD-VAR variants obtain the highest content and style alignment scores among the listed multi-token methods on CSD-100 (Nguyen et al., 18 Jul 2025). Seg-VAR reformulates image segmentation as conditional autoregressive mask generation with a discrete “seglat” representation, and reports gains over Mask2Former and prior generative methods across panoptic, instance, and semantic segmentation benchmarks, including PQ 59.7 on COCO panoptic and mIoU 85.82 on Cityscapes semantic segmentation (Zheng et al., 16 Nov 2025).

In 3D-related dense prediction, “Visual Autoregressive Modelling for Monocular Depth Estimation” adapts a large-scale text-to-image VAR model using a scale-wise conditional upsampling mechanism with classifier-free guidance. The method performs inference in ten fixed autoregressive stages, requires 74K synthetic samples for fine-tuning, and reports best performance on NYU and ScanNet under the “≤ 3 M fine-tune samples & ≤ 100 M pretraining” regime, with AbsRel 6.4% and $$p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,$$7 94.8% on NYU (El-Ghoussani et al., 27 Dec 2025). These extensions suggest that coarse-to-fine latent generation can function not only as an image prior, but also as a general-purpose mechanism for conditional prediction in tasks requiring structured spatial outputs.