---
title: Vision AutoRegressive Model (VAR)
url: https://www.emergentmind.com/topics/vision-autoregressive-model-var
type: topic
---

# Vision AutoRegressive Model (VAR)

Searching arXiv for the primary VAR paper and closely related follow-up work to ground the article in current literature.
Visual AutoRegressive modeling (VAR) is a scale-level autoregressive paradigm for image generation that redefines autoregressive learning on images as coarse-to-fine “next-scale prediction” or “next-resolution prediction,” rather than raster-scan “next-token prediction.” In VAR, an image is represented as a hierarchy of discrete token maps, and the joint distribution is factorized across scales so that all tokens within a single scale are generated in parallel. This formulation preserves spatial locality while reducing the number of autoregressive steps to the number of scales, and it has been reported to make GPT-like autoregressive models surpass diffusion transformers in image generation on ImageNet 256×256, improving FID from 18.65 to 1.73 and IS from 80.4 to 350.2, with around 20x faster inference speed [2404.02905]. Within the broader taxonomy of autoregressive models in vision, VAR is a canonical instance of scale-level factorization, positioned between pixel-level and token-level approaches in its trade-off among fidelity, efficiency, and spatial coherence [2411.05902].

## 1. Probabilistic formulation and representation strategy

The defining probabilistic structure of VAR is a coarse-to-fine factorization over multi-scale token maps. If an image is encoded into a hierarchy of discrete maps \(\{r_1,r_2,\dots,r_K\}\), where each \(r_k\in[V]^{h_k\times w_k}\), then VAR models the joint distribution as
\[
p(r_1,\dots,r_K)\;=\;\prod_{k=1}^K p\bigl(r_k\mid r_{<k}\bigr)\,.
\]
Crucially, within a single scale \(k\), all \(h_k w_k\) tokens are generated in parallel, so the autoregressive dependence is imposed across scales rather than across all spatial locations [2404.02905].

This places VAR in the “scale-level” family of visual autoregressive models. The survey literature distinguishes pixel-level, token-level, and scale-level factorization strategies, and characterizes scale-level models as balancing efficiency and spatial locality by generating coarse maps before finer details. In that account, VAR’s multi-scale token maps \(R_1,\dots,R_K\) are generated as
\[
p(R_{1:K}) = p(R_1)\prod_{k=2}^K p(R_k \mid R_1\dots R_{k-1})\,,
\]
with tokens inside each scale generated in parallel under a block-wise causal mask [2411.05902].

Several later works restate the same idea in closely related notation. M-VAR describes an image as a hierarchy \(\mathbf{x}=\{\mathbf{x}^1,\mathbf{x}^2,\dots,\mathbf{x}^S\}\) and writes
\[
p(\mathbf{x}) \;=\; \prod_{s=1}^S p\bigl(\mathbf{x}^s\mid \mathbf{x}^{<s}\bigr)\,,
\]
while emphasizing that original VAR implements each conditional with a Transformer attending over the entire flattened sequence \([\mathbf{x}^1;\dots;\mathbf{x}^{s-1};\mathbf{x}^s]\) [2411.10433]. CSD-VAR expresses the same factorization for VQ-VAE token maps \(s_1,\dots,s_K\), with optional conditioning \(c\),
\[
p(s_{1},s_{2},\dots,s_{K}) \;=\; \prod_{k=1}^{K}p_{\theta}\bigl(s_{k}\mid s_{1},s_{2},\dots,s_{k-1};\,c\bigr),
\]
and further notes that preceding maps are upsampled by interpolation to keep the number of tokens fixed per scale [2507.13984].

A distinct but related formulation appears in adaptation work that presents VAR through standard autoregressive likelihood over VQ-GAN codes. There, an image \(x\in\mathbb{R}^{H\times W\times 3}\) is tokenized into \(N\) discrete codes \(x=(x_1,\dots,x_N)\), and the model writes
\[
P(x) = \prod_{i=1}^N P(x_i \mid x_{<i}),
\qquad
\mathcal{L}_{AR}(\theta)= -\mathbb{E}_{x\sim\mathcal{D}}[\ell(x;\theta)]\,,
\]
while noting that VAR groups tokens by “scale” levels and attaches both 2D positional embeddings and level embeddings [2507.11441]. This suggests that the scale-level factorization can be viewed as a structured autoregressive decomposition layered atop a conventional likelihood objective.

## 2. Core architecture and generation pipeline

The original VAR architecture combines a multi-scale tokenizer, a decoder-only Transformer backbone, and a reconstruction path back to RGB. The tokenizer is a multi-scale VQVAE: a CNN encoder \(\mathcal{E}\) produces a feature map \(f\), which is downsampled to each scale, quantized by a shared codebook \(Z\), decoded back to the original latent resolution, and subtracted through a small convolution \(\phi_k\). This residual construction ensures that each scale depends only on earlier scales. Training of the tokenizer uses
\[
\mathcal{L}_{\rm VQVAE}
=\|im-\hat{im}\|_2
+\|f-\hat f\|_2
+\lambda_P\,\mathcal{L}_P(\hat{im})
+\lambda_G\,\mathcal{L}_G(\hat{im}),
\]
where \(\mathcal{L}_P\) is a perceptual loss and \(\mathcal{L}_G\) a GAN-style discriminator loss [2404.02905].

The Transformer itself follows a GPT-2–style decoder-only stack. In the class-conditional setting, it uses Adaptive LayerNorm conditioned on a class embedding prepended as a special token; it uses absolute position embeddings, and the reported implementation “does not” incorporate RoPE or SwiGLU. At generation step \(k\), the input sequence consists of the coarser scales \(r_1,\dots,r_{k-1}\) plus blank placeholders for scale \(k\), and a single forward pass produces logits for all \(h_k w_k\) positions simultaneously. No specialized U-Net or cross-attention modules are required; the model is described as “self-attended in one monolithic transformer” [2404.02905].

The more formal architectural write-up from the circuit-complexity study decomposes VAR into three phases. In Phase 1, next-scale prediction processes token maps \(M_1,\dots,M_k\) through a stack of \(m=O(1)\) Transformer layers of the form
\[
X\mapsto g_i\bigl(\mathsf{Attn}_i(\phi_{\rm up}(X))\bigr),
\]
where \(\phi_{\rm up}\) is bicubic up-interpolation, \(\mathsf{Attn}\) is a standard self-attention block,
\[
A_{i,j}=\exp\bigl(X_{i,*}W_QW_K^\top X_{j,*}^\top\bigr),\quad
\mathsf{Attn}(X)=D^{-1}AXW_V,\;D=\diag(A\mathbf1),
\]
and \(g_i\) consists of an MLP and a layer normalization [2501.04299]. Phase 2 upsamples each \(M_k\) to the final resolution and applies one or more standard \(c\)-channel convolutions. Phase 3 uses a small ResNet of convolutions + ReLU, plus attention, plus up-sampling, mapping the latent feature map back to an RGB image [2501.04299].

Training of the VAR Transformer is standard teacher-forced cross-entropy over scales and positions:
\[
\mathcal{L}_{\rm VAR}(\theta)
=
-\sum_{k=1}^K
\sum_{i=1}^{h_k}\sum_{j=1}^{w_k}
\log p_\theta\!\bigl(r_k^{(i,j)} \mid r_{<k}\bigr)\,.
\]
At inference time, generation proceeds for \(k=1\) to \(K\): construct the input from previous scales, run one forward pass, sample all tokens at scale \(k\) in parallel, and finally decode all scales through the VQVAE decoder to RGB. The original report characterizes the resulting complexity as only \(K=\mathcal{O}(\log n)\) passes, with an example of 10 steps for \(256^2\), versus \(n^2\) passes in raster-scan autoregression [2404.02905].

## 3. Empirical performance, scaling laws, and comparison with diffusion

On ImageNet 256×256, the original VAR paper reports that the method improves an autoregressive baseline from FID 18.65 to 1.73 and IS from 80.4 to 350.2, while delivering around 20x faster inference speed [2404.02905]. The detailed 256×256 comparison further lists VQGAN (AR) at FID 18.65 and IS 80.4, DiT-XL/2 at FID 2.27 and IS 278.2, VAR-d30 at FID 1.97 and IS 334.7, and VAR-d30-re at FID 1.80 and IS 356.4, with VAR requiring 10 steps rather than 250 for DiT-XL/2 [2404.02905].

The same work reports that VAR outperforms the Diffusion Transformer in image quality, inference speed, data efficiency, and scalability, and emphasizes two properties that it associates with large language models: scaling laws and zero-shot task generalization. Scaling experiments exhibit power-law relations between performance and model size or training compute, with Pearson correlation coefficients near \(-0.998\) and \(-0.99\) in log-log space. For example,
\[
L_{\rm last}\;=\;(2.0\;N)^{-0.23},\qquad
L_{\rm avg}\;=\;(2.5\;N)^{-0.20},
\]
and analogous relations are reported for token error rate and compute-optimal loss [2404.02905].

The survey treatment places these empirical results in a broader generative-model comparison. It notes that autoregressive models have historically traded sharper samples and stable maximum-likelihood training against slower sampling, whereas scale-wise models such as VAR materially reduce that sampling burden. In its application taxonomy, the survey lists “VAR-d30 (2.0 B params): FID=1.92, IS=323.1 on ImageNet 256” as a representative scale-wise autoregressive result, and frames scale-level factorization as a major reason why recent visual autoregressive models have become competitive with diffusion systems [2411.05902].

Later analyses revisit why VAR can be both fast and high-fidelity. The iterative-refinement perspective argues that three design choices are central: refinement in a learned latent space, prediction as discrete classification over code indices, and partitioning by spatial frequency. In that framework, the forward process constructs a deterministic Laplacian-style latent pyramid, and the backward process reconstructs it in a small number of coarse-to-fine steps:
\[
p_\theta(b^{(S)},r^{(0:S-1)}) = p_\theta(b^{(S)}) \prod_{k=1}^S p_\theta(r^{(k-1)}|L^{(k)}).
\]
This work further states that large-scale ImageNet VAR with \(S=8\) scales achieves FID≈8–10 in ∼50 ms per sample on a V100, roughly 2× faster than a 250-step latent diffusion baseline at similar FID [2510.02826]. A plausible implication is that the reported efficiency advantage is not only a consequence of fewer autoregressive stages, but also of the choice to predict discrete residual codes in latent space rather than denoise a full-resolution continuous sample.

## 4. Theoretical interpretation and expressive limitations

A formal complexity-theoretic analysis of VAR establishes a circuit upper bound on its expressive power. Let \(n\) be the total token count, \(d\le O(n)\) the hidden dimension, \(p\le \mathrm{poly}(n)\) the bit precision, and \(m=O(1)\) the number of Transformer layers. The main theorem states that there exists a \(\mathrm{DLOGTIME}\)-uniform family of depth-\(O(1)\) threshold circuits \(\{C_n\}\) of size \(S(n)=\mathrm{poly}(n)\) such that for every input token maps \(X\in\mathbb{F}_p^{n\times d}\),
\[
\bigl|\mathrm{VAR}(X)-C_n(X)\bigr|\le 2^{-p}
\quad\text{(coordinate-wise)}.
\]
Equivalently, under these assumptions VAR lies in uniform \(\mathsf{TC}^0\) with polynomial size, constant depth, and polynomial precision [2501.04299].

The proof outline proceeds module by module. Bicubic up-interpolation, convolution, matrix multiplication in attention, exponential approximation, MLP layers, and layer normalization are each shown to admit constant-depth, polynomial-size threshold-circuit simulations. The paper summarizes these bounds as
\[
d_{\rm up}=2d_{\rm std}+2d_\oplus,\quad
d_{\rm attn}=3(d_{\rm std}+d_\oplus)+d_{\rm exp},\quad
d_{\rm mlp}=2d_{\rm std}+d_\oplus,\quad
d_{\rm ln}=5d_{\rm std}+2d_\oplus+d_{\rm sqrt},
\]
with all component depths \(O(1)\), and concludes that composing \(m=O(1)\) layers plus convolutions and up-sampling in the VQ-VAE decoder yields
\[
\Delta=O(1),\qquad S(n)=\mathrm{poly}(n),\qquad p\le \mathrm{poly}(n),\qquad d\le O(n).
\]
The authors explicitly present this as the first rigorous study to highlight limitations in the expressive power of VAR models despite their impressive performance [2501.04299].

The same paper draws several expressiveness consequences. Because every such VAR model can be simulated by a uniform \(\mathsf{TC}^0\) circuit, it “cannot compute or approximate functions outside \(\mathsf{TC}^0\),” and “increasing hidden dimension \(d=O(n)\) or width alone will not escape the constant-depth limitation.” It further states that “richer dependencies (e.g. arbitrary nesting, full hierarchical reasoning) that provably lie outside \(\mathsf{TC}^0\) remain unattainable without adding depth or more powerful gates,” and proposes increasing circuit depth, introducing non-threshold nonlinearities such as gating or recurrence, or incorporating unbounded-fan-in arithmetic gates beyond majority as routes beyond the bound [2501.04299].

This theoretical perspective tempers a common misconception that strong empirical image-generation results imply unrestricted expressiveness. The complexity result does not contradict the practical success of VAR; rather, it identifies a precise constant-depth regime under which that success occurs. This suggests that the architectural efficiency of next-scale prediction and the formal ceiling of uniform \(\mathsf{TC}^0\) are compatible descriptions of the same model class.

## 5. Efficiency-oriented reinterpretations and architectural variants

A substantial line of follow-up work starts from the observation that original VAR uses full-context next-scale prediction: each scale conditions on all previous scales. Markov-VAR reformulates this as a non-full-context Markov process in which each scale is treated as a Markov state, while a sliding window compresses several previous scales into a fixed-size history vector. Its central approximation is
\[
p(R_t\mid R_{<t}) \approx p\bigl(R_t\mid s_{t-1},h_{t-1}\bigr),
\]
where \(h_{t-1}=\mathcal C(X_{t-w:t-1})\) is a compressed history vector and \(s_{t-1}\) is a dynamic state updated by a small Transformer [2511.23334]. The work states that full-context attention at scale \(t\) costs
\[
\mathcal O\Bigl(\bigl(\sum_{i< t} n^i\bigr)\times n^t\Bigr)
\]
per attention head, with key-value cache memory \(\mathcal O(\sum_{i<t} n^i)\) per layer, and argues that this becomes prohibitive at high resolution. On ImageNet, it reports that Markov-VAR reduces FID by 10.5% at 256×256 and decreases peak memory consumption by 83.8% at 1024×1024 relative to VAR [2511.23334].

M-VAR takes a different route by decoupling scale-wise autoregression into intra-scale and inter-scale modeling. It argues that original VAR attention can be decomposed into bidirectional attention within each scale and unidirectional cross-scale attention from coarser scales into finer ones. Its empirical analysis states that intra-scale interactions account for ≈80% of total attention mass but only ≈25–30% of computation, whereas inter-scale interactions consume ≈70–75% of compute while contributing only ≈20% of attention mass. M-VAR therefore retains standard multi-head self-attention within each scale and replaces the costly inter-scale attention with Mamba, a structured state-space model, yielding \(O(Nd^2)\) complexity instead of \(O(N^2d)\) for the cross-scale component [2411.10433]. The largest reported model, M-VAR-d32, achieves FID 1.78 on ImageNet 256×256 and improves over prior autoregressive and diffusion baselines listed in the paper [2411.10433].

A third reinterpretation frames multi-scale autoregressive models as “Laplacian, discrete, and latent diffusion models in disguise.” In this account, the deterministic forward process constructs a Laplacian-style latent pyramid:
\[
L^{(k)}=\mathcal S_k(L^{(k-1)}),\qquad
R^{(k-1)} = L^{(k-1)} - \mathcal U_k(L^{(k)}),
\]
followed by quantization of residuals and the base latent into code indices. The backward process then predicts the base codes and residual index maps scale by scale, with each map generated in one shot by a scale-conditioned Transformer [2510.02826]. This perspective does not change the coarse-to-fine generation rule, but it emphasizes that VAR can be interpreted as a few-step iterative-refinement process closely related to denoising diffusion, except that the forward process is deterministic and the backward process is categorical over discrete latent codes.

Taken together, these variants indicate that “VAR” denotes both a specific architecture and a broader design space. The common invariant is coarse-to-fine prediction over a hierarchy of latent maps. What changes across variants is the dependency structure among scales, the mechanism used for cross-scale communication, or the interpretation of the multiscale process itself.

## 6. Adaptation, reinforcement learning, and downstream applications

VAR has also been adapted beyond unconditional or class-conditional image generation. One adaptation study benchmarks non-private and differentially private fine-tuning strategies for downstream datasets such as Food-101, CUB-200, Oxford-Flowers, Cars, and Pets. It considers full-parameter fine-tuning (FFT), LoRA, and LayerNorm tuning. On non-private adaptation, the paper reports that “FFT on VAR-d20 outperforms DiffFit by ∼30–40% in FID across all five tasks,” that “LoRA on VAR matches FFT’s quality with ≲2% of parameters,” and that VAR-d20 FFT achieves FID 5.38 on Food-101 versus 10.46 for DiT-XL-2, while LoRA on VAR yields FID 6.29 on CUB-200 and 11.16 on Flowers [2507.11441]. Under DP-SGD, however, the same work reports degradation: at \(\epsilon=10\), LoRA on Oxford-Flowers gives FID 69.92 for \(k=1\) augmentations and 63.24 for \(k=128\), and acceptable quality is said to occur only at \(\epsilon\ge 50\)–100 [2507.11441].

A separate reinforcement-learning line addresses what it calls “asynchronous policy conflicts” in visual autoregressive generation. Because VAR generates heterogeneous token grids \(\mathbf r_1,\mathbf r_2,\dots,\mathbf r_T\) with spatial sizes \(h_t\times w_t\), RL updates that use a single terminal reward can be dominated by later, finer stages with larger token counts. The paper formalizes an effective step-weight
\[
\omega_t \propto \mathrm{Var}\bigl[\nabla_\theta \mathrm{KL}(\pi_{\rm old}\|\pi_\theta)\bigr] \approx h_{t+1}w_{t+1},
\]
and proposes three GRPO enhancements: Value as Middle Return (VMR), Per-Action Normalization Weighting (PANW), and Mask Propagation (MP) [2601.02256]. On text rendering, it reports that NextFlow-RL improves Word Accuracy from 0.5536 to 0.7841 and NED from 0.7816 to 0.9081 on CVTG-2K; on HPSv3 it improves the overall score from 8.43 to 10.64 [2601.02256].

VAR has also been extended to structured conditional generation tasks. CSD-VAR uses the scale-wise generation process for content-style decomposition, introducing a scale-aware alternating optimization strategy, an SVD-based rectification method, and augmented key-value memory. Its empirical analysis states that small scales \(\{1,2,3,10\}\) primarily encode style while medium scales \(\{4,\dots,9\}\) carry content structure, and reports that CSD-VAR variants obtain the highest content and style alignment scores among the listed multi-token methods on CSD-100 [2507.13984]. Seg-VAR reformulates image segmentation as conditional autoregressive mask generation with a discrete “seglat” representation, and reports gains over Mask2Former and prior generative methods across panoptic, instance, and semantic segmentation benchmarks, including PQ 59.7 on COCO panoptic and mIoU 85.82 on Cityscapes semantic segmentation [2511.12594].

In 3D-related dense prediction, “Visual Autoregressive Modelling for Monocular Depth Estimation” adapts a large-scale text-to-image VAR model using a scale-wise conditional upsampling mechanism with classifier-free guidance. The method performs inference in ten fixed autoregressive stages, requires 74K synthetic samples for fine-tuning, and reports best performance on NYU and ScanNet under the “≤ 3 M fine-tune samples & ≤ 100 M pretraining” regime, with AbsRel 6.4% and \(\delta_1\) 94.8% on NYU [2512.22653]. These extensions suggest that coarse-to-fine latent generation can function not only as an image prior, but also as a general-purpose mechanism for conditional prediction in tasks requiring structured spatial outputs.

Source: https://www.emergentmind.com/topics/vision-autoregressive-model-var