---
title: Visual Autoregressive Models (VAR)
url: https://www.emergentmind.com/topics/visual-autoregressive-models-var
type: topic
---

# Visual Autoregressive Models (VAR)

Visual Autoregressive Models (VAR) are autoregressive image generators that replace raster-scan next-token prediction with coarse-to-fine **next-scale prediction**. Instead of emitting one visual token at a time in a flattened sequence, a VAR model predicts an entire token map at each resolution scale, conditioning each finer scale on all previously generated coarser scales. This yields a hierarchical discrete generative process over multi-scale latent representations, typically produced by a VQ-VAE-style tokenizer, and has been positioned as a scalable alternative to both raster autoregression and diffusion-based sampling for image synthesis [2404.02905].

## 1. Core generative formulation

The defining factorization in VAR is scale-wise rather than token-wise. A standard formulation writes
$$
p(r_1,r_2,\dots,r_K)=\prod_{k=1}^{K} p(r_k \mid r_1,\dots,r_{k-1}),
$$
where each $r_k$ is a 2D token map at scale $k$, with the coarsest map generated first and the finest latent map decoded last [2404.02905]. Closely related formulations describe the latent representation as multi-scale residual token maps $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$, with progressive reconstruction
$$
\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),
$$
so that autoregression occurs over scales while positions within a scale are predicted in parallel [2512.16483].

The original VAR tokenizer is a modified VQGAN-style autoencoder with multiscale quantization. An image is encoded into a continuous feature map $f=\mathcal E(im)$, quantized at each scale as $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$, and reconstructed through residual accumulation with a shared codebook of size $V=4096$ and latent dimension $32$ [2404.02905]. Later papers preserve the same basic hierarchy but express it through residual maps, binary spherical quantization, or other tokenizer variants, while retaining the coarse-to-fine autoregressive dependency structure [2509.22400].

Architecturally, the baseline VAR generator is a decoder-only transformer, GPT-2-like in style, using Adaptive LayerNorm (AdaLN) rather than standard LayerNorm; for class-conditional synthesis, the class embedding is used as a start token and also injected into AdaLN conditioning [2404.02905]. The resulting system has far fewer autoregressive steps than raster-scan image AR, because the number of generation stages is the number of scales rather than the number of image tokens.

## 2. Computational and theoretical characterizations

A central motivation for VAR is computational. The original formulation argues that raster-scan AR over an $n\times n$ latent map requires $\mathcal O(n^2)$ decoding steps and $\mathcal O(n^6)$ total compute, whereas next-scale prediction reduces the number of steps to $\mathcal O(\log n)$ and the total compute to $\mathcal O(n^4)$ [2404.02905]. Empirically, the paper reports that VAR makes GPT-like AR models surpass diffusion transformers in class-conditional ImageNet generation, with a reported improvement from FID $18.65$ to $1.73$ in the abstract, and a detailed entry for VAR-\(d30\)-re at FID $1.80$, IS $356.4$, 2.0B parameters, and 10 sampling steps on ImageNet $256\times256$ [2404.02905].

This computational picture has also been formalized through fine-grained complexity theory. A complexity analysis proves that, assuming SETH, when $d=O(\log n)$ and the input-entry bound satisfies $R=\Theta(\sqrt{\log n})$, there is no algorithm that approximates the VAR model up to $1/\mathrm{poly}(n)$ additive error in $O(n^{4-\Omega(1)})$ time; when $R=o(\sqrt{\log n})$, the paper gives an almost quadratic-time approximation algorithm with runtime $O(n^{2+o(1)})$ [2501.04377]. This establishes a threshold phenomenon in the norm bound $R$ for fast approximate VAR computation.

The expressivity of the architecture has likewise been studied directly. A universality result shows that even a deliberately minimal VAR transformer—single-head attention, a single self-attention layer, a single interpolation layer, and a simple feed-forward layer—is universal for broad classes of image-to-image Lipschitz mappings [2502.06167]. In the paper’s notation, the simplified VAR block is
$$
\mathsf{TF}_{\mathrm{var}}=\mathsf{FFN}\circ\mathsf{Attn}\circ \phi_{\rm up},
$$
and compositions of such blocks already suffice for universality.

A complementary line of work reinterprets VAR through iterative refinement rather than purely autoregressive factorization. In that view, VAR performs deterministic decomposition into a latent Laplacian pyramid in the forward direction and learned coarse-to-fine reconstruction in the backward direction. The formulation introduces latent levels $L^{(k)}$, residuals $R^{(k-1)}=L^{(k-1)}-\mathcal U_k(L^{(k)})$, and discrete residual codes sampled by a transformer-conditioned backward process [2510.02826]. That paper isolates three design choices behind VAR’s efficiency and fidelity: refinement in a learned latent space, prediction as discrete classification over code indices, and partitioning the task by spatial frequency. The original VAR work additionally reports LLM-like empirical scaling laws, with linear correlation coefficients near $-0.998$ for size scaling and near $-0.99$ for compute scaling [2404.02905].

## 3. Error dynamics, diversity, and acceleration

Although VAR reduces decoding depth relative to token-level AR, the hierarchical dependency across scales introduces its own failure modes. AID-VAR formulates this explicitly as **cascading error propagation**: coarse-scale mistakes are repeatedly conditioned upon, amplified by upsampling and subsequent decoding, and finally appear as structural distortions, texture drift, and semantic artifacts. Under a Lipschitz propagation assumption, the appendix derives
$$
\mathbb{E}\left[\|\boldsymbol{\Delta}_K\|^2\right] \ge \sigma^2 \cdot \frac{(1+L^2)^K - 1}{L^2},
$$
framing VAR failure as error accumulation driven by discrete quantization, autoregressive dependence, and spatial upsampling amplification [2605.24843]. The proposed fix is a frozen-backbone adversarial correction scheme: a lightweight guidance injector $P_\phi$ perturbs the internal state at each scale, while a projected discriminator in RGB space diagnoses fidelity gaps. With soft-label VQ-VAE decoding for differentiability and a dynamic refresh strategy, the method improves FID from $3.55$ to $3.24$ on VAR-d16, from $2.95$ to $2.54$ on VAR-d20, and from $2.33$ to $2.08$ on VAR-d24; the paper highlights a 16% FID improvement for AID-VAR-d20 with only a 3% parameter increase [2605.24843].

A separate limitation is **diversity collapse**. DiverseVAR argues that diversity in VAR is formed mainly at early scales, while later scales largely stabilize an already committed structure. It introduces a distinction between pivotal and auxiliary components in early-scale feature maps, then applies Soft-Suppression Regularization (SSR) to dominant singular directions before prediction and Soft-Amplification Regularization (SAR) to the output afterward [2511.17074]. On Infinity-2B at $1024\times1024$, the method improves COCO 2014 Recall from $0.316$ to $0.385$, Coverage from $0.651$ to $0.690$, and FID from $28.48$ to $22.96$, while CLIPScore remains at $0.313$; GenEval and DPG remain close to the vanilla model [2511.17074]. This localizes diversity control to early-scale block computations rather than final-scale decoding.

Acceleration research has converged on a related empirical observation: early scales establish semantics and structure, while later scales mostly refine fidelity. StageVAR formalizes three stages—semantic establishment, structure establishment, and fidelity refinement—and exploits two late-stage properties: semantic irrelevance of text conditioning and low-rank feature structure [2512.16483]. Its training-free pipeline turns off CFG in late stages, projects features into a reduced space with random projection, and restores full features with Representative Token Restoration, yielding up to $3.4\times$ speedup with only a $0.01$ drop on GenEval and a $0.26$ decrease on DPG [2512.16483]. StepVAR reaches a similar conclusion from a pruning perspective, but preserves both texture and structure by fusing a high-pass textural score with a PCA-based structural score and then reconstructing dense features with nearest-neighbor propagation; the paper notes that the final high-resolution stages account for more than 75% of inference time, and reports $2.0\times$ speedup on Infinity with latency reduced from $3.27$s to $1.60$s while keeping MJHQ-30K FID at $9.85$ versus $10.09$ for the original model [2603.01757].

## 4. Controlled generation and downstream vision tasks

The scale-wise hierarchy of VAR has made it a natural substrate for controlled generation. The original VAR paper already demonstrated zero-shot in-painting, out-painting, and class-conditional editing by teacher forcing or masking selected regions in latent space, without task-specific retraining [2404.02905]. Later work turned these qualitative capabilities into explicit frameworks.

CSD-VAR adapts VAR to **content-style decomposition** from a single image. Its main claim is that early scales tend to encode style-like properties such as global color, texture, and coarse appearance, whereas later scales carry more content-specific structure such as object shape and fine details. On that basis, it introduces scale-aware alternating optimization, SVD-based rectification of the style embedding to reduce content leakage, and augmented key-value memory to preserve content identity [2507.13984]. The accompanying CSD-100 benchmark contains about 100 images with 63 distinct objects and 53 unique styles, and the reported CSD-VAR variants on Switti and Infinity achieve the best or near-best content and style alignment metrics in the provided comparison [2507.13984].

SAVAREdit addresses **text-guided image editing** with VAR backbones, particularly Infinity-2B. It replaces earlier token-reassembly heuristics with Coarse-to-Fine Token Localization for editable-mask refinement, feature injection from cached source features, and reinforcement-learning-based Adaptive Feature Injection that learns scale- and layer-specific injection ratios [2603.28367]. The paper’s feature analysis identifies the 0-th layer and scales 5–8 as having the highest structural relevance for injection. On PIE-Bench, SAVAREdit reports Structure Distance $0.0225$, PSNR $25.73$, SSIM $0.8521$, and LPIPS $0.0636$, improving over AREdit on structure preservation while remaining competitive on CLIP similarity [2603.28367].

Seg-VAR extends the paradigm beyond image generation to **semantic, instance, and panoptic segmentation** by reframing segmentation as conditional autoregressive mask generation. It introduces an image encoder for the prior $p_\psi(z\mid x)$, a spatial-aware seglat encoder for the posterior $q_\phi(z\mid c)$, and a decoder $p_\theta(c\mid z)$, trained in three stages: seglat learning, latent learning, and prior alignment [2511.12594]. The method uses a location-sensitive color mapping to distinguish instances in seglat space and reports, with Swin-L backbones, 59.7 PQ on COCO panoptic segmentation, 52.7 AP on COCO instance segmentation, 85.82 mIoU on Cityscapes semantic segmentation, and 54.90 mIoU on ADE20K, all above the listed Mask2Former baselines [2511.12594]. This suggests that the coarse-to-fine latent hierarchy can also be used for spatial perception tasks requiring low-level precision.

## 5. Safety, personalization, and alignment

Safety-oriented extensions of VAR have focused first on provenance and erasure. Safe-VAR is presented as the first watermarking framework specifically designed for autoregressive text-to-image generation, motivated by the observation that the timing of watermark injection strongly affects quality and that different watermark complexities prefer different scales [2503.11324]. Its Adaptive Scale Interaction Module selects compatible image–watermark scale pairs, while Cross-Scale Fusion combines them through mixture of heads and mixture of experts. At 1024 resolution on LAION-Aesthetics, the paper reports PSNR $32.526$, SSIM $0.918$, and LPIPS $0.104$ for cover-versus-watermarked images, together with strong robustness under crop, noise, JPEG, and combined perturbations; it also reports zero-shot generalization to QR Codes [2503.11324].

For concept removal, S-VARE argues that diffusion-style erasure objectives are structurally mismatched to VAR because VAR predicts discrete next-scale tokens rather than continuous noise and is strongly autoregressive across scales. The proposed VARE framework stabilizes erasure by conditioning optimization on auxiliary visual tokens from the teacher model, and S-VARE adds a filtered cross-entropy erasure loss plus a preservation KL loss to avoid language drift and quality collapse [2509.22400]. The paper’s headline claim is that the method erases about 97% of sensitive concepts with less than 2% CLIP degradation, while attack success rates drop from 4.1% to 0.8% on I2P, from 21.7% to 3.5% on MMA, and from 75.9% to 7.4% on Ring-A-Bell [2509.22400].

Personalization introduces a different tension between plasticity and retention. CPC-VAR presents the first systematic study of continual personalized generation in VAR, targeting both catastrophic forgetting in sequential concept learning and feature entanglement in multi-concept synthesis [2605.19750]. For continual learning it uses Gradient-based Concept Neuron Selection (GCNS), which identifies concept-relevant neurons in cross-attention blocks and regularizes only the overlap with previously important parameters. For multi-concept synthesis it uses context-aware multi-branch feature modeling and localized cross-attention fusion guided by bounding boxes. On its eight-task continual benchmark, the method reports average DINO $69.35$, average CLIP-I $83.76$, and average CLIP-T $30.18$, above the listed baselines [2605.19750].

Alignment under reinforcement learning exposes another VAR-specific difficulty: heterogeneous token-grid actions across timesteps. “VAR RL Done Right” describes this as **asynchronous policy conflicts** and modifies GRPO with a stabilizing middle return, per-action normalization weighting, and mask propagation inspired by Reward Feedback Learning [2601.02256]. On text rendering, NextFlow-RL improves Word Accuracy from $0.5536$ to $0.7841$, NED from $0.7816$ to $0.9081$, and CLIPScore from $0.8068$ to $0.8224$; on HPS refinement, the overall “All” score rises from $8.43$ to $10.64$ [2601.02256]. The broader implication is that scale-heterogeneous autoregression changes the geometry of policy optimization and requires timestep-aware credit assignment.

## 6. Architectural variants and deployment

Recent work has also modified VAR at the systems level, either by changing the backbone or by improving deployment efficiency. VIAR replaces the explicit deep stack per scale with an implicit equilibrium layer surrounded by shallow pre- and post-blocks, while retaining next-scale autoregression [2605.01220]. The equilibrium state $z_k^*$ satisfies
$$
z_k^* = f_{\text{imp}}(z_k^*, I_k, c; \theta_{\text{imp}}),
$$
and is trained with stochastic Jacobian-Free Backpropagation, giving constant training memory for the implicit core and an inference-time per-scale iteration knob. On ImageNet $256\times256$, VIAR reports FID $2.16$ and sFID $8.07$ with 770.9M parameters, which is 38.4% of the parameters of VAR-D30; by changing the iteration schedule, peak memory drops from 19.24 GB to 8.53 GB and throughput rises from 15.16 to 32.08 images/s on a single RTX 4090, without retraining [2605.01220]. This preserves the next-scale interface while making compute allocation adjustable across scales.

Post-training quantization has required VAR-specific adaptations as well. Shift-and-Sum Quantization identifies two PTQ failure modes: large reconstruction errors in attention-value products, especially at coarse scales where attention is more concentrated, and mismatch between codebook-entry sampling frequencies and predicted probabilities under limited calibration data [2606.16131]. To address them, it applies a shift-and-sum quantization kernel to attentive value tokens and a calibration-data resampling scheme that aligns empirical token frequencies with predicted probabilities. On ImageNet class-conditional generation, the method improves over BRECQ across VAR-d16, d20, d24, and d30; for example, on VAR-d24 at 4/6-bit quantization, BRECQ gives IS $215.8$ and FID $4.89$, whereas the proposed method gives IS $230.7$ and FID $3.96$ [2606.16131]. The paper further reports only about 2% lower CPU throughput than BRECQ and much faster execution than LiteVAR, indicating that PTQ for VAR is becoming a deployable systems problem rather than only an accuracy problem.

Taken together, these lines of work portray VAR less as a single model family than as a general coarse-to-fine discrete generation interface. The common invariant is next-scale prediction over hierarchical latent token maps; the main points of variation are how scales are parameterized, corrected, compressed, aligned, or repurposed for tasks beyond unconditional synthesis. This suggests that the defining contribution of VAR is not merely faster sampling than raster AR, but an alternative organization of visual autoregression around scale, hierarchy, and parallelism.

Source: https://www.emergentmind.com/topics/visual-autoregressive-models-var