Visual Autoregressive Models (VAR)

• VAR is an autoregressive image synthesis technique that generates multi-scale token maps hierarchically, replacing token-wise raster scanning.
• It significantly reduces decoding steps and overall compute by predicting entire token maps per scale instead of one token at a time.
• VAR’s design enables advanced tasks like image editing, segmentation, and watermarking through scale-conditioned refinement and controlled generation.

Visual Autoregressive Models (VAR) are autoregressive image generators that replace raster-scan next-token prediction with coarse-to-fine next-scale prediction. Instead of emitting one visual token at a time in a flattened sequence, a VAR model predicts an entire token map at each resolution scale, conditioning each finer scale on all previously generated coarser scales. This yields a hierarchical discrete generative process over multi-scale latent representations, typically produced by a VQ-VAE-style tokenizer, and has been positioned as a scalable alternative to both raster autoregression and diffusion-based sampling for image synthesis (Tian et al., 2024).

1. Core generative formulation

The defining factorization in VAR is scale-wise rather than token-wise. A standard formulation writes

$$p(r_1,r_2,\dots,r_K)=\prod_{k=1}^{K} p(r_k \mid r_1,\dots,r_{k-1}),$$

where each $r_k$ is a 2D token map at scale $k$, with the coarsest map generated first and the finest latent map decoded last (Tian et al., 2024). Closely related formulations describe the latent representation as multi-scale residual token maps $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$, with progressive reconstruction

$$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$

so that autoregression occurs over scales while positions within a scale are predicted in parallel (Li et al., 18 Dec 2025).

The original VAR tokenizer is a modified VQGAN-style autoencoder with multiscale quantization. An image is encoded into a continuous feature map $f=\mathcal E(im)$, quantized at each scale as $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$, and reconstructed through residual accumulation with a shared codebook of size $V=4096$ and latent dimension $32$ (Tian et al., 2024). Later papers preserve the same basic hierarchy but express it through residual maps, binary spherical quantization, or other tokenizer variants, while retaining the coarse-to-fine autoregressive dependency structure (Zhong et al., 26 Sep 2025).

Architecturally, the baseline VAR generator is a decoder-only transformer, GPT-2-like in style, using Adaptive LayerNorm (AdaLN) rather than standard LayerNorm; for class-conditional synthesis, the class embedding is used as a start token and also injected into AdaLN conditioning (Tian et al., 2024). The resulting system has far fewer autoregressive steps than raster-scan image AR, because the number of generation stages is the number of scales rather than the number of image tokens.

2. Computational and theoretical characterizations

A central motivation for VAR is computational. The original formulation argues that raster-scan AR over an $n\times n$ latent map requires $r_k$0 decoding steps and $r_k$1 total compute, whereas next-scale prediction reduces the number of steps to $r_k$2 and the total compute to $r_k$3 (Tian et al., 2024). Empirically, the paper reports that VAR makes GPT-like AR models surpass diffusion transformers in class-conditional ImageNet generation, with a reported improvement from FID $r_k$4 to $r_k$5 in the abstract, and a detailed entry for VAR-$r_k$6-re at FID $r_k$7, IS $r_k$8, 2.0B parameters, and 10 sampling steps on ImageNet $r_k$9 (Tian et al., 2024).

This computational picture has also been formalized through fine-grained complexity theory. A complexity analysis proves that, assuming SETH, when $k$0 and the input-entry bound satisfies $k$1, there is no algorithm that approximates the VAR model up to $k$2 additive error in $k$3 time; when $k$4, the paper gives an almost quadratic-time approximation algorithm with runtime $k$5 (Ke et al., 8 Jan 2025). This establishes a threshold phenomenon in the norm bound $k$6 for fast approximate VAR computation.

The expressivity of the architecture has likewise been studied directly. A universality result shows that even a deliberately minimal VAR transformer—single-head attention, a single self-attention layer, a single interpolation layer, and a simple feed-forward layer—is universal for broad classes of image-to-image Lipschitz mappings (Chen et al., 10 Feb 2025). In the paper’s notation, the simplified VAR block is

$k$7

and compositions of such blocks already suffice for universality.

A complementary line of work reinterprets VAR through iterative refinement rather than purely autoregressive factorization. In that view, VAR performs deterministic decomposition into a latent Laplacian pyramid in the forward direction and learned coarse-to-fine reconstruction in the backward direction. The formulation introduces latent levels $k$8, residuals $k$9, and discrete residual codes sampled by a transformer-conditioned backward process (Hong et al., 3 Oct 2025). That paper isolates three design choices behind VAR’s efficiency and fidelity: refinement in a learned latent space, prediction as discrete classification over code indices, and partitioning the task by spatial frequency. The original VAR work additionally reports LLM-like empirical scaling laws, with linear correlation coefficients near $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$0 for size scaling and near $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$1 for compute scaling (Tian et al., 2024).

3. Error dynamics, diversity, and acceleration

Although VAR reduces decoding depth relative to token-level AR, the hierarchical dependency across scales introduces its own failure modes. AID-VAR formulates this explicitly as cascading error propagation: coarse-scale mistakes are repeatedly conditioned upon, amplified by upsampling and subsequent decoding, and finally appear as structural distortions, texture drift, and semantic artifacts. Under a Lipschitz propagation assumption, the appendix derives

$\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$2

framing VAR failure as error accumulation driven by discrete quantization, autoregressive dependence, and spatial upsampling amplification (Bi et al., 24 May 2026). The proposed fix is a frozen-backbone adversarial correction scheme: a lightweight guidance injector $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$3 perturbs the internal state at each scale, while a projected discriminator in RGB space diagnoses fidelity gaps. With soft-label VQ-VAE decoding for differentiability and a dynamic refresh strategy, the method improves FID from $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$4 to $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$5 on VAR-d16, from $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$6 to $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$7 on VAR-d20, and from $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$8 to $\mathbf R=(\mathbf R_1,\dots,\mathbf R_K)$9 on VAR-d24; the paper highlights a 16% FID improvement for AID-VAR-d20 with only a 3% parameter increase (Bi et al., 24 May 2026).

A separate limitation is diversity collapse. DiverseVAR argues that diversity in VAR is formed mainly at early scales, while later scales largely stabilize an already committed structure. It introduces a distinction between pivotal and auxiliary components in early-scale feature maps, then applies Soft-Suppression Regularization (SSR) to dominant singular directions before prediction and Soft-Amplification Regularization (SAR) to the output afterward (Wang et al., 21 Nov 2025). On Infinity-2B at $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$0, the method improves COCO 2014 Recall from $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$1 to $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$2, Coverage from $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$3 to $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$4, and FID from $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$5 to $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$6, while CLIPScore remains at $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$7; GenEval and DPG remain close to the vanilla model (Wang et al., 21 Nov 2025). This localizes diversity control to early-scale block computations rather than final-scale decoding.

Acceleration research has converged on a related empirical observation: early scales establish semantics and structure, while later scales mostly refine fidelity. StageVAR formalizes three stages—semantic establishment, structure establishment, and fidelity refinement—and exploits two late-stage properties: semantic irrelevance of text conditioning and low-rank feature structure (Li et al., 18 Dec 2025). Its training-free pipeline turns off CFG in late stages, projects features into a reduced space with random projection, and restores full features with Representative Token Restoration, yielding up to $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$8 speedup with only a $$\mathbf F_k = \sum_{i=1}^k \mathrm{Up}(\mathbf R_i,(h,w)),$$9 drop on GenEval and a $f=\mathcal E(im)$0 decrease on DPG (Li et al., 18 Dec 2025). StepVAR reaches a similar conclusion from a pruning perspective, but preserves both texture and structure by fusing a high-pass textural score with a PCA-based structural score and then reconstructing dense features with nearest-neighbor propagation; the paper notes that the final high-resolution stages account for more than 75% of inference time, and reports $f=\mathcal E(im)$1 speedup on Infinity with latency reduced from $f=\mathcal E(im)$2s to $f=\mathcal E(im)$3s while keeping MJHQ-30K FID at $f=\mathcal E(im)$4 versus $f=\mathcal E(im)$5 for the original model (Liu et al., 2 Mar 2026).

4. Controlled generation and downstream vision tasks

The scale-wise hierarchy of VAR has made it a natural substrate for controlled generation. The original VAR paper already demonstrated zero-shot in-painting, out-painting, and class-conditional editing by teacher forcing or masking selected regions in latent space, without task-specific retraining (Tian et al., 2024). Later work turned these qualitative capabilities into explicit frameworks.

CSD-VAR adapts VAR to content-style decomposition from a single image. Its main claim is that early scales tend to encode style-like properties such as global color, texture, and coarse appearance, whereas later scales carry more content-specific structure such as object shape and fine details. On that basis, it introduces scale-aware alternating optimization, SVD-based rectification of the style embedding to reduce content leakage, and augmented key-value memory to preserve content identity (Nguyen et al., 18 Jul 2025). The accompanying CSD-100 benchmark contains about 100 images with 63 distinct objects and 53 unique styles, and the reported CSD-VAR variants on Switti and Infinity achieve the best or near-best content and style alignment metrics in the provided comparison (Nguyen et al., 18 Jul 2025).

SAVAREdit addresses text-guided image editing with VAR backbones, particularly Infinity-2B. It replaces earlier token-reassembly heuristics with Coarse-to-Fine Token Localization for editable-mask refinement, feature injection from cached source features, and reinforcement-learning-based Adaptive Feature Injection that learns scale- and layer-specific injection ratios (Xia et al., 30 Mar 2026). The paper’s feature analysis identifies the 0-th layer and scales 5–8 as having the highest structural relevance for injection. On PIE-Bench, SAVAREdit reports Structure Distance $f=\mathcal E(im)$6, PSNR $f=\mathcal E(im)$7, SSIM $f=\mathcal E(im)$8, and LPIPS $f=\mathcal E(im)$9, improving over AREdit on structure preservation while remaining competitive on CLIP similarity (Xia et al., 30 Mar 2026).

Seg-VAR extends the paradigm beyond image generation to semantic, instance, and panoptic segmentation by reframing segmentation as conditional autoregressive mask generation. It introduces an image encoder for the prior $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$0, a spatial-aware seglat encoder for the posterior $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$1, and a decoder $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$2, trained in three stages: seglat learning, latent learning, and prior alignment (Zheng et al., 16 Nov 2025). The method uses a location-sensitive color mapping to distinguish instances in seglat space and reports, with Swin-L backbones, 59.7 PQ on COCO panoptic segmentation, 52.7 AP on COCO instance segmentation, 85.82 mIoU on Cityscapes semantic segmentation, and 54.90 mIoU on ADE20K, all above the listed Mask2Former baselines (Zheng et al., 16 Nov 2025). This suggests that the coarse-to-fine latent hierarchy can also be used for spatial perception tasks requiring low-level precision.

5. Safety, personalization, and alignment

Safety-oriented extensions of VAR have focused first on provenance and erasure. Safe-VAR is presented as the first watermarking framework specifically designed for autoregressive text-to-image generation, motivated by the observation that the timing of watermark injection strongly affects quality and that different watermark complexities prefer different scales (Wang et al., 14 Mar 2025). Its Adaptive Scale Interaction Module selects compatible image–watermark scale pairs, while Cross-Scale Fusion combines them through mixture of heads and mixture of experts. At 1024 resolution on LAION-Aesthetics, the paper reports PSNR $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$3, SSIM $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$4, and LPIPS $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$5 for cover-versus-watermarked images, together with strong robustness under crop, noise, JPEG, and combined perturbations; it also reports zero-shot generalization to QR Codes (Wang et al., 14 Mar 2025).

For concept removal, S-VARE argues that diffusion-style erasure objectives are structurally mismatched to VAR because VAR predicts discrete next-scale tokens rather than continuous noise and is strongly autoregressive across scales. The proposed VARE framework stabilizes erasure by conditioning optimization on auxiliary visual tokens from the teacher model, and S-VARE adds a filtered cross-entropy erasure loss plus a preservation KL loss to avoid language drift and quality collapse (Zhong et al., 26 Sep 2025). The paper’s headline claim is that the method erases about 97% of sensitive concepts with less than 2% CLIP degradation, while attack success rates drop from 4.1% to 0.8% on I2P, from 21.7% to 3.5% on MMA, and from 75.9% to 7.4% on Ring-A-Bell (Zhong et al., 26 Sep 2025).

Personalization introduces a different tension between plasticity and retention. CPC-VAR presents the first systematic study of continual personalized generation in VAR, targeting both catastrophic forgetting in sequential concept learning and feature entanglement in multi-concept synthesis (Li et al., 19 May 2026). For continual learning it uses Gradient-based Concept Neuron Selection (GCNS), which identifies concept-relevant neurons in cross-attention blocks and regularizes only the overlap with previously important parameters. For multi-concept synthesis it uses context-aware multi-branch feature modeling and localized cross-attention fusion guided by bounding boxes. On its eight-task continual benchmark, the method reports average DINO $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$6, average CLIP-I $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$7, and average CLIP-T $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$8, above the listed baselines (Li et al., 19 May 2026).

Alignment under reinforcement learning exposes another VAR-specific difficulty: heterogeneous token-grid actions across timesteps. “VAR RL Done Right” describes this as asynchronous policy conflicts and modifies GRPO with a stabilizing middle return, per-action normalization weighting, and mask propagation inspired by Reward Feedback Learning (Sun et al., 5 Jan 2026). On text rendering, NextFlow-RL improves Word Accuracy from $r_k=\mathcal Q(\text{interpolate}(f,h_k,w_k))$9 to $V=4096$0, NED from $V=4096$1 to $V=4096$2, and CLIPScore from $V=4096$3 to $V=4096$4; on HPS refinement, the overall “All” score rises from $V=4096$5 to $V=4096$6 (Sun et al., 5 Jan 2026). The broader implication is that scale-heterogeneous autoregression changes the geometry of policy optimization and requires timestep-aware credit assignment.

6. Architectural variants and deployment

Recent work has also modified VAR at the systems level, either by changing the backbone or by improving deployment efficiency. VIAR replaces the explicit deep stack per scale with an implicit equilibrium layer surrounded by shallow pre- and post-blocks, while retaining next-scale autoregression (Jiang et al., 2 May 2026). The equilibrium state $V=4096$7 satisfies

$V=4096$8

and is trained with stochastic Jacobian-Free Backpropagation, giving constant training memory for the implicit core and an inference-time per-scale iteration knob. On ImageNet $V=4096$9, VIAR reports FID $32$0 and sFID $32$1 with 770.9M parameters, which is 38.4% of the parameters of VAR-D30; by changing the iteration schedule, peak memory drops from 19.24 GB to 8.53 GB and throughput rises from 15.16 to 32.08 images/s on a single RTX 4090, without retraining (Jiang et al., 2 May 2026). This preserves the next-scale interface while making compute allocation adjustable across scales.

Post-training quantization has required VAR-specific adaptations as well. Shift-and-Sum Quantization identifies two PTQ failure modes: large reconstruction errors in attention-value products, especially at coarse scales where attention is more concentrated, and mismatch between codebook-entry sampling frequencies and predicted probabilities under limited calibration data (Moon et al., 15 Jun 2026). To address them, it applies a shift-and-sum quantization kernel to attentive value tokens and a calibration-data resampling scheme that aligns empirical token frequencies with predicted probabilities. On ImageNet class-conditional generation, the method improves over BRECQ across VAR-d16, d20, d24, and d30; for example, on VAR-d24 at 4/6-bit quantization, BRECQ gives IS $32$2 and FID $32$3, whereas the proposed method gives IS $32$4 and FID $32$5 (Moon et al., 15 Jun 2026). The paper further reports only about 2% lower CPU throughput than BRECQ and much faster execution than LiteVAR, indicating that PTQ for VAR is becoming a deployable systems problem rather than only an accuracy problem.

Taken together, these lines of work portray VAR less as a single model family than as a general coarse-to-fine discrete generation interface. The common invariant is next-scale prediction over hierarchical latent token maps; the main points of variation are how scales are parameterized, corrected, compressed, aligned, or repurposed for tasks beyond unconditional synthesis. This suggests that the defining contribution of VAR is not merely faster sampling than raster AR, but an alternative organization of visual autoregression around scale, hierarchy, and parallelism.