Masked Conditional Autoregressive Models

Updated 7 April 2026

Masked conditional autoregressive models are deep generative models that blend autoregressive factorization with explicit masking to control generation order and conditioning.
They integrate techniques such as masked convolutions and Transformer-based self-attention to support efficient parallelization and multimodal conditional generation.
Empirical results show that MCAR models achieve state-of-the-art performance in image, video, and language tasks while offering accelerated sampling and improved parameter efficiency.

Masked conditional autoregressive models (MCAR models) are a class of deep generative models leveraging both autoregressive factorization and explicit masking strategies to offer control over generation order, conditioning, and parallelization. They have become essential in domains such as image, video, and language generation due to their capacity for order-agnostic sampling, conditional infilling, fast decoding, and parameter-efficient ensembling. The following provides a comprehensive treatment of MCAR models, with a particular emphasis on their mathematical formulation, architectural components, conditional mechanisms, parallelization and acceleration strategies, and empirical performance.

1. Mathematical Formulation and Factorization Principles

The foundational principle of MCAR models is the chain-rule-based autoregressive factorization of a joint distribution. For a $D$ -dimensional data vector $x = (x_1, \ldots, x_D)$ , the autoregressive decomposition with respect to a permutation (generation order) $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ is

$p_\theta(x) = p_\theta(x_{\pi(1)}) \prod_{i=2}^D p_\theta(x_{\pi(i)} \mid x_{\pi(1)}, \ldots, x_{\pi(i-1)})\,,$

where $\pi$ can be fixed (e.g., raster-scan order) or variable, with $\Pi$ a set of generation orders sampled uniformly during training and selected (or ensembled) at inference (Jain et al., 2020).

In masked variants, the state of knowledge about $x$ at each generation step is explicitly encoded by a mask $M \in \{0,1\}^D$ , multiplying inputs to forcibly hide or reveal specific dimensions. This allows models to perform conditional infilling, arbitrary-order autoregression, and support for context-dependent masking such as outpainting, inpainting, or imputation.

For multi-dimensional (e.g., image, video) or hierarchical data, blockwise masked AR decompositions are adopted:

Hierarchical MAR: $p(x) = \prod_{k=1}^K p(x^{(k)} \mid x^{(<k)})$ , with each $x^{(k)}$ representing tokens at a resolution scale or time step (Kumbong et al., 4 Jun 2025), and blockwise within-scale masked autoregression: $x = (x_1, \ldots, x_D)$ 0.
Video MAR: $x = (x_1, \ldots, x_D)$ 1 (Li et al., 15 Oct 2025).

The mask scheduling determines the set of positions to be revealed (predicted) in each step, which can follow arbitrary orderings or be optimized for downstream conditional sampling.

2. Architectural Mechanisms: Masked Convolutions, Transformers, and Unified Decoders

A prototypical architectural mechanism in MCAR models is the application of \textbf{locally masked convolution} (LMConv), wherein binary masks $x = (x_1, \ldots, x_D)$ 2 are used to modulate convolutional kernels at each spatial location $x = (x_1, \ldots, x_D)$ 3 depending on the generation order $x = (x_1, \ldots, x_D)$ 4. For image modeling,

$x = (x_1, \ldots, x_D)$ 5

with $x = (x_1, \ldots, x_D)$ 6 indicating whether pixel $x = (x_1, \ldots, x_D)$ 7 precedes $x = (x_1, \ldots, x_D)$ 8 under $x = (x_1, \ldots, x_D)$ 9 (Jain et al., 2020). Parameter sharing across all orders is realized by reusing $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 0 and precomputing / caching all masks.

In Transformer-based MCAR models, bidirectional self-attention is combined with mask logic—typically, self-attention patterns or softmax logits are masked so that tokens at masked positions attend only to unmasked (observed or previously generated) tokens (Wang et al., 22 May 2025, Kumbong et al., 4 Jun 2025, Qu et al., 2024). Multi-head self-attention operates on the concatenation of context tokens, conditional tokens, and generation tokens, with customized attention masks reflecting causal/bidirectional dependencies.

For conditional MAR, multimodal signals (text, image, or class embedding) are incorporated either by (i) concatenation into the sequence processed by self-attention ("same-space" or self-control fusion (Qu et al., 2024)), or (ii) traditional cross-attention (Text/Image tokens as keys/values). The former approach unifies all modalities in a single representational space, improving parameter efficiency and conditional fidelity.

3. Conditional Mechanisms and Multimodal Control

MCAR models for conditional generation integrate conditioning variables $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 1 by constructing factorizations such as: $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 2 where $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 3 may encode text prompts (in text-to-image, text-to-panorama), class labels (class-conditional generation), or both. In \textbf{continuous MCAR models} (e.g., (Qu et al., 2024)), all relevant cues are input as tokens to a unified self-attention stack, with segmentation enforced by block-specific attention masks: causal for text, bidirectional for image/generation.

Advanced conditional mechanisms include:

\textbf{Classifier-Free Guidance (CFG)}: For conditional image/text generation, CFG combines outputs from conditional and unconditional forward passes, linearly interpolated with a guidance weight $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 4:

$\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 5

(Yan et al., 16 Mar 2025, Li et al., 15 Oct 2025).

\textbf{Compositional Guidance}: In video MAR, compositional CFG enables simultaneous upweighting of temporal and spatial (canvas) priors via multiplicative weights $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 6, $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 7 on the respective conditional densities (Li et al., 15 Oct 2025).
\textbf{Circular Padding}: To generate continuous and seam-free panoramic images in ERP (equirectangular projection), dual-phase circular padding is used to extend the model's receptive field across horizontal boundaries, preserving equivariance (Wang et al., 22 May 2025).

This conditional flexibility allows MCAR models to unify tasks such as text-to-image, panorama outpainting, and multimodal video synthesis in a single framework.

4. Efficient Parallelization, Acceleration, and Sampling Schedules

Unlike sequential AR models, MCAR architectures exploit partial parallelization via blockwise masked prediction, group-wise infilling, and hierarchical scheduling.

Key strategies:

\textbf{Multi-Step Masked Generation}: Within each fine‐to‐coarse (or vice versa) scale, groups of tokens are predicted in each iteration, selected either uniformly at random or by confidence (Kumbong et al., 4 Jun 2025, Li et al., 15 Oct 2025). This allows flexible adjustment of the number of prediction steps $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 8 at inference, trading speed for synthesis quality.
\textbf{Two-Stage Sampling (GtR)}: The Generation-then-Reconstruction (GtR) paradigm partitions sampling into a slow “structure generation” stage for global semantic scaffolding and a fast “detail reconstruction” stage for completing details. This enables acceleration by focusing computation on critical regions and leveraging local context during detail infilling, with up to 3.72× speedup at negligible FID/IS degradation (Yan et al., 20 Oct 2025).
\textbf{Token and Condition Caching}: To overcome the lack of standard KV caching in bidirectional MAR models, LazyMAR implements token caching (reusing redundant token features across MAR steps) and condition caching (reusing stable conditional-unconditional residuals in CFG), achieving 2.83× acceleration in large image models (Yan et al., 16 Mar 2025).
\textbf{Blockwise and Strided Parallel Inference}: In autoregressive masked diffusion (ARMD), blockwise autoregressive factorization is leveraged to generate multiple non-overlapping streams (blocks) in parallel, significantly reducing the number of sequential network calls with minimal increase in perplexity (Karami et al., 23 Jan 2026).

A summary of acceleration methods is provided below:

Method	Principle	Speedup	Quality Drop (FID/IS)
Multi-step masking	Masked batch prediction	×1.7–3.0	Negligible
GtR (Yan et al., 20 Oct 2025)	Two-stage, detail-weighted	×3.7	<0.1 FID
LazyMAR (Yan et al., 16 Mar 2025)	Feature and condition caching	×2.8	≲0.1 FID
ARMD Blockwise (Karami et al., 23 Jan 2026)	S–way parallel block sampling	×2–4	Modest (1–2 ppl)

Adjusting sampling order (raster scan, random, checkerboard, etc.) and exploiting frequency content for token selection further optimize the efficiency/quality trade-off (Yan et al., 20 Oct 2025).

5. Empirical Results and Theoretical Guarantees

MCAR models achieve state-of-the-art quality and flexibility across modalities:

Image Modeling: On CIFAR-10, LMConv obtains 2.89 bpd (ensemble, unconditional), surpassing PixelCNN++ (2.92 bpd) (Jain et al., 2020). HMAR achieves FID = 1.95 and IS = 334.5 on ImageNet 256×256, outperforming both VAR and diffusion baselines (Kumbong et al., 4 Jun 2025). Two-stage GtR and LazyMAR strategies match these scores with up to 3–4× inference acceleration and negligible ( $\pi: \{1, \ldots, D\} \to \{1, \ldots, D\}$ 9FID ≈ 0.06) degradation (Yan et al., 20 Oct 2025, Yan et al., 16 Mar 2025).
Video Modeling: CanvasMAR achieves FVD ≈ 6.2 on Kinetics-600 (5→16 frames), rivaling leading diffusion rollouts at a fraction of the autoregressive sampling steps (Li et al., 15 Oct 2025).
Language Modeling: ARMD matches GPT-2 perplexities (e.g., PTB ARMD(300K) 97.75 vs. GPT-2 123.14) while requiring 3–8× fewer training steps (Karami et al., 23 Jan 2026). Universal masked and AR models such as u-PMLM and MARIA attain or surpass GPT-level quality, support arbitrary infilling, and maintain high throughput due to retained KV caching (Liao et al., 2020, Israel et al., 9 Feb 2025).
Translation and Multimodal Tasks: CeMAT pretraining for masked conditional language modeling yields +7.9 average BLEU on autoregressive NMT, +2.5 BLEU on NAT, with a single bidirectional decoder architecture supporting both settings (Li et al., 2022).

6. Limitations, Extensions, and Open Challenges

MCAR models, despite their advances, encounter several technical challenges:

\textbf{Caching and State Staleness}: Bidirectional attention, while enabling masked parallel decoding, complicates feature caching. Though LazyMAR provides effective solutions, the quadratic complexity in tokens remains for each MAR step (Yan et al., 16 Mar 2025).
\textbf{Mask Ordering and Training Schedules}: Optimal mask selection, progressive permutation curriculums (ARMD), and frequency-based prioritization require careful tuning for maximal efficiency/quality balance (Karami et al., 23 Jan 2026, Yan et al., 20 Oct 2025).
\textbf{Conditioning Fineness}: Resolving blurred or inconsistent conditioning (e.g., video motion, high-frequency details in images) remains challenging; integrating multi-scale canvases or adaptive mask granularity is a potential direction (Li et al., 15 Oct 2025).
\textbf{Universal Conditionality}: Unifying text, image, video, and other modalities in a single, parameter-efficient MCAR stack (e.g., self-control networks) is feasible, but demands careful architectural search, especially for fusion and attention-masking schemes (Qu et al., 2024).
\textbf{Scalability}: State-of-the-art performance at higher resolutions or longer sequences still implies superlinear compute/memory; advancements in block-sparse attention (Kumbong et al., 4 Jun 2025) mitigate but do not fully resolve this scaling bottleneck.

A plausible implication is that further integration of MCAR concepts (arbitrary-order generation, blockwise factorization, efficient masking, and unified conditional modeling) with large pretrained AR/diffusion backbones could produce models with both state-of-the-art speed and flexibility.

7. Theoretical Connections and Unification

At the core of MCAR theory lies the equivalence between certain probabilistic masking schemes and permutation-invariant autoregressive modeling. For example, u-PMLM, trained with a uniform prior over masking ratios, is strictly equivalent to an autoregressive model learned over all $p_\theta(x) = p_\theta(x_{\pi(1)}) \prod_{i=2}^D p_\theta(x_{\pi(i)} \mid x_{\pi(1)}, \ldots, x_{\pi(i-1)})\,,$ 0 permutations of the data, thus supporting arbitrary-order, cloze-style, and lexically constrained generation with no additional architectural components (Liao et al., 2020). This provides both a theoretical guarantee and a practical route to merging the strengths of masked and standard AR paradigms.

Moreover, recent diffusion-AR unification (e.g., ARMD) demonstrates that denoising diffusion training objectives can be reformulated as blockwise masked conditional autoregressive losses, realized by a strictly causal, permutation-equivariant network architecture. This aligns MCAR approaches with the broader generative modeling literature, demonstrating that the core MCAR machinery is theoretically and practically generalizable across discrete, continuous, spatial, and sequential data (Karami et al., 23 Jan 2026).

In aggregate, masked conditional autoregressive models offer a unified, theoretically-grounded, and empirically robust framework for high-dimensional, conditional, and parallelizable generative modeling, with demonstrated applications and scalability across vision, language, and multimodal domains (Jain et al., 2020, Yan et al., 20 Oct 2025, Kumbong et al., 4 Jun 2025, Li et al., 15 Oct 2025, Qu et al., 2024, Karami et al., 23 Jan 2026, Yan et al., 16 Mar 2025).