Conditional Autoregressive Training

• Conditional autoregressive training is a method that generates output sequences by factorizing probability into sequential univariate conditionals based on both past tokens and side signals.
• Innovative strategies like ECM and MAC leverage control adapters and mask-tuned protocols to enhance efficiency, reduce compute redundancy, and improve sample quality.
• Hybrid approaches incorporating blockwise conditioning, RL, and diffusion methods offer robust, scalable solutions across diverse data modalities.

Conditional autoregressive training strategies encompass methods for learning explicit probabilistic models $\{p_\theta(x_{1:T}|c)\}$ where an output sequence $x_{1:T}$ is generated conditional on signal $c$ (class, text, image, metadata, etc.) and modeled via the autoregressive factorization: $$p_\theta(x_{1:T}|c) = \prod_{t=1}^{T} p_\theta(x_t|x_{<t},c).$$ Modern research addresses conditional control, inference efficiency, robustness, and control over variable or arbitrary subsets through a range of technical strategies and architectural innovations.

1. Core Principles and Definitions

Conditional autoregressive training treats the generation of $x_{1:T}$ as a chain of univariate distributions, each conditioned on both the autoregressive prefix $x_{<t}$ and side information $c$. The conditioning signal can be class labels, structural control signals (e.g., edges, depth maps), arbitrary observed subsets, or "past" context (e.g., frames for dynamics). Deviations from naive teacher-forcing training seek to align model training with inference, allow efficient arbitrary conditioning, and yield better controllability or sample quality.

Key technical axes include:

• Efficient parameterization of conditional signals and their integration into deep architectures
• Sampling and training protocols to maximize learning of high-utility conditionals and reduce redundancy
• Mechanisms for leveraging control at multiple scales or blocks to optimize both local and global structure
• Variants for arbitrary subset conditioning and data completion

2. Lightweight Conditional Control: The ECM Framework

Recent progress in scale-based visual autoregressive modeling is exemplified by the Efficient Control Model (ECM), which introduces a plug-and-play control module to inject externally-provided control tokens (such as spatial constraints) without end-to-end retraining of the frozen base AR model (Liu et al., 7 Oct 2025).

ECM architecture:

• Inserts compact control adapters (denoted $F_\phi$) at selected Transformer layers.
• Each adapter fuses context-aware attention over AR tokens and control signals, producing an additive correction to the token embeddings.
• Uses a shared, gated FFN across adapters, with layer-specific gating for efficient use of adapter capacity.
• The output of the adapter for scale $k$, $F_\phi(k)$, is summed with the token embedding at each step, yielding the controlled AR factorization:

$$p_\theta(s_k | \text{cls} + F_\phi(1), s_1 + F_\phi(2),\,\ldots)$$

• Only the adapter parameters $\phi$ are trained, leaving the base model $\theta$ frozen.

Early-centric sampling:

• Training preferentially samples early (coarse) scales by drawing a truncation scale $s \sim \gamma(s) \propto (s/S)^\alpha$ (with $\alpha > 1$), focusing computation on large-scale semantics and reducing per-step compute by up to 80%.
• Accommodates reduced coverage for later (fine) scales by annealing the sampling temperature at inference:

$$T_s = T_\mathrm{high} + (T_\mathrm{low} - T_\mathrm{high}) \cdot (s/S)^2$$

• Loss is accumulated only on the selected prefix.

Empirical impact:

• On 256×256 C2I ImageNet, ECM adapters with $\sim$58M parameters achieve FID $=5.77$ (vs. ControlVAR baseline $=16.20$) and IS $=181$ with $22.5\%$ of baseline compute.
• Inference overhead (relative to VAR) is minimal ($0.23\,\mathrm{s}$/image vs. $0.19\,\mathrm{s}$/image).

3. Any-Order and Mask-based Conditional Training

Conditional requirements in data completion and arbitrary conditioning tasks motivate training protocols that directly align the conditional usage at train and test time. Any-Order ARMs (Shih et al., 2022) and OA++ (Voisin et al., 2017) are representative.

Mask-tuned Arbitrary Conditional Models (MAC):

• Parameterizes all univariate conditionals $p(x_j|x_e)$ for all subsets $e\subset\{1,\ldots,N\}$ containing $j$, but only trains a minimal set of edges defined by a global index ordering—removing redundancy in the lattice of conditionals.
• Edge weights $w(i, S)$ in the likelihood objective are proportional to their expected frequency under the test time mask distribution $M$:

$$L(\theta) = \sum_{(i,S)} w(i,S)\, [ -\log p_\theta(x_i|x_S)]$$

• Sampling protocol selects edges deterministically (e.g., by always removing the max index), ensuring all test-time queries can be answered with a minimal consistent set of learned conditionals.
• Empirical results show consistent improvements ($2$–$10\%$) in joint and marginal likelihoods across modalities.

OA++ Procedure for Data Completion:

• Trains only the conditionals expected to be used under a specified inference-query distribution $\mathcal{D}$ over observed variable subsets.
• Loss function:

$$\mathcal{I}_{\mathrm{OA++}}(\theta) = \mathbb{E}_{x,\,\text{obs}\sim\mathcal{D},\,o\in\mathcal{O}_K} \left[ -\log p(x^\mathrm{missing}|x^\mathrm{obs};\theta,o) \right]$$

• Efficient: smaller ensemble of orderings (size $K \ll D!$), reduced overfitting, and faster convergence than full OA.
• Achieves lower negative log-likelihood on standard benchmarks and benefits especially from prior knowledge of inference-time mask distributions.

4. Blockwise and Chunked Autoregressive Conditioning

For structured data (e.g., images, audio, video, fields), concurrent chunks or blocks can serve as the AR unit, boosting efficiency and enabling custom inductive biases on local transitions.

Blockwise diffusion–autoregressive hybrid (ACDiT):

• Partitions the sequence into $N = L/B$ blocks; each block is generated via conditional diffusion, conditioned on all previous (clean) blocks (Hu et al., 10 Dec 2024).
• Uses the Skip-Causal Attention Mask (SCAM) to enforce attention to the clean prefix:
  • Each block $n_i^{(t)}$ is denoised using only $c_{<i}$ and itself.
• By varying block size $B$, interpolates between tokenwise AR and full-sequence diffusion.
• Attains SOTA FID on ImageNet256 (FID $=2.45$ with 677M params, AR length $=4$) and transfer ability to discriminative tasks.

Chunked Autoregressive GAN (CARGAN):

• For conditional waveform synthesis, generates in chunks of $k$ samples; each chunk is conditioned on $n$ previous samples and the corresponding spectrogram frames (Morrison et al., 2021).
• AR context encoding allows the generator to learn phase continuity and pitch, outperforming non-AR GANs in pitch RMSE and subjective listening (e.g., 43% RMSE reduction).
• Generalizes to other domains (text, images, video) via autoregressive chunking and local context encoding.

Locality-constrained AR normalizing flows:

• For lattice field theory, factorizes the field into blocks (e.g., time slices), with each block modeled by a conditional normalizing flow conditioned only on required neighbors (R., 2023).
• This reduces parametric size and dramatically decreases mixing times in high-dimensional sampling tasks.

5. Extensions: RL, Diffusion, and Energy-based Conditional Training

Advanced autoregressive conditional strategies have yielded strong results by integrating principles from RL, diffusion modeling, and energy-based learning.

Reinforcement-Learning enhanced AR (AR-GRPO):

• Refines autoregressive image generation via online RL and Group Relative Policy Optimization (Yuan et al., 9 Aug 2025).
• The AR Transformer is treated as a policy, producing image tokens conditional on class/text; rewards are computed on the final image for semantic alignment (CLIP/HPSv2), realism (VLM-Qwen), and perceptual quality (MANIQA).
• Group-level relative advantages are computed, combining per-sample rewards with clipped policy gradient, and a KL penalty to a reference to ensure stability.
• Demonstrated $10$–$20\%$ IS improvement and improved human preference metrics.

Autoregressive Conditional Diffusion for temporal dynamics:

• Conditional diffusion models (DDPMs) may be rolled out in an AR manner for sequence modeling (e.g., turbulent flows) (Kohl et al., 2023).
• At each step, previous $k$ states (and parameters) are concatenated as noisy conditioning and the model is trained single-step; at inference, AR rollout enables long-horizon predictions with improved stability.
• No multi-step unrolling is needed and the noise-injected conditioning is critical for robustness over very long sequences.

Energy-based autoregressive models (E-ARM):

• Reinterprets AR models as energy-based models by defining an unnormalized energy on $(x_{<t}, x_t, c)$ and training using a contrastive divergence objective (Wang et al., 2022).
• Both positive (teacher-forced) and negative (model-sampled) phases are computed; importance weighting is applied to the negative phase for stability.
• Directly addresses exposure bias and long-range coherence issues common in standard AR training, yielding improved BLEU on NMT benchmarks.

Noise-Conditional Maximum Likelihood (NCML):

• Leverages a continuum of noise conditional likelihoods, introducing a noise parameter $\sigma$ into the AR factorization for every sample (Li et al., 2022).
• The model predicts $p_\theta(\tilde{x}|\sigma)$ where $\tilde{x}=x+\varepsilon$ and samples $\varepsilon \sim \mathcal{N}(0,\sigma^2I)$.
• Robustifies the AR model, improves test likelihood, and enables score-based sampling to mitigate covariate shift.
• Demonstrated FID $=12.1$ on CIFAR-10 vs. $37.5$ for the best vanilla AR model.

6. Summary Table of Key Conditional AR Strategies

Method / Paper	Conditional Control	Training Adaptation	Empirical Impact
ECM (Liu et al., 7 Oct 2025)	Control adapters in AR model (e.g., edge, depth)	Early-centric sampling, only adapters trained	FID improvement (5.77 vs 16.2 C2I), $<25\%$ compute
MAC (AO-ARM) (Shih et al., 2022)	Any observable mask	Edge selection, usage-proportional weights	SOTA bpd in joint/marginal likelihood
ACDiT (Hu et al., 10 Dec 2024)	Blockwise AR-diffusion, continuous or discrete	SCAM, interpolating block size	FID 2.45, SOTA transferability
CARGAN (Morrison et al., 2021)	Chunked audio AR, mel and AR context	Chunk-based adversarial + AR context	−43% pitch RMSE, $-58\%$ training time
AR-GRPO (Yuan et al., 9 Aug 2025)	Class/text-conditional, RL refinement	Group RL, group-relative PG, KL reg	IS increase 10–20%, better user prefs
E-ARM (Wang et al., 2022)	Seq2seq, conditioning, energy-based	Positive/negative phase, importance weighting	BLEU $+0.6$ on NMT
NCML (Li et al., 2022)	Noise-level-conditional AR	Joint $\sigma$ likelihood, score-based sampling	FID 12.1 (vs 37.5)
l-ACNF (R., 2023)	Locality-blocked CNFs	Per-block flow, local dependencies only	$>10^2\times$ autocorrelation reduction

Each strategy emphasizes minimizing compute or redundancy, maximizing conditional expressivity and inference tractability, and aligning training with inference distributional properties.

7. Outlook and Research Directions

Conditional autoregressive training strategies remain an active area, with current work targeting:

• More scalable control/conditioning adapters and chunked/block parameterizations for multimodal sequence data
• Efficient training protocols for rare/infrequent conditional edges (e.g. in tabular, inpainting, or partial completion)
• Further integration of score-based, energy-based, and diffusion methods for robust and sample-efficient AR control
• Extensions to more general architectures (graph-structured, non-sequential, online RL) and transfer to discriminative settings
• Analyses of sample complexity, inductive bias, and long-range statistical properties under different AR conditioning regimes

Factual claims, empirical findings, and specific training setups trace to (Liu et al., 7 Oct 2025, Shih et al., 2022, Hu et al., 10 Dec 2024, Morrison et al., 2021, Li et al., 2022, Wang et al., 2022, R., 2023), and (Yuan et al., 9 Aug 2025).