Continuous Autoregressive Modeling

• Continuous autoregressive modeling is a framework for generating high-dimensional sequences by factorizing joint distributions over continuous latent variables.
• It employs diverse architectures—including Transformers, diffusion models, and normalizing flows—to bypass quantization artifacts and enhance sample fidelity.
• This approach demonstrates scalable performance across domains like vision, speech, and language, while addressing challenges in error accumulation and inference efficiency.

Continuous autoregressive modeling encompasses a diverse family of generative models that factorize the joint probability of high-dimensional, real-valued sequences—such as images, audio, video, or language representations—via an autoregressive chain over continuous latent variables. This paradigm aims to combine the sequential modeling strengths demonstrated in discrete autoregressive models (e.g., LLMs) with the expressivity and fidelity of continuous representations, bypassing the information loss and artifacts introduced by quantization. Modern continuous autoregressive models employ a range of architectures, loss functions, density parameterizations, and application-specific factorization techniques, yielding scalable, statistically principled, and empirically competitive generative systems across vision, speech, and beyond.

1. Foundations and Mathematical Formulation

At the core, continuous autoregressive models factorize a joint probability distribution over a sequence of real-valued vectors $x_1, x_2, \dots, x_T \in \mathbb{R}^d$ as

$p(x_{1:T}) = \prod_{t=1}^{T} p(x_t | x_{<t})$

where each conditional $p(x_t | x_{<t})$ models a potentially high-dimensional, unbounded continuous density. This formulation is a natural generalization of discrete AR models to continuous token spaces and underlies approaches in image, audio, and language generation (Shao et al., 12 May 2025, Banerjee et al., 2024, Yu et al., 7 Mar 2025).

In continuous-time settings, as in Lévy-driven multivariate CAR($p$) processes, the continuous autoregressive structure is formalized via systems of stochastic differential equations, with the joint law characterized by the SDE and the driving noise process (Lucchese et al., 2023, Basse-O'Connor et al., 2017).

2. Modeling Architectures and Tokenizations

Continuous autoregressive models generally require a transformation from raw data to a structured latent space:

• Continuous Tokenizers: Encoder–decoder pairs (e.g., VAEs) map input data (images, waveforms, etc.) to grids or sequences of continuous tokens. For images, each $z_i\in\mathbb{R}^d$ is a patchwise embedding; for speech and language, continuous vectors compress fixed-length chunks or frames (Yu et al., 7 Mar 2025, Wu et al., 26 Aug 2025, Lin et al., 3 Feb 2025, Shao et al., 31 Oct 2025).
• Latent Structure: Images adopt 2D grids (spatial), audio uses 1D frames/patches, video stacks tokens per frame, and language may chunk tokens via autoencoders for next-vector prediction (Yu et al., 7 Mar 2025, Zhang et al., 1 Jul 2025, Yu et al., 17 Jun 2025, Shao et al., 31 Oct 2025).
• Model Backbones: Causal or masked self-attention Transformers, sometimes augmented with bidirectional, conditional, or flow-based modules, remain the backbone for long-range autoregressive dependency modeling (Hang et al., 24 Apr 2025, Yu et al., 17 Jun 2025, Hu et al., 2024).

3. Density Parameterizations and Generative Heads

Constructing valid and expressive conditional densities over continuous tokens is a principal challenge. Several strategies prevail:

• Diffusion/Flow Heads: The conditional $p(x_t| x_{<t})$ is realized implicitly via a diffusion process or flow-matching ODE, where a neural network predicts score or velocity fields to perform denoising from noise to data (Yu et al., 7 Mar 2025, Hang et al., 24 Apr 2025, Banerjee et al., 2024, Jia et al., 6 Feb 2025, Wu et al., 26 Aug 2025).
• Energy-Based Generative Heads: Autoregressive models can use strictly proper scoring rules (e.g., the energy score)

$$S_\text{energy}(p,x) = \mathbb{E}_{x'\sim p}\|x'-x\|^\alpha - \frac{1}{2}\mathbb{E}_{x_1,x_2\sim p}\|x_1-x_2\|^\alpha$$

to define and optimize over implicit distributions, sidestepping the need for explicit likelihoods (Shao et al., 12 May 2025, Shao et al., 31 Oct 2025).

• Gaussian Mixture/Normalizing Flows: For speech and language, Gaussian mixture models or autoregressive normalizing flows provide analytic density parameterizations, allowing for maximum likelihood training and tractable sampling (Lin et al., 3 Feb 2025, Zhang et al., 1 Jul 2025).
• Conditional Diffusion by Blocks or Levels: Some frameworks operate on blocks or frequency bands, applying conditional diffusion per block, often enabling flexible interpolation between full sequence denoising (diffusion) and token-wise AR (Hu et al., 2024, Yu et al., 7 Mar 2025).

4. Training Objectives and Losses

Continuous autoregressive models require losses tailored for continuous densities:

• Diffusion and Denoising Loss: Models optimize per-token or per-block denoising score-matching objectives, minimizing

$$\mathbb{E}_{t,\epsilon}[\|\epsilon-\epsilon_\theta(x_t | t, \text{context})\|^2]$$

as in the MAR/FAR or DiTAR frameworks (Yu et al., 7 Mar 2025, Hang et al., 24 Apr 2025, Jia et al., 6 Feb 2025).

• Energy Score Maximization: Likelihood-free, strictly proper scoring rules such as the energy score ensure unique statistical consistency without density evaluation (Shao et al., 12 May 2025, Shao et al., 31 Oct 2025).
• Negative Log-Likelihood: For explicit Gaussian mixture or flow models, the negative log-likelihood (or ELBO in VAE settings) remains standard (Lin et al., 3 Feb 2025, Zhang et al., 1 Jul 2025).
• Auxiliary Losses: KL-regularization (autoencoders), adversarial, or reconstruction losses are used where relevant, especially during tokenizer/autoencoder training (Wu et al., 26 Aug 2025, Shao et al., 31 Oct 2025).

5. Efficiency, Inference, and Error Handling

Inference efficiency and error control remain major development axes:

• Parallel and Blockwise Generation: To mitigate sequential token bottlenecks, multistage AR methods generate coarser-resolution maps or blocks in parallel (E-CAR, ACDiT), serializing only at higher resolutions (Yuan et al., 2024, Hu et al., 2024).
• Flow-Shortcut and Few-Step Methods: Replacing iterative diffusion with shortcut flows (FAR head) or reducing denoising steps enables speedups of $2\times$–$10\times$ over previous diffusion-AR baselines (Hang et al., 24 Apr 2025, Yuan et al., 2024).
• Error Accumulation and Noise Augmentation: Distributional drift from error accumulation during AR generation is directly addressed by noise augmentation during training and inference-time noise injection, stabilizing long-horizon outputs especially for audio and sequence tasks (Pasini et al., 2024).
• Streaming and Low-Latency Architecture: Causal VAE decoders, fast AR heads, and interleaved text-audio sequences facilitate streaming synthesis, reducing first-frame and packet delay for speech applications (Wu et al., 26 Aug 2025, Wang et al., 14 Jun 2025).
• Temperature and Diversity Control: Temperature in continuous models is implemented via noise-injection time (diffusion ODE) or rejection sampling at the decoder (Jia et al., 6 Feb 2025, Shao et al., 31 Oct 2025).

6. Application Domains and Empirical Results

Continuous autoregressive modeling defines SOTA or near-SOTA architectures in several modalities:

Domain	Notable Frameworks	Key Metrics Achieved
Image Generation	FAR, E-CAR, VAR, DisCon, ACDiT	FID ∼1.4–3.4, Inception Score ∼250–300 (Yu et al., 7 Mar 2025, Yuan et al., 2024, Shao et al., 12 May 2025, Zheng et al., 2 Jul 2025, Hu et al., 2024)
Video Generation	VideoMAR, ACDiT	FVD ∼90–104 (UCF-101), low GPU budgets (Yu et al., 17 Jun 2025, Hu et al., 2024)
Speech Synthesis	CLEAR, DiTAR, StreamMel, GMM-LM, CAM	WER 1.74–2.8%, RTF 0.18–0.3, SIM-o/SIM >0.55 (Wu et al., 26 Aug 2025, Lin et al., 3 Feb 2025, Jia et al., 6 Feb 2025, Wang et al., 14 Jun 2025, Pasini et al., 2024)
Language Modeling	CALM, TarFlowLM	BrierLM 5.72, >4× step speedup, competitive PPL (Shao et al., 31 Oct 2025, Zhang et al., 1 Jul 2025)

These models demonstrate that continuous AR approaches, exploiting modern neural density parameterizations, outperform or match discrete-token AR and even pure diffusion baselines with reduced latency and improved sample fidelity.

7. Continuous-Time Stochastic and Theoretical Foundations

Beyond deep learning, continuous autoregressive processes in stochastic calculus provide a rigorous probabilistic foundation:

• CAR($p$) and MCAR($p$) SDEs: Higher-order CAR processes are formulated via state-space SDEs driven by Lévy noise, yielding explicit convolution solutions and links to ARMA models under discrete sampling (Lucchese et al., 2023, Basse-O'Connor et al., 2017).
• Graphical MCAR: In settings with structured dependencies (e.g., multivariate time series with known graphs), GrCAR models use adjacency-informed drift matrices, supporting parsimonious parameter estimation (Lucchese et al., 2023).
• Estimation and Inference: Maximum likelihood estimators for CAR($p$) processes are explicit or can be discretized (Riemann sums, finite-difference, thresholding). Under high-frequency, possibly irregular, sampling, these estimators retain consistency and asymptotic normality even under finite or infinite activity jump noise (Lucchese et al., 2023).
• Links to Discrete Models: Sampling a CAR process on a grid recovers a discrete-time ARMA process; SDE parameterizations allow natural interpolation for irregularly sampled discrete data (Basse-O'Connor et al., 2017, Lucchese et al., 2023).
• Empirical Validation: Simulation confirms the rapid concentration and asymptotic normality of feasible estimates under various noise regimes (Brownian, finite/infinite-activity Lévy) (Lucchese et al., 2023).

8. Challenges, Limitations, and Future Directions

While continuous autoregressive modeling offers significant advantages, several challenges and open questions persist:

• Modeling Complex Continuous Densities: Flow/diffusion-based conditional models require careful tuning, and may be sensitive to training instabilities or out-of-distribution drift (Shao et al., 12 May 2025, Pasini et al., 2024).
• Error Accumulation: Without regularization, autoregressive chains over high-dimensional continuous outputs are prone to accumulating errors, potentially causing sample drift—addressed via explicit noise augmentation or blockwise/multistage generation (Pasini et al., 2024, Hu et al., 2024).
• Sampling and Controllability: Temperature and diversity trade-offs are less straightforward than in softmax-based discrete models, requiring procedure-specific interventions (Jia et al., 6 Feb 2025, Shao et al., 31 Oct 2025).
• Scalability and Architectural Overhead: Efficiency gains hinge on flow-matching, blockwise parallelism, and new training regimes (curriculum learning, multistage flows). Very high compression ratios in the tokenizer can make preserving fidelity challenging, especially in speech (Wu et al., 26 Aug 2025, Shao et al., 31 Oct 2025).
• Theoretical Gaps: Understanding strict propriety, calibration, and the statistical underpinnings of these implicit models in high-dimensional spaces remains an active research area (Shao et al., 12 May 2025).

Future developments target end-to-end learned tokenizers, richer autoregressive density models (e.g., hierarchical, context-aware flows), tighter integration of discrete and continuous signals, and further reductions in inference cost—aiming for seamless, universal, high-fidelity sequence modeling suitable for text, audio, vision, and multimodal synthesis.