Multimodal Autoregressive Pre-training

• Multimodal autoregressive pre-training is a framework that linearizes text, images, audio, and other signals into a unified token stream to enable joint understanding and generation.
• It employs a causal Transformer with advanced tokenization strategies, such as VQ-VAE and continuous representations, to optimize cross-modal reasoning and parameter efficiency.
• This approach achieves robust performance in tasks like visual question answering, captioning, and embodied control by minimizing architectural complexity and using progressive curriculum training.

Multimodal autoregressive pre-training is a foundational paradigm for constructing unified models capable of jointly understanding and generating signals from multiple modalities—including language, vision, speech, audio, action, and more—via a single, left-to-right autoregressive process. These models convert input data from each modality into a shared or interleaved sequence of tokens and employ a Transformer (or other AR sequence model) to predict the next token conditioned on all previous tokens, imposing a strict causality constraint across modality boundaries. This approach underlies the latest advances in large-scale multimodal AI models, facilitating cross-modal reasoning, multimodal sequence generation, and instruction following, while minimizing architectural complexity and maximizing parameter/data efficiency.

1. Autoregressive Modeling Framework

Modern multimodal autoregressive models linearize all inputs and outputs—across modalities—into a strictly ordered token stream. Each token represents a unit of text, a discretized visual patch/code, an audio frame, an action, or other structured element. The core architectural principle is to employ a single causal Transformer (decoder-only or encoder-decoder) that autoregressively predicts $p(x_t|x_{<t})$ over this mixed-modal sequence. This enables seamless next-token prediction, infilling, and mixed-modality generation and understanding:

• Tokenization: Text is tokenized using a standard BPE or SentencePiece vocabulary; images are typically tokenized via VQ-VAE or advanced quantizers (bitwise, hierarchical, or continuous representations); audio and actions are mapped via specialized tokenizers adapted to their respective domains (Peng et al., 16 Jun 2026, Qin et al., 15 Dec 2025, Lu et al., 2023, Fini et al., 2024).
• Sequence Construction: Token streams are concatenated with modality delimiters (e.g., [SOI]/[EOI], <boi>/<eoi>, etc.) to mark modality boundaries and support interleaved or segmented multimodal streams (Tang et al., 27 Mar 2025, Cheng et al., 25 Jan 2026).
• Objective: The loss is a sum (sometimes weighted) of next-token negative log-likelihood over all tokens, jointly over modalities:

$$\mathcal{L} = -\sum_{t=1}^{T} \log p_\theta(x_t | x_{<t}), \quad x_t \in \bigcup_{\text{modality}} \mathcal{V}_\text{modality}$$

Optionally, auxiliary perceptual or reconstruction losses may be used for specialized tokens (e.g., visual or acoustic alignment) (Cheng et al., 25 Jan 2026).

• Training: Massive datasets of aligned or interleaved modalities are used, frequently with multi-stage instruction tuning, curriculum learning, or progressive data schedules (Tang et al., 27 Mar 2025, Qin et al., 15 Dec 2025, Fini et al., 2024, Lu et al., 2023).

2. Tokenization Strategies and Unification Mechanisms

The design of tokenization is central to effective multimodal AR pre-training. Key strategies include:

• Discrete Visual Tokenization: Images are encoded with VQ-VAE or similar discrete quantizers—sometimes with hierarchical, merged, or bitwise schemes. For example, UniAR employs lookup-free binary spherical quantization (BSQ) to generate bit-vectors, enabling joint semantic and fine-grained detail preservation while dramatically reducing sequence length via spatial and multi-level merging (Peng et al., 16 Jun 2026). STAR introduces high-capacity VQ with 65,536 codebook entries for visual detail (Qin et al., 15 Dec 2025).
• Continuous and Hybrid Representations: MMAR encodes images as continuous tokens via an LDM KL-16 tokenizer and then maps these through a lightweight transformer, avoiding VQ-induced information loss while exploiting an added diffusion head for high-fidelity generation (Yang et al., 2024).
• Unified Token Space: Models such as Unified-IO 2 and AR-Omni design a joint vocabulary, interleaving or fusing text, visual, audio, action, and geometry tokens within a universal sequence. This supports any-to-any mapping and enables a shared semantic space (Lu et al., 2023, Cheng et al., 25 Jan 2026).
• Modality Bridging: UGen employs progressive visual vocabulary activation, incrementally introducing visual token IDs to mitigate modality interference at scale (Tang et al., 27 Mar 2025). VW-LMM projects visual features into the text vocabulary, treating each as a pseudo-one-hot "visual word" (Peng et al., 2024).

A central theme is the push towards minimal, often singular, modality-specific components to achieve true unification. Some models freeze all but the AR backbone (e.g., STAR, Unified-IO 2), some design shared/bridged tokenizer pipelines (e.g., UniAR), while others (e.g., JAM) fuse entire pretrained specialized models via cross-attention or parameter averaging (Aiello et al., 2023).

3. Training Objectives, Data, and Scheduling

Multimodal AR pre-training typically uses large-scale, mixed-modality corpora and a unified, non-contrastive sequence modeling objective:

• Data Mixtures and Schedules: Most frameworks train on text-only, image→text (understanding), and text→image (generation), and, where applicable, audio or action demonstrations, in fixed or scheduled ratios. Some employ batch-mixing, task-aware sampling, or progressive curriculum (e.g., UGen's 3:2:5 data ratio and progressive activation) (Tang et al., 27 Mar 2025, Qin et al., 15 Dec 2025, Liu et al., 30 Dec 2025).
• Loss Formulation: The base loss is cross-entropy over the joint concatenated sequence. Additional tasks may require regression (for pixels or control actions), contrastive alignment (as in COSMO), or auxiliary regularization on discrete/continuous visual tokens (Wang et al., 2024, Yang et al., 2024).
• Hybrid and Staged Training: Several systems decouple learning into multiple stages—e.g., cold start on text, phase-wise activation of visual codes (UGen), separate visual-word fitting stages (VW-LMM), or curriculum from understanding to generation to editing (STAR) (Tang et al., 27 Mar 2025, Qin et al., 15 Dec 2025, Peng et al., 2024).

The interaction between modalities, balancing per-modality loss scaling and addressing modality imbalance in training, is an active area of empirical optimization. For instance, AR-Omni introduces task-aware token-level weighting to mitigate long-modality domination, and SWIN-norm blocks to stabilize deep AR training (Cheng et al., 25 Jan 2026).

4. Unified Generation, Understanding, and Action

Multimodal AR pre-training enables a rich array of downstream capabilities within a single backbone:

• Text and Vision Tasks: Unified models can perform visual question answering (VQA), captioning, text→image synthesis, image→text captioning, document parsing, and fine-grained retrieval. UGen, STAR, AIMv2, and MMAR demonstrate state-of-the-art or near-SOTA results across text and vision evaluation suites (Tang et al., 27 Mar 2025, Qin et al., 15 Dec 2025, Fini et al., 2024, Yang et al., 2024).
• Mixed-modality Generation: A single model can interleave text with generated images in one stream, supporting cases such as document chunking or illustrative dialogue ("generate paragraphs interspersed with illustrative images" (Tang et al., 27 Mar 2025)).
• Audio and Speech: Unified-IO 2 and AR-Omni extend AR modeling to audio, supporting both speech-to-text and text-to-speech in streaming fashion using discrete codec or VQ quantization (Lu et al., 2023, Cheng et al., 25 Jan 2026).
• Robotics and Embodied Action: FACT (flow-matching action tokenizer) discretizes continuous action trajectories for AR modeling, closing the gap between reasoning and precise control, as evaluated on the ERIQ robotic reasoning benchmark (Liu et al., 30 Dec 2025). PACT and Unified-IO 2 extend these concepts to embodied navigation, mapping, and multi-sensor fusion (Bonatti et al., 2022, Lu et al., 2023).

State-of-the-art systems are capable of maintaining high-fidelity signal generation and robust understanding without sacrificing performance on either front, a significant improvement over earlier "discriminative" or unimodal AR pipelines.

5. Architectural and Algorithmic Innovations

Recent years have seen a proliferation of architectural advances enabling true AR multimodal unification:

• Layer Freezing and Progressive Stacking: STAR freezes a pretrained multimodal AR model and progressively stacks new AR layers to extend generation/editing capabilities while preserving core understanding (Qin et al., 15 Dec 2025).
• Shared Context-Visual Tokenization: UniAR's lookup-free, bitwise quantizer and multi-level feature stacking allow exploitation of extremely large implicit codebooks without splitting modality spaces or introducing re-encoding, resulting in marked speed and fidelity gains (Peng et al., 16 Jun 2026).
• Decoupled Diffusion Heads: MMAR and some other architectures decouple the AR backbone from the diffusion-based image generation process, allowing for lossless, numerically stable, and efficient multimodal prediction (Yang et al., 2024).
• Cross-Modality Fusion and Parameter Sharing: JAM, COSMO, and similar architectures introduce explicit layer or attention-fusion strategies (e.g., cross-attention between text and image transformer stacks), weight averaging, or compact parameter-efficient multimodal adapters (Aiello et al., 2023, Wang et al., 2024).
• Modality-bridging Objectives: VW-LMM enforces a direct mapping of vision features into the LLM vocabulary space, aligning semantic distributions across modalities (Peng et al., 2024). VDEP formulates hidden state reconstruction of image embeddings as a hybrid metric in the AR loss (Li et al., 13 Feb 2025).
• Mixed Paradigm Training: Some systems generalize the UL2 Mixture-of-Denoisers (MoD) paradigm to multimodal denoising and generation (Unified-IO 2), or design multi-phase, task-progressive curriculums (Lu et al., 2023, Qin et al., 15 Dec 2025).

6. Empirical Results, Scaling, and Limitations

Across a wide spectrum of benchmarks, multimodal AR pre-training sets new performance bars and highlights important scaling behaviors:

System	Model Size	Notable SOTA Results	Key Innovations
UGen (Tang et al., 27 Mar 2025)	1.1B	+13.3% over vanilla AR, matched/exceeded specialized models	Progressive vocab activation
STAR (Qin et al., 15 Dec 2025)	~B-scale	GenEval=0.91, DPG-Bench=87.44, ImgEdit=4.34	Task-progressive stacking, high-cap VQ
UniAR (Peng et al., 16 Jun 2026)	8B+	GenEval=0.85, ImgEdit-Bench=3.73, competitive OCR/VQA	Shared BSQ tokenizer, parallel bitwise
AIMv2 (Fini et al., 2024)	3B	IN-1K acc=89.5% (frozen), SOTA/competitive open-vocab/segm/VQA	Prefix-attn ViT, joint AR decoder
Unified-IO 2 (Lu et al., 2023)	6.8B	67.0 on GRIT, SOTA in image/audio/video/action gen and understanding	Mixture-of-denoisers, universal tokens
MMAR (Yang et al., 2024)	7B	AVE@18Und=46.52, FID=17.1	Lossless continuous visual tokens
AR-Omni (Cheng et al., 25 Jan 2026)	7B	Real-time TTS (RTF=0.88), competitive text/image/speech gen	Modality reweighting, perceptual loss
VW-LMM (Peng et al., 2024)	7B	Outperforms contemporary VQA and toolkit baselines	Visual-word projection/bridging

Scaling up model and data size almost universally improves performance, with no signs of saturation up to multi-billion parameter scale (Fini et al., 2024, Yang et al., 2024). Empirical ablations across methods consistently show that:

• Curriculum in activation, mixing, or stacking improves multi-task joint performance (Tang et al., 27 Mar 2025, Qin et al., 15 Dec 2025).
• Parallel and hierarchical visual tokenization (multi-level, spatial merging, or bitwise grouping) reduces AR sequence length and boosts generation throughput without sacrificing fidelity (Peng et al., 16 Jun 2026).
• Balancing supervision across modalities (e.g., loss reweighting, KL-calibrated alignment) is essential for stability and fair capacity allocation (Cheng et al., 25 Jan 2026).
• Visual quantization remains a performance bottleneck for extremely fine-grained tasks; continuous or hybrid approaches (e.g., MMAR) reduce this but at increased computational/architectural cost (Yang et al., 2024).
• Interleaved training with additional modalities (audio, action, dense/structured geometry) is feasible and beneficial when paired with universal token design and careful numerical/mechanism stabilization (Lu et al., 2023, Liu et al., 30 Dec 2025, Bonatti et al., 2022).

7. Outlook, Open Challenges, and Trends

Recent advances in multimodal autoregressive pre-training have shifted the landscape towards truly unified, instruction-following, and generative-capable agents that handle text, vision, audio, and action in a single model. Current trends include:

• Scaling to larger models, datasets, and more modalities, with continued gains in zero-/few-shot and in-context learning (Fini et al., 2024, Lu et al., 2023).
• Tokenization Advances: Novel tokenizer designs (e.g., lookup-free bitwise codes, continuous-discrete hybrids) seek to unify context across tasks and minimize information loss and sequence length (Peng et al., 16 Jun 2026, Yang et al., 2024).
• Principled Curriculum and Loss Balancing: As models absorb increasingly diverse modalities and tasks, curriculum mechanisms (progressive activation, staged stacking, multi-phased tuning) and sophisticated loss reweighting become critical for convergent, sample-efficient training (Tang et al., 27 Mar 2025, Cheng et al., 25 Jan 2026).
• Fine-Grained Editing, World Knowledge, and Embodied Control: New models extend beyond passive understanding and generation, supporting dynamic editing, complex world-level reasoning, and embodied action with unified AR objectives (Qin et al., 15 Dec 2025, Liu et al., 30 Dec 2025).
• Stabilization and Efficient Scaling: Mechanisms to address numerical instability (QK normalization, residual post-norm, v-pred diffusion) and context/window packing are increasingly central (Unified-IO 2, MMAR) (Yang et al., 2024, Lu et al., 2023).

Open challenges remain: optimizing cross-modality alignment at scale, minimizing information loss in quantization, scaling to longer/more complex interleaved sequences, and defining robust user interfaces for multi-turn, multi-modal dialogue. However, the unification of autoregressive pre-training frameworks across modalities is now the dominant recipe for broad, scalable, and instruction-following multimodal AI.