---
title: Autoregressive Multimodal LLMs
url: https://www.emergentmind.com/topics/autoregressive-multimodal-large-language-models-mllms
type: topic
---

# Autoregressive Multimodal LLMs

Autoregressive Multimodal Large Language Models (MLLMs) are unified generative models that extend the autoregressive next-token prediction paradigm of large language models (LLMs) to encompass multiple data modalities—such as text, images, audio, video, motion, and coordinate sequences. These models are designed to ingest, process, and generate complex multimodal sequences in a left-to-right causal fashion, leveraging shared or extended vocabularies and unified embedding spaces. Key innovations span from unified tokenization schemes (spatial, discrete, or continuous) and modality-specific architectural adaptations to dynamic reasoning mode switching, automated cross-modal grounding, and scalable training regimes that preserve both general language and modality-specific capabilities.

## 1. Architectural Foundations and Unified Tokenization

Autoregressive MLLMs rely on a single backbone Transformer, typically decoder-only, whose core next-token prediction objective is extended to accommodate new modalities via unified or specialized tokenization approaches. Depending on the model, input modalities are discretized (e.g., VQ‐VAE visual tokens, "visual words"), projected into the language space via adapters or X2L interfaces, or represented as continuous embeddings (e.g., causal VAE latents for motion). Token streams are concatenated for joint attention, often with specialized boundary, modality, or task tokens ([2412.04332], [2512.10554]).

Major tokenization approaches include:

| Approach                    | Description                                 | Key Models              |
|-----------------------------|---------------------------------------------|-------------------------|
| Spatial Discretization      | VQ‐VAE/VQGAN codes, patch tokens            | Liquid, AR-Omni, JAM    |
| Visual Words                | Map image patches to text-vocab distributions| VW-LMM                  |
| Diffusion Timestep Tokens   | Recursive, order-sensitive visual language  | DDT-LLaMA               |
| Point Tokens/Embeddings     | Continuous/embedded waypoints for trajectories | AutoTraces            |
| Audio/Speech Quantization   | Single-codebook acoustic tokens             | AR-Omni, Llama-AVSR     |
| Causal Continuous Latents   | Streaming continuous embeddings for motion  | LLaMo                   |
| Explicit Spatial Tokens     | Grid and offset tokens for 2D reasoning     | GETok                   |

This design allows a unified decoder to process arbitrary interleavings of text and non-text modalities, supporting "any-to-any" generation by virtue of a joint vocabulary and/or interleaved sequence—without requiring modality-specific decoders ([2601.17761], [2412.04332]).

## 2. Modality Integration and Representation Strategies

Modality integration is achieved through a combination of architectural and representational enhancements:

- **Adapters and Projection Layers:** Frozen modality-specific encoders (e.g., CLIP, Whisper, AV-HuBERT, VQVAE) map raw data into fixed-length vectors, which are projected into the LLM token space via lightweight adapters or multi-layer perceptrons ([2408.02503], [2409.12319]).
- **Modality-Specific Decoder Branches and MoEs:** Modality-dependent "mixture-of-experts" blocks (MoT, Visual Attention Experts) enable adaptation of attention/QKV projections for distinct token types, while preserving the frozen parameters for core language modeling ([2602.12370], [2406.18193]).
- **Specialized Token/-Type Vocabularies:** Dedicated task, grounding, spatial, or reasoning tokens are appended to signal modality boundaries, support downstream routing (UnifiedMLLM), or enable fine-grained spatial control (GETok) ([2408.02503], [2512.10554]).
- **Hybrid Discrete-Continuous Reasoning:** Certain autoregressive MLLMs jointly generate both discrete tokens (text, special markers) and continuous embeddings ("visual thoughts," motion latents) under a unified likelihood ([2602.06040], [2602.12370]).

This hybridization unlocks the model's ability to ground, reason, and generate in modality-appropriate representations, while maintaining autoregressive tractability.

## 3. Training Paradigms and Supervision

Training autoregressive MLLMs requires both multi-modal data and supervisory objectives that bridge language and non-language domains:

- **Unified Losses:** Most frameworks sum cross-entropy losses over all tokens (text, modality-specific) in the sequence. Continuous modalities typically introduce regression (L2, MSE, or flow-matching) losses on continuous tokens ([2602.06040], [2602.12370]).
- **Staged Curriculum:** Progressive pipelines are common—starting with modality alignment (captioning/text-only pretraining), followed by instruction tuning, and, in some cases, expert fine-tuning or reinforcement learning for grounding/localization ([2408.02503], [2512.10554], [2403.07720]).
- **Adapters/Parameter-Efficient Updates:** LoRA, MoE, or head-specific updates enable models to learn modality integration and generation while largely freezing core LLM or encoder parameters, guarding against catastrophic forgetting ([2409.12319], [2503.06542]).
- **Automated Chain-of-Thought and Reasoning Mode Curation:** For tasks needing compositional or spatio-temporal reasoning, automated generation of chain-of-thought traces (via auxiliary VLMs) or dynamic reasoning-mode supervision (text-only, vision-only, interleaved) guides the model's internal rollout strategy ([2603.07989], [2602.06040]).

## 4. Inference Procedures and Interaction Dynamics

Inference in autoregressive MLLMs hinges on task- and modality-aware decoding, including:

- **Stability vs. Creativity via Decoding State:** Finite-state decoding automata flexibly choose between deterministic (greedy) and generative (sampling) modes, essential for traversing tasks such as transcription (ASR/TTS), open-ended generation (T2I), and interactive dialog ([2601.17761]).
- **Dynamic Mode Switching:** Hybrid models (e.g., SwimBird) learn to switch among pure text, pure vision, and interleaved modes, deciding positionally when to emit discrete tokens or continuous embeddings based on the input and prompt ([2602.06040]).
- **Task Routing and Expert Selection:** Models such as UnifiedMLLM emit explicit task and grounding tokens that are parsed by a routing function, which then dispatches context and arguments to downstream expert modules (classifiers, segmenters, local image editors) ([2408.02503]).
- **Spatial and Temporal Chaining:** For tasks such as trajectory forecasting and motion generation, sequence tokenization (with point or latent tokens) ensures that each output step is conditioned only on the causal past (including visual scene, prior predictions, and goal metadata), supporting flexible horizon and long-range temporal coherence ([2603.07989], [2602.12370]).

## 5. Empirical Performance and Capabilities

Autoregressive MLLMs have demonstrated strong performance across a spectrum of multimodal tasks, including:

- **Multimodal Understanding:** Vision-language understanding (VQA, captioning, OCR, grounding) at or above the level of established baselines. For instance, LIQUID achieves 68.0 VQAv2 and 56.1 GQA scores in the zero-shot setting ([2412.04332]).
- **Multimodal Generation:** High-fidelity text-to-image (e.g., Liquid FID=5.47, DDT-LLaMA GenEval=0.66), speech synthesis (AR-Omni real-time factor=0.88), streaming motion generation (>30 FPS in LLaMo), and flexible-length trajectory generation (AutoTraces IEAcc=99.92%) ([2412.04332], [2504.14666], [2601.17761], [2603.07989]).
- **Spatial Reasoning and Grounding:** GETok significantly outperforms patch/box representations for referring tasks (Acc@0.5 ≈ 88.2%), while grid/offset tokens yield precise, iterative localization ([2512.10554]).
- **Task Generalization and Scalability:** UnifiedMLLM attains SOTA cIoUs on referring segmentation (RefCOCO=76.3%), multi-task alignment (Joint Autoregressive Mixture, Liquid), and task extensibility via modular expert routing ([2408.02503], [2309.15564]).
- **Robustness and Efficiency:** Models employing efficient adapters, frozen encoder/backbone strategies, or unified embedding spaces show reduced catastrophic forgetting and lower compute/memory cost (e.g., ARMOR: only ~0.7B new params, order-of-magnitude less compute vs. unified from-scratch) ([2503.06542]).

## 6. Limitations and Prospective Directions

Current limitations of autoregressive MLLMs include:

- **Inference Cost and Sequence Lengths:** Incorporation of large VLM/LLM backbones and long modality streams can cause latency bottlenecks, especially in real-time or streaming contexts ([2603.07989], [2601.17761]).
- **Quantization and Fidelity Gaps:** Discrete codebook-based approaches may suffer from reconstruction artifacts or insufficient granularity (noted for images, speech, and motion), though diffusion-timestep tokens and continuous latent models mitigate some of these issues ([2504.14666], [2602.12370]).
- **Physics and Dynamic Constraint Oversight:** Models generating physical trajectories or motion often do not enforce dynamic or kinodynamic admissibility unless explicitly constrained or regularized ([2603.07989]).
- **Expert Dependence:** Some frameworks require routing to external or expert models for particular tasks, potentially limiting true “unification” ([2408.02503]).
- **Scaling and Cross-Modality Interference:** Earlier models observed that joint training degraded text task performance at small scale, though scaling laws in Liquid show this effect vanishes at larger model sizes ([2412.04332]).

Future directions highlighted include the integration of physics-informed modules, generalized multi-agent contexts, plug-and-play extension to new modalities (video, audio, 3D), and improvements in tokenization or decoder sharing to bridge fidelity gaps relative to diffusion models ([2603.07989], [2412.04332], [2601.17761]). The paradigm of recursive, order-sensitive "visual languages" and learnable spatial token vocabularies opens new avenues for grounded reasoning and RL-compatible multimodal agents ([2504.14666], [2512.10554]).

---

**Selected References**

- AutoTraces: Autoregressive Trajectory Forecasting via Multimodal Large Language Models [2603.07989]
- UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model [2408.02503]
- SwimBird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMs [2602.06040]
- LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens [2602.12370]
- Liquid: Language Models are Scalable and Unified Multi-modal Generators [2412.04332]
- AR-Omni: A Unified Autoregressive Model for Any-to-Any Generation [2601.17761]
- ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability [2503.06542]
- Jointly Training Large Autoregressive Multimodal Models [2309.15564]
- Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens [2504.14666]
- Grounding Everything in Tokens for Multimodal Large Language Models [2512.10554]
- Multi-modal Auto-regressive Modeling via Visual Words [2403.07720]
- MammothModa: Multi-Modal Large Language Model [2406.18193]
- X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages [2305.04160]
- Large Language Models are Strong Audio-Visual Speech Recognition Learners [2409.12319]

Source: https://www.emergentmind.com/topics/autoregressive-multimodal-large-language-models-mllms