VLUAS: Unified Vision-Language Autoregressive Model

• VLUAS is a unified multimodal paradigm that reformulates vision and language processing as autoregressive token prediction to improve fine-grained alignment.
• It integrates advanced tokenization and transformer architectures by interleaving visual patches and text tokens, facilitating both understanding and generation tasks.
• Empirical results show VLUAS models achieve state-of-the-art performance in visual generation, segmentation, and detection, validating the unified training approach.

Vision-Language Unified Autoregressive Supervision (VLUAS) is a paradigm that reformulates multimodal model training by casting both visual and linguistic modalities as generative targets under a joint autoregressive likelihood. Instead of treating visual content as conditional context for language modeling, VLUAS compels models to predict both image and text tokens within a single unified output stream. By doing so, it overcomes the coarse-grained representation bottlenecks of conventional vision-language architectures and enables models to excel at both visual understanding and visual generation, as well as downstream vision-centric tasks such as segmentation, detection, and grounding.

1. Mathematical Formulation and Principle

VLUAS formalizes multimodal learning as next-token or next-scale prediction over interleaved visual and textual tokens. Let $T = \{t_1,\dots,t_N\}$ denote a sequence of text tokens, and $V = \{v_1,\dots,v_M\}$ denote a sequence of discrete visual tokens obtained by quantization. The model predicts the flattened joint sequence $S=(s_1,\dots,s_{L})$, where $L=N+M$ and $s_i \in T \cup V$, via the autoregressive factorization:

$$P(S;\theta) = \prod_{i=1}^L P(s_i\mid s_{<i};\theta)$$

The training objective is a weighted sum of cross-entropy losses for text and image tokens:

$$\mathcal{L}_{\mathrm{VLUAS}} = \mathcal{L}_{\mathrm{text}} + \lambda\;\mathcal{L}_{\mathrm{image}}$$

where

$$\mathcal{L}_{\mathrm{text}} = -\sum_{i \in I_t} \log P(t_i | s_{<i};\theta), \qquad \mathcal{L}_{\mathrm{image}} = -\sum_{j \in I_v} \log P(v_j | s_{<j};\theta)$$

The hyperparameter $\lambda$ governs the balance between the visual and linguistic supervision signals. In advanced extensions, such as for dense prediction, VLUAS supports multi-label autoregressive supervision (NTP-M), formulating segmentation, counting, and detection as Bernoulli next-token prediction over patch-wise label sets (Wei et al., 27 Jan 2026).

For visual generation, VLUAS integrates a next-scale prediction paradigm. For example, VARGPT employs a cross-entropy loss over multi-scale VQVAE codewords for each scale $s$ as

$$L_{\text{scale}}(\theta,\phi,P_{\text{gen}}) = -E_{(X^{\text{query}},R)}\sum_{s=1}^S \log p_{\theta,\phi}(R_s \mid X^{\text{query}},R_{<s})$$

with the total loss composed of text, scale, and alignment components (Zhuang et al., 21 Jan 2025).

2. Model Architectures and Tokenization

VLUAS requires converting both modalities into a common autoregressive token space. Most approaches use:

• Visual tokenization: Images are quantized into discrete patchwise tokens via advanced vector quantization—e.g., residual VQ-VAE (codebook size up to $K=150{,}000$ for Youtu-VL, $K=8{,}192$ for ASVR), or continuous VAE tokens in UniFluid (Wei et al., 27 Jan 2026, Wang et al., 10 Jun 2025, Fan et al., 17 Mar 2025).
• Text tokenization: Language remains discrete (BPE/SentencePiece vocabulary).
• Integration in the transformer: Tokens are concatenated as one sequence and processed by a causal, decoder-only transformer (e.g., LLaMA-2-7B, Qwen2.5-VL-3B, Vicuna-7B, Gemma-2B). No distinction is made between modalities at the decoder level; attention and feed-forward blocks operate over the unified stream.
• Modality/position encoding: Positional and modality tags may be used to annotate tokens, but the core supervision is unified.

Distinct architectural variations exist:

• VARGPT separates “understanding projector” and “generation projector,” feeding image features into the LLM for text prediction and into a visual decoder for image generation (Zhuang et al., 21 Jan 2025).
• Youtu-VL fuses high-resolution SigLIP-2 and DINOv3 features, then quantizes them via index backpropagation (Wei et al., 27 Jan 2026).
• UniFluid uses continuous VAE latents with a diffusion head for image token regression (Fan et al., 17 Mar 2025).
• Unified-IO 2 tokenizes text, images, audio, and coordinates into a shared vocabulary for a universal encoder-decoder (Lu et al., 2023).

3. Training Strategies and Loss Engineering

Canonical VLUAS training involves multiple stages:

• Vision tower alignment: Vision transformers are pre-trained with contrastive (CLIP-style) and reconstruction losses $\mathcal{L}_{\mathrm{contra}}$ and $\mathcal{L}_{\mathrm{recon}}$ until discrete tokens are both semantically and structurally aligned with text (Wu et al., 2024).
• Autoregressive multimodal pretraining: The unified model is trained end-to-end with next-token prediction over mixed sequences, supporting both understanding and generation tasks (Zhuang et al., 21 Jan 2025).
• Instruction-tuning: Instruction datasets covering multimodal Q&A, image captioning, and generation are used to calibrate mixed-modal performance (Lu et al., 2023, Zhuang et al., 21 Jan 2025).
• Loss balancing: Empirical guidance shows that weighting parameters (e.g., $\lambda \in [0.005, 1]$) must be carefully tuned to prevent collapse in one direction; over-weighting understanding suppresses generation (FID $>50$), while under-weighting degrades comprehension (Fan et al., 17 Mar 2025).
• Task curricula: Tasks are interleaved to force models to switch seamlessly between modalities (e.g., next-token text, next-scale image, or multi-label vision-centric outputs) (Zhuang et al., 21 Jan 2025, Wei et al., 27 Jan 2026).

For vision-centric tasks, Youtu-VL uses unified coordinate vocabularies (2,048 tokens per axis) and casts bounding box and pose outputs as direct token sequence predictions (Wei et al., 27 Jan 2026). Dense prediction tasks (semantic segmentation, depth) are implemented via patch-wise logit aggregation and argmax over category tokens (no extra decoders).

4. Paradigms: Understanding, Generation, and Mixed Tasks

VLUAS fully integrates both vision understanding (VQA, reasoning, captioning) and generation (text-to-image synthesis, instruction-to-image) within a single model and training objective.

• Understanding: Standard next-token autoregressive language modeling over text answers conditioned on visual tokens or context embeddings.
• Generation: Next-scale prediction over VQVAE codebook tokens (VARGPT), next-token production of quantized image patches (VILA-U, Youtu-VL), or continuous-token diffusion head (UniFluid).
• Mixed-task sequencing: Models learn to switch between text and image output modes, triggered by special tokens (e.g., <image_gen_start> in VARGPT) (Zhuang et al., 21 Jan 2025). Sequences can encode multi-turn dialogues where both modalities are interleaved.

Vision-centric tasks are mapped to token sequences (bounding box, segmentation, counting) using predefined vocabularies; the same next-token loss applies.

5. Empirical Performance and Ablation Insights

VLUAS-trained models achieve state-of-the-art, or near-SOTA, performance across both generalist and vision-centric benchmarks.

• Visual generation: VARGPT (FID=12.6, CLIP=27.4), VILA-U (FID=12.8 at 256px, 7.7 at 384px on MJHQ), UniFluid (FID=7.2 at best λ) (Zhuang et al., 21 Jan 2025, Wu et al., 2024, Fan et al., 17 Mar 2025).
• Visual understanding / Q&A: Youtu-VL outperforms or matches 4–8 B VLMs on 45 multimodal tasks; ASVR yields +5.0 average points over LLaVA-1.5 on 14 benchmarks (Wei et al., 27 Jan 2026, Wang et al., 10 Jun 2025). VARGPT scores up to 67.6 on MMBench and 54.1 on TextVQA (Zhuang et al., 21 Jan 2025).
• Vision-centric tasks: Youtu-VL achieves 54.2 mIoU (ADE20K segmentation), 92.7% δ₁ (NYUv2 depth), 47.1 mAP (COCO detection), and >91.8% (RefCOCO visual grounding) (Wei et al., 27 Jan 2026).
• Ablation studies: Removal of VLUAS supervision on visual targets causes a pronounced drop in fine-grained comprehension and generation quality, confirming the necessity of patch-level autoregressive losses. Using appearance-only tokenization degrades semantic understanding, while semantic token reconstruction boosts alignment and performance (Wang et al., 10 Jun 2025).

6. Extensions, Limitations, and Open Directions

Extensions of VLUAS include:

• Continuous-visual token models: UniFluid demonstrates that continuous VAE tokens can be handled within the next-token framework, albeit with careful loss balancing (Fan et al., 17 Mar 2025).
• Multi-label next-token prediction: Youtu-VL generalizes the framework for dense-multilabel tasks with NTP-M (Wei et al., 27 Jan 2026).
• Action modeling: Unified Vision-Language-Action frameworks (UniVLA, AutoVLA) autoregressively predict intermixed vision, language, and action tokens for embodied and autonomous systems (Wang et al., 24 Jun 2025, Zhou et al., 16 Jun 2025).
• Scaling and curriculum: Staged curriculum training, larger codebooks, and adaptive loss weighting are key factors; current limitations persist for fine-granularity (small/thin objects, high-resolution), geometry-aware tasks (pose, depth), and high-level symbolic reasoning (Wei et al., 27 Jan 2026, Wu et al., 2024).

Future directions comprise adaptive codebook learning for multi-scale visual representation, vector-quantized regression for continuous attributes, unified autoregressive supervision for video-language, and integration with contrastive learning for zero-shot transfer (Wei et al., 27 Jan 2026).

7. Significance and Impact

VLUAS provides a principled and empirically validated solution for unified multimodal modeling. By engaging autoregressive supervision over both linguistic and visual outputs, models attain dense, fine-grained grounding and retain generative capabilities without specialized decoders or modality-specific heads. This approach enables seamless task switching, improves representation granularity, and facilitates end-to-end optimization for mixed-modal benchmarks and vision-centric applications. VLUAS frameworks like Youtu-VL, VARGPT, VILA-U, and UniFluid establish a robust foundation for the development of comprehensive generalist agents in AI research and practice (Wei et al., 27 Jan 2026, Zhuang et al., 21 Jan 2025, Wu et al., 2024, Fan et al., 17 Mar 2025).