---
title: 'VA-GPT: Unified Multimodal Autoregressive Models'
url: https://www.emergentmind.com/topics/va-gpt
type: topic
---

# VA-GPT: Unified Multimodal Autoregressive Models

Searching arXiv for papers relevant to “VA-GPT” and closely related usages of the term.
Search query: VA-GPT OR VARGPT OR BIM-GPT OR VLN-GPT
Search query: "Generative Pre-trained Transformer for Vietnamese Community-based COVID-19 Question Answering"
Search query: "BIM-GPT: a Prompt-Based Virtual Assistant Framework for BIM Information Retrieval"
Search query: "Vision-and-Language Navigation Generative Pretrained Transformer"
VA-GPT denotes a non-uniform line of GPT-based systems whose common feature is the use of autoregressive language-modeling machinery as the control core for multimodal understanding, generation, retrieval, or decision-making. In the most direct sense represented in the cited literature, VA-GPT corresponds to **VARGPT**, a multimodal large language model that unifies visual understanding and visual generation in a single autoregressive framework [2501.12327]. In broader adjacent usage, the label also aligns with GPT-based **virtual assistant** frameworks for domain-specific information retrieval, decoder-only models for **vision-and-language navigation**, and Vietnamese GPT-based generative question answering systems, although one such work explicitly does **not** introduce a formal method named “VA-GPT” [2310.14602].

## 1. Terminological scope and usage

The designation is not standardized across the cited works. One paper explicitly presents **VARGPT (or VA-GPT)** as a unified multimodal autoregressive model for visual understanding and generation [2501.12327]. Other papers instantiate related patterns without using the same name in a formally identical sense. **BIM-GPT** defines a prompt-based virtual assistant for BIM information retrieval, where GPT acts as the language engine for query interpretation, summarization, and question answering [2304.09333]. **VLN-GPT** applies a GPT-2-style decoder to model trajectory sequence dependencies in vision-and-language navigation [2405.16994]. The Vietnamese COVID-19 question answering study evaluates GPT-2 for Vietnamese community-based generative QA, but the work explicitly does **not** introduce a formal framework named “VA-GPT” [2310.14602].

| Work | Domain | Relation to VA-GPT |
|---|---|---|
| VARGPT | Multimodal understanding and image generation | Direct usage: “VARGPT (or VA-GPT)” |
| BIM-GPT | BIM information retrieval | GPT-based virtual assistant framework |
| VLN-GPT | Vision-and-language navigation | GPT-style generative policy |
| Vietnamese GPT-2 QA | Community-based COVID-19 QA | Closely related implementation, not a formal VA-GPT proposal |

This suggests that “VA-GPT” functions less as a single canonical architecture than as a family of GPT-centered systems in which autoregressive sequence modeling replaces or reduces task-specific engineering. That interpretation remains an inference from the cited works rather than a universal naming convention.

## 2. Core formulation in VARGPT

In its most explicit form, VA-GPT is a **unified multimodal large language model** that performs both **visual understanding** and **visual generation** inside one autoregressive system [2501.12327]. The central methodological claim is that these two capabilities need not be split between an MLLM for comprehension and a separate diffusion model for synthesis. Instead, the model uses **next-token prediction** for understanding and **next-scale prediction** for image generation.

For visual understanding, the model follows a LLaVA-style autoregressive formulation. Image features $\mathbf{H}^{img}$ and query text $\mathbf{X}^{query}$ are fed to the LLM, which predicts the next textual token as

$$
\mathbf{Y}^{txt}_t \sim p_{\theta}(\mathbf{Y}^{txt}_t \mid \mathbf{X}^{img}, \mathbf{X}^{query}, \mathbf{Y}^{txt}_{<t}).
$$

For visual generation, the image is converted into multi-scale discrete feature maps

$$
(\mathbf{R}_1, \mathbf{R}_2, \cdots, \mathbf{R}_K),
$$

and the model generates them sequentially as

$$
\mathbf{R}_t \sim p_{\{\theta,\phi\}}(\mathbf{R}_t \mid \mathbf{X}^{query}, \mathbf{R}_{<t}).
$$

The conceptual distinction is precise. **Next-token prediction** governs multimodal understanding and textual response generation, whereas **next-scale prediction** governs image synthesis by predicting coarse visual structure first and finer scales later. This differs from flat discrete token generation for images and places VARGPT closer to a VAR-style visual autoregressive process than to diffusion-based generation.

A defining property is **mixed-modal input/output**. The model can accept text and/or image inputs, answer in text, and also emit image-generation tokens during the same dialogue. The token inventory includes `<image>` for input images and `<image_gen_start>`, `<image_gen>`, and `<image_gen_end>` for image-generation segments. When `<image_gen_start>` is emitted, subsequent `<image_gen>` slots are routed through the visual-generation pathway.

## 3. Architecture and unified training pipeline

VARGPT explicitly extends **LLaVA-1.5-7B**. Its understanding path uses a **Vicuna-7B-v1.5** language model, a **CLIP ViT/14 visual encoder**, and a projector that maps image features into the LLM embedding space [2501.12327]. For generation, it adds a **2B-parameter visual decoder** with **30 Transformer layers** and **two extra visual feature projectors**. The paper states that these extra parameters avoid “knowledge conflict” between text decoding and image generation. The visual decoder uses **block causal attention** rather than standard LLM causal attention, and absolute positional encoding is added for visual tokens.

Training proceeds in **three stages**. In **Stage 1: Pretraining**, the model uses ImageNet-derived data to construct **1.28M single-round dialogue samples**. All parameters are frozen except the two generation projectors, with the stated goal of aligning textual and visual feature spaces. In **Stage 2: SFT for Visual Understanding**, the model unfreezes the **LLM** and the **projector for visual-encoder output**, and trains on **1.18M mixed samples**, including **LLaVA-1.5-665K**, **508K** samples from **LLaVA-OneVision**, and **5K samples** from **ImageNet-Instruct-130K**. In **Stage 3: SFT for Visual Generation**, the model unfreezes the **visual decoder** and the **two generation projectors**, freezes everything else, and trains on **1.4M** instruction pairs from **ImageNet-Instruct-1270K**.

The generation process also incorporates **classifier-free guidance (CFG)**. Using projected conditional features $\mathbf{H}^g$ and Gaussian noise $\mathbf{H}^e$, the appendix writes the guidance mechanism as

$$
\begin{aligned}
\mathbf{R}_t &\sim p_{\{\theta,\phi\}}(\mathbf{R}_t \mid \mathbf{H}^g, \mathbf{R}_{<t}); \\
\mathbf{R}_e &\sim p_{\{\theta,\phi\}}(\mathbf{R}_t \mid \mathbf{H}^e, \mathbf{R}_{<t}); \\
\mathbf{R}_t &= (1+\lambda)\mathbf{R}_t - \lambda \mathbf{R}_e.
\end{aligned}
$$

The overall training objective is described operationally rather than as a single combined loss equation: maximize next-token likelihood for understanding and next-scale likelihood for generation.

## 4. Empirical performance and limitations

On vision-centric understanding benchmarks, VARGPT is reported to outperform **LLaVA-1.5** while also supporting generation [2501.12327]. In the benchmark table, **VARGPT (7B+2B)** reaches **MMBench 67.6** versus **62.7**, **SEED 67.9** versus **65.4**, **MMMU 36.44** versus **35.24**, **MME 1488.8** versus **1480.1**, and improves **POPE** across settings to **84.40 / 85.90 / 87.37** versus **83.60 / 85.77 / 86.97**. On the VQA-style table it reaches **GQA 62.3**, **TextVQA 54.1**, **VQAv2 78.4**, **SciQA-img 80.1**, **OKVQA 55.8**, and **VizWizQA 56.83**; the authors explicitly highlight the **12.2-point improvement** on **SciQA-img** over LLaVA-1.5.

| Benchmark | VARGPT | LLaVA-1.5 |
|---|---:|---:|
| MMBench | 67.6 | 62.7 |
| MMMU | 36.44 | 35.24 |
| MME | 1488.8 | 1480.1 |
| TextVQA | 54.1 | 48.8 |
| SciQA-img | 80.1 | 67.9 |

For generation, evaluation uses **50,000 text instructions** and reports **FID** and **CLIP score**. The full model achieves **FID 12.6** and **CLIP 27.4**. Ablations indicate that the full three-stage pipeline is necessary: removing stage 3 worsens performance to **FID 20.1** and **CLIP 20.2**. On the understanding side, freezing the projector in stage 2 reduces results to **MMMU 35.19 / MME 1452.0**, while freezing the LLM reduces them further to **33.51 / 1392.5**.

The paper also states clear limitations. Generation is constrained by an **ImageNet-based** corpus that is smaller and lower-quality than the large datasets used by advanced diffusion systems such as **SDv2.1** or **FLUX**, so generation quality still lags behind top diffusion models. The current model supports only **256×256** image generation, and some nuanced instruction details are not always faithfully reflected in outputs. These limitations are important because they delimit the extent of the claimed unification: the model is multimodally unified, but not yet competitive with the best specialized image generators.

## 5. Related VA-GPT-style systems in other domains

A distinct but conceptually related use of GPT appears in **BIM-GPT**, which is defined as a **prompt-based virtual assistant (VA) framework** for BIM information retrieval [2304.09333]. Its architecture has **three modules**—a **Web-based User Interface (UI)**, an **NLP module**, and a **Data Management (DM) module**—with a **prompt manager** orchestrating the workflow. The dynamic prompt template has **five components**: **System**, **Relevant Database Information**, **Task Instruction**, **Few-shot Examples**, and **User**. The framework decomposes natural-language understanding into intent classification, category identification, parameter identification, and value recognition. On the **BINLQ dataset**, reported accuracies include **83.5%** and **99.5%** for text-classification labels in zero-shot and few-shot settings, respectively; **98.6%** and **99.5%** for object category classification; and **86.7%** and **95.1%** for predicted/extracted values combined. The hospital-building prototype further validates retrieval, summarization, general BIM question answering, and 3D visualization.

In **VLN-GPT**, GPT is used not as a retrieval assistant but as a **decoder-only navigation policy** for **Vision-and-Language Navigation** [2405.16994]. The model uses **Sentence-BERT** for instruction encoding, **ViT** for panoramic observations, and a **GPT-2 base** decoder to model serialized **returns, states, and actions**. The trajectory is written as

$$
\tau_T=(r_0,s_0,a_0,r_1,s_1,a_1,\ldots,r_T,s_T,a_T),
$$

and action prediction is conditioned on the full past trajectory rather than on a separate history encoder. Training is separated into **offline pre-training with imitation learning** and **online fine-tuning with reinforcement learning**. On **Room-to-Room (R2R)**, the reported results include **SAP 78 seen / 72 unseen**, and main navigation results of **76** seen SR, **72** seen SPL, **65** unseen SR, and **61** unseen SPL, exceeding the listed PREVALENT and RecBERT baselines.

The Vietnamese COVID-19 QA study occupies

Source: https://www.emergentmind.com/topics/va-gpt