VA-GPT: Unified Multimodal Autoregressive Models

Updated 8 July 2026

VA-GPT is a family of GPT-based systems that unifies next-token prediction for text and next-scale prediction for image synthesis in a multimodal framework.
It employs a unified training pipeline with pretraining and staged supervised fine-tuning to align visual and textual feature spaces.
Empirical results show VA-GPT outperforms benchmarks in visual understanding while offering competitive image generation despite current resolution and dataset limitations.

Searching arXiv for papers relevant to “VA-GPT” and closely related usages of the term. Search query: VA-GPT OR VARGPT OR BIM-GPT OR VLN-GPT Search query: "Generative Pre-trained Transformer for Vietnamese Community-based COVID-19 Question Answering" Search query: "BIM-GPT: a Prompt-Based Virtual Assistant Framework for BIM Information Retrieval" Search query: "Vision-and-Language Navigation Generative Pretrained Transformer" VA-GPT denotes a non-uniform line of GPT-based systems whose common feature is the use of autoregressive language-modeling machinery as the control core for multimodal understanding, generation, retrieval, or decision-making. In the most direct sense represented in the cited literature, VA-GPT corresponds to VARGPT, a multimodal LLM that unifies visual understanding and visual generation in a single autoregressive framework (Zhuang et al., 21 Jan 2025). In broader adjacent usage, the label also aligns with GPT-based virtual assistant frameworks for domain-specific information retrieval, decoder-only models for vision-and-language navigation, and Vietnamese GPT-based generative question answering systems, although one such work explicitly does not introduce a formal method named “VA-GPT” (Vo et al., 2023).

1. Terminological scope and usage

The designation is not standardized across the cited works. One paper explicitly presents VARGPT (or VA-GPT) as a unified multimodal autoregressive model for visual understanding and generation (Zhuang et al., 21 Jan 2025). Other papers instantiate related patterns without using the same name in a formally identical sense. BIM-GPT defines a prompt-based virtual assistant for BIM information retrieval, where GPT acts as the language engine for query interpretation, summarization, and question answering (Zheng et al., 2023). VLN-GPT applies a GPT-2-style decoder to model trajectory sequence dependencies in vision-and-language navigation (Hanlin, 2024). The Vietnamese COVID-19 question answering study evaluates GPT-2 for Vietnamese community-based generative QA, but the work explicitly does not introduce a formal framework named “VA-GPT” (Vo et al., 2023).

Work	Domain	Relation to VA-GPT
VARGPT	Multimodal understanding and image generation	Direct usage: “VARGPT (or VA-GPT)”
BIM-GPT	BIM information retrieval	GPT-based virtual assistant framework
VLN-GPT	Vision-and-language navigation	GPT-style generative policy
Vietnamese GPT-2 QA	Community-based COVID-19 QA	Closely related implementation, not a formal VA-GPT proposal

This suggests that “VA-GPT” functions less as a single canonical architecture than as a family of GPT-centered systems in which autoregressive sequence modeling replaces or reduces task-specific engineering. That interpretation remains an inference from the cited works rather than a universal naming convention.

2. Core formulation in VARGPT

In its most explicit form, VA-GPT is a unified multimodal LLM that performs both visual understanding and visual generation inside one autoregressive system (Zhuang et al., 21 Jan 2025). The central methodological claim is that these two capabilities need not be split between an MLLM for comprehension and a separate diffusion model for synthesis. Instead, the model uses next-token prediction for understanding and next-scale prediction for image generation.

For visual understanding, the model follows a LLaVA-style autoregressive formulation. Image features $\mathbf{H}^{img}$ and query text $\mathbf{X}^{query}$ are fed to the LLM, which predicts the next textual token as

$\mathbf{Y}^{txt}_t \sim p_{\theta}(\mathbf{Y}^{txt}_t \mid \mathbf{X}^{img}, \mathbf{X}^{query}, \mathbf{Y}^{txt}_{<t}).$

For visual generation, the image is converted into multi-scale discrete feature maps

$(\mathbf{R}_1, \mathbf{R}_2, \cdots, \mathbf{R}_K),$

and the model generates them sequentially as

$\mathbf{R}_t \sim p_{\{\theta,\phi\}}(\mathbf{R}_t \mid \mathbf{X}^{query}, \mathbf{R}_{<t}).$

The conceptual distinction is precise. Next-token prediction governs multimodal understanding and textual response generation, whereas next-scale prediction governs image synthesis by predicting coarse visual structure first and finer scales later. This differs from flat discrete token generation for images and places VARGPT closer to a VAR-style visual autoregressive process than to diffusion-based generation.

A defining property is mixed-modal input/output. The model can accept text and/or image inputs, answer in text, and also emit image-generation tokens during the same dialogue. The token inventory includes <image> for input images and <image_gen_start>, <image_gen>, and <image_gen_end> for image-generation segments. When <image_gen_start> is emitted, subsequent <image_gen> slots are routed through the visual-generation pathway.

3. Architecture and unified training pipeline

VARGPT explicitly extends LLaVA-1.5-7B. Its understanding path uses a Vicuna-7B-v1.5 LLM, a CLIP ViT/14 visual encoder, and a projector that maps image features into the LLM embedding space (Zhuang et al., 21 Jan 2025). For generation, it adds a 2B-parameter visual decoder with 30 Transformer layers and two extra visual feature projectors. The paper states that these extra parameters avoid “knowledge conflict” between text decoding and image generation. The visual decoder uses block causal attention rather than standard LLM causal attention, and absolute positional encoding is added for visual tokens.

Training proceeds in three stages. In Stage 1: Pretraining, the model uses ImageNet-derived data to construct 1.28M single-round dialogue samples. All parameters are frozen except the two generation projectors, with the stated goal of aligning textual and visual feature spaces. In Stage 2: SFT for Visual Understanding, the model unfreezes the LLM and the projector for visual-encoder output, and trains on 1.18M mixed samples, including LLaVA-1.5-665K, 508K samples from LLaVA-OneVision, and 5K samples from ImageNet-Instruct-130K. In Stage 3: SFT for Visual Generation, the model unfreezes the visual decoder and the two generation projectors, freezes everything else, and trains on 1.4M instruction pairs from ImageNet-Instruct-1270K.

The generation process also incorporates classifier-free guidance (CFG). Using projected conditional features $\mathbf{H}^g$ and Gaussian noise $\mathbf{H}^e$ , the appendix writes the guidance mechanism as

$\begin{aligned} \mathbf{R}_t &\sim p_{\{\theta,\phi\}}(\mathbf{R}_t \mid \mathbf{H}^g, \mathbf{R}_{<t}); \ \mathbf{R}_e &\sim p_{\{\theta,\phi\}}(\mathbf{R}_t \mid \mathbf{H}^e, \mathbf{R}_{<t}); \ \mathbf{R}_t &= (1+\lambda)\mathbf{R}_t - \lambda \mathbf{R}_e. \end{aligned}$

The overall training objective is described operationally rather than as a single combined loss equation: maximize next-token likelihood for understanding and next-scale likelihood for generation.

4. Empirical performance and limitations

On vision-centric understanding benchmarks, VARGPT is reported to outperform LLaVA-1.5 while also supporting generation (Zhuang et al., 21 Jan 2025). In the benchmark table, VARGPT (7B+2B) reaches MMBench 67.6 versus 62.7, SEED 67.9 versus 65.4, MMMU 36.44 versus 35.24, MME 1488.8 versus 1480.1, and improves POPE across settings to 84.40 / 85.90 / 87.37 versus 83.60 / 85.77 / 86.97. On the VQA-style table it reaches GQA 62.3, TextVQA 54.1, VQAv2 78.4, SciQA-img 80.1, OKVQA 55.8, and VizWizQA 56.83; the authors explicitly highlight the 12.2-point improvement on SciQA-img over LLaVA-1.5.

Benchmark	VARGPT	LLaVA-1.5
MMBench	67.6	62.7
MMMU	36.44	35.24
MME	1488.8	1480.1
TextVQA	54.1	48.8
SciQA-img	80.1	67.9

For generation, evaluation uses 50,000 text instructions and reports FID and CLIP score. The full model achieves FID 12.6 and CLIP 27.4. Ablations indicate that the full three-stage pipeline is necessary: removing stage 3 worsens performance to FID 20.1 and CLIP 20.2. On the understanding side, freezing the projector in stage 2 reduces results to MMMU 35.19 / MME 1452.0, while freezing the LLM reduces them further to 33.51 / 1392.5.

The paper also states clear limitations. Generation is constrained by an ImageNet-based corpus that is smaller and lower-quality than the large datasets used by advanced diffusion systems such as SDv2.1 or FLUX, so generation quality still lags behind top diffusion models. The current model supports only 256×256 image generation, and some nuanced instruction details are not always faithfully reflected in outputs. These limitations are important because they delimit the extent of the claimed unification: the model is multimodally unified, but not yet competitive with the best specialized image generators.

A distinct but conceptually related use of GPT appears in BIM-GPT, which is defined as a prompt-based virtual assistant (VA) framework for BIM information retrieval (Zheng et al., 2023). Its architecture has three modules—a Web-based User Interface (UI), an NLP module, and a Data Management (DM) module—with a prompt manager orchestrating the workflow. The dynamic prompt template has five components: System, Relevant Database Information, Task Instruction, Few-shot Examples, and User. The framework decomposes natural-language understanding into intent classification, category identification, parameter identification, and value recognition. On the BINLQ dataset, reported accuracies include 83.5% and 99.5% for text-classification labels in zero-shot and few-shot settings, respectively; 98.6% and 99.5% for object category classification; and 86.7% and 95.1% for predicted/extracted values combined. The hospital-building prototype further validates retrieval, summarization, general BIM question answering, and 3D visualization.

In VLN-GPT, GPT is used not as a retrieval assistant but as a decoder-only navigation policy for Vision-and-Language Navigation (Hanlin, 2024). The model uses Sentence-BERT for instruction encoding, ViT for panoramic observations, and a GPT-2 base decoder to model serialized returns, states, and actions. The trajectory is written as

$\tau_T=(r_0,s_0,a_0,r_1,s_1,a_1,\ldots,r_T,s_T,a_T),$

and action prediction is conditioned on the full past trajectory rather than on a separate history encoder. Training is separated into offline pre-training with imitation learning and online fine-tuning with reinforcement learning. On Room-to-Room (R2R), the reported results include SAP 78 seen / 72 unseen, and main navigation results of 76 seen SR, 72 seen SPL, 65 unseen SR, and 61 unseen SPL, exceeding the listed PREVALENT and RecBERT baselines.