---
title: 'Qwen-VL-Chat: Advanced Multimodal LVLM'
url: https://www.emergentmind.com/topics/qwen-vl-chat
type: topic
---

# Qwen-VL-Chat: Advanced Multimodal LVLM

Qwen-VL-Chat is a multimodal large vision-language model (LVLM) developed within the Qwen-VL series, characterized by integrated visual and linguistic reasoning, grounded localization, and advanced text-reading abilities. Built atop the Qwen-LM 7B parameter large language model, Qwen-VL-Chat establishes new state-of-the-art results for generalist LVLMs of similar scale across a spectrum of benchmarks, including image captioning, visual question answering (VQA), visual grounding, and real-world multimodal dialogue tasks [2308.12966].

## 1. Model Architecture

### 1.1 Visual Receptor

The vision encoder is based on ViT-bigG (OpenCLIP) with patch size 14, pre-trained on CLIP. During Stage 1 pre-training, images are input at $224 \times 224$ resolution, yielding a sequence length of 256; in subsequent stages, the input is $448 \times 448$ with a sequence length of 1024. The encoder outputs a sequence $\{f_i \in \mathbb{R}^{d_v}\}_{i=1}^L$ of per-patch embeddings where $L=(H/14)\cdot (W/14)$.

Following the encoder, a single cross-attention "position-aware vision–language adapter" is applied, featuring 256 learnable query vectors $\{q_j\}_{j=1}^{256}$. Keys/values derive from ViT patch embeddings $f_i$ with added 2D positional encoding $p_i$. The attention mechanism receives a 2D positional bias $\phi_{\mathrm{pos}}$ applied to each query–key dot product:
$$
\operatorname{Attn}(Q, K, V) = \operatorname{softmax}\left( (QW_q)(KW_k)^\top/\sqrt{d} + \phi_{\mathrm{pos}} \right) (VW_v)
$$
This compresses the variable-length ViT output to a fixed-length sequence of 256 "visual tokens," later mapped to the language model embedding space.

### 1.2 Input–Output Interface

Image inputs are marked by <img>…</img> tokens; visual tokens are interleaved into the LLM’s input space. Bounding boxes for outputs are normalized to $[0,1000)$, formatted as "(x₁,y₁),(x₂,y₂)", and wrapped in <box>…</box> tokens; referenced regions use <ref>…</ref>. All tokens (textual and visual) are concatenated and autoregressively processed by the Transformer LLM (Qwen-7B). At inference, the model produces serializations for answers, captions, or box coordinates via greedy decoding [2308.12966].

## 2. Three-Stage Training Pipeline

### 2.1 Stage 1: Vision–Language Pre-training

This stage utilizes 1.4 billion cleaned image-text pairs (77.3% English, 22.7% Chinese). The language model is frozen, while the ViT and adapter are trained. The loss is a standard autoregressive cross-entropy over text tokens:
$$
L_1 = - \sum_{(I,T)} \sum_{t=1}^{|T|} \log P(T_t\,|\,T_{<t},\,I)
$$
Input images use a $224 \times 224$ resolution, and the batch size is 30,720, over 50,000 steps.

### 2.2 Stage 2: Multi-Task Pre-training

This stage covers seven tasks and approximately 69 million samples, including image captioning, VQA, grounding, referring grounding, grounded captioning, OCR, and pure-text autoregression. High-resolution images ($448\times448$) and multi-task sequences up to length 2048 are used. All model parameters are updated. Multi-task cross-entropy is applied:
$$
L_2 = \sum_{\textrm{task}} \alpha_\textrm{task} \cdot L_{CE}(\textrm{task}),\;\;\; L_{CE}(\textrm{task}) = -\sum \log P(\textrm{label}_t\,|\,\textrm{history})
$$
$\alpha_\textrm{task}$ is typically uniform. Training proceeds for 19,000 steps with batch size 4,096.

### 2.3 Stage 3: Supervised Fine-Tuning (SFT) — Qwen-VL-Chat

Instruction-tuning uses approximately 350,000 multimodal instruction–response pairs in ChatML format. Dialogues span LLM-generated single-image, manually annotated multi-image/multi-round, grounding, text-reading, and pure-text conversations. The visual encoder is frozen; only the adapter and LLM are trained with cross-entropy on assistant reply tokens:
$$
L_3 = - \sum_{\textrm{example}} \sum_{t \in \text{answer}} \log P(a_t\,|\,\textrm{history})
$$
Training runs for 8,000 steps with batch size 128.

## 3. Multilingual and Multimodal Data Pipeline

The pre-training corpus is drawn from multiple sources (LAION-en, LAION-COCO, DataComp, COYO, CC12M, CC3M, SBU, COCO Captions, LAION-zh, in-house Chinese) resulting in 1.4 billion filtered image-text pairs (retention rate 28% from an initial 5 billion). The data-cleaning procedure includes:

1. Filtering on aspect ratio and image size.
2. Removal based on CLIP score, non-Latin characters, emojis, HTML, extreme sequence lengths.
3. Deduplication of academic captions, retaining the longest.

The pipeline incorporates both synthetic and real data, supporting multilingual and multimodal learning at scale.

## 4. Grounding and Text-Reading Capabilities

### 4.1 Grounding

For spatial localization and phrase grounding, Qwen-VL-Chat is trained on noun/phrase grounding datasets (GRiT, Visual Genome, RefCOCO/+/g) with $\sim$8.7M samples each for referring grounding and grounded captioning. Phrases to be localized are wrapped in <ref>…</ref>, followed by their bounding boxes in <box>…</box>. Grounding is learned autoregressively from the token serialization rather than with a separate regression loss.

### 4.2 Text Reading (OCR)

The model leverages 24.8 million synthetic OCR samples (SynthDoG on COCO backgrounds, English/Chinese fonts), as well as rendered web-crawled PDFs and HTML, pairing rendered images with text and quadrilateral coordinates. The task is denoted by a textual prefix (“OCR with grounding:”) and quads encoded in <quad>…</quad>. The same cross-entropy framework is used for both content recognition and positional grounding.

## 5. Instruction Tuning and Conversational Protocol

Qwen-VL-Chat is distinguished by full instruction tuning for multi-turn multimodal dialogue. Training data is a mixture of LLM-generated and hand-annotated conversations, including multi-image, multi-round, localization, and text-reading examples, together with pure-text dialogues for LLM skill retention. The ChatML format formalizes dialog turns with <im_start>, <im_end>, and manages multiple images as “Picture i: <img>…</img>.”

During SFT, the objective is always cross-entropy loss on assistant-side outputs, with the vision encoder weights held fixed for stability. This facilitates robust multimodal conversational capabilities encompassing image, text, spatial, and linguistic reasoning.

## 6. Benchmark Performance

Qwen-VL-Chat, together with its base model Qwen-VL, achieves leading results across multiple visual-linguistic evaluation protocols:

### Image Captioning and General VQA (Zero-Shot)

- Nocaps (val): Qwen-VL-Chat 120.2
- Flickr30K (karpathy): 81.0
- VQAv2: 78.2
- OKVQA: 56.6
- GQA: 57.5
- ScienceQA-Img: 68.2
- VizWiz: 38.9

### Text-Oriented VQA

- TextVQA: 61.5
- DocVQA (ANLS): 62.6
- ChartQA: 66.3
- AI2D: 57.7
- OCR-VQA: 70.5

### Referring Expression Comprehension

- RefCOCO (val/testA/testB): 88.55/92.27/84.51
- Qwen-VL-Chat achieves top-tier results on RefCOCO+/g and GRiT as well.

### Few-Shot In-Context Learning

On OKVQA, VizWiz, TextVQA, and Flickr30K, Qwen-VL (7B) surpasses Flamingo-9B and demonstrably rivals Flamingo-80B performance in 1–8 shot settings.

### Real-World Multimodal Dialogue

- TouchStone (GPT-4 score): 645.2 (English), 401.2 (Chinese); best prior ≈605 (English)
- SEED-Bench (MC VQA): 58.2 overall (Image 65.4, Video 37.8); vs. InstructBLIP 53.4
- MME (Yes/No per-question): Perception 1487.6, Cognition 360.7; best prior ≈1212/292

A plausible implication is that the three-stage curriculum (weakly supervised pre-training, multi-task alignment, fine-grained instruction tuning) tightly integrates visual and linguistic processing, enabling flexible, high-performance multimodal reasoning [2308.12966].

Source: https://www.emergentmind.com/topics/qwen-vl-chat