---
title: 'Xmodel-VLM: Compact Multimodal Vision-Language Model'
url: https://www.emergentmind.com/topics/xmodel-vlm
type: topic
---

# Xmodel-VLM: Compact Multimodal Vision-Language Model

Xmodel-VLM is a multimodal vision-language model (VLM) engineered for efficient deployment on consumer GPU servers, targeting the prohibitive computational and service costs that hinder the adoption of large-scale multimodal systems. With approximately one billion parameters and an architecture distilled from the LLaVA paradigm for modal alignment, Xmodel-VLM achieves a balance between model compactness, inference speed, and performance on classic multimodal benchmarks. The approach comprises a bespoke language model, a CLIP-based vision encoder, and a two-layer projector, trained via a two-step instruction-tuning framework. Xmodel-VLM, code, and checkpoints are publicly released by the Xiaoduo AI Lab [2405.09215].

## 1. Model Architecture

Xmodel-VLM employs a streamlined “vision encoder → projector → language decoder” architecture containing roughly 1.1 billion total parameters:

- **Vision Encoder**: Utilizes a frozen CLIP ViT-L/14 backbone with 336×336 input resolution. The encoder produces 576 patch embeddings $Z_v \in \mathbb{R}^{576 \times 1024}$.
- **Projector ("XDP")**: Implements a two-layer MLP with Mish activations, converting $d_v=1024$ to $d_\ell=2048$ (the LLM’s embedding dimension) and down-sampling tokens from 576 to 144.
  
  $$
  H_v = P(Z_v) = Mish(Z_v W_1 + b_1) W_2 + b_2 \in \mathbb{R}^{144 \times 2048}
  $$

- **Language Model (Xmodel-LM 1.1B)**: Comprises 24 transformer layers, each with hidden size 2048 and 32 attention heads, context length 4096 tokens, and total parameters ≈1.1B. The architecture matches LLaMA for compatibility.

**Schematic:**
```
[Image X_v] → ViT-L/14 → 576×1024 Z_v
      ↓
   [XDP Projector: 2-layer MLP + Mish + Downsampling]
      ↓
  144×2048 H_v
      ↓
[Text Prompt: “Human: ... Assistant: ...”]
      ↓
   Xmodel-LM → Autoregressive Output
```
**Autoregressive Generation Objective**:
For image $X_v$, question tokens $X_q$, and previous answer tokens $X_{a,<i}$, the objective is:
$$
p( X_a | X_v, X_q ) = \prod_{i=1}^L p_\theta( x_i | H_v, X_q, X_{a,<i} )
$$
with standard cross-entropy loss:
$$
L_{CE} = -\sum_{i=1}^L \log p_\theta( x_i | \cdot )
$$

## 2. LLaVA-Style Modal Alignment

Xmodel-VLM adopts the two-stage LLaVA instruction-tuning paradigm to align visual and textual modalities without resorting to additional contrastive losses or parameter-efficient fine-tuning:

- **Stage 1: Feature Alignment (Projector-Only)**
  - **Data**: 595K image–caption pairs from Conceptual Captions 3M.
  - **Trainable parameters**: Only the projector; CLIP encoder and LLM are frozen.
  - **Loss**: Standard autoregressive cross-entropy on the multimodal instruction prompt.
  - **Goal**: Ensure $P(Z_v)$ resides in the same representation manifold as LLM word embeddings.

- **Stage 2: End-to-End Fine-Tuning**
  - **Data**: 150K GPT-4–augmented instruction-following triples (LLaVA-Instruct-150K).
  - **Trainable parameters**: Projector and full Xmodel-LM; visual encoder remains frozen.
  - **Loss**: Cross-entropy for all answer tokens.

**Mathematical Objective**:
$$
L(\theta_P,\theta_\ell) = -\sum_{(X_v,X_q,X_a)} \sum_{i=1}^L \log p_{\theta_P, \theta_\ell}(x_i \mid P(g(X_v)), X_q, X_{a,<i})
$$
Only AdamW weight decay (0.0) is used as regularization.

## 3. Training Setup

- **Datasets**
  - **Pretrain**: CC-595K (595K image–caption pairs).
  - **Instruction-tune**: LLaVA-Instruct-150K (150K GPT-4–augmented triples).
- **Optimization Protocol**
  - **Stage 1 (Projector)**: AdamW, learning rate $1 \times 10^{-3}$, batch size 64, one epoch.
  - **Stage 2 (Projector + LLM)**: AdamW, learning rate $4 \times 10^{-5}$, batch size 32, one epoch.
  - **AdamW settings**: $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=1e{-8}$.
- **Compute Requirements**
  - Trained on commodity GPUs such as the NVIDIA RTX 3090.
  - Approximately 750K samples processed.
  - Estimated wall-clock time: less than 24 hours on a single high-end GPU.

## 4. Benchmarking and Empirical Performance

Xmodel-VLM’s performance is assessed on nine standard vision-language benchmarks and compared to open-source baselines:

| Model                    | LLM (size)       | VizWiz (%) | SciQA-IMG | TextVQA (%) | POPE (%) | GQA (%) | MMB (%) | MMB-CN (%) | MM-Vet (%) | MME (Σ tokens) |
|--------------------------|------------------|------------|-----------|-------------|----------|---------|---------|------------|------------|----------------|
| InstructBLIP (7B)        | Vicuna-7B        | 34.5       | 60.5      | 50.1        | —        | 49.2    | 36.0    | 23.7       | 26.2       | —              |
| LLaVA-1.5 (13B)          | Vicuna-13B       | 53.6       | 71.6      | 61.3        | 85.9     | 63.3    | 67.7    | 63.6       | 35.4       | 1531.3         |
| MobileVLM (1.7B)         | MobileLLaMA      | 26.3       | 54.7      | 41.5        | 84.5     | 56.1    | 53.2    | 16.7       | 21.7       | 1196.2         |
| **Xmodel-VLM (1.1B)**    | Xmodel-LM 1.1B   | 31.9       | 54.4      | 38.9        | 86.1     | 57.4    | 48.5    | 44.2       | 23.4       | 1251.5         |

Despite a smaller parameter count, Xmodel-VLM matches or surpasses larger baselines on tasks such as POPE and GQA.

### Inference Latency (SQA-IMG, RTX 3090)

| Model           | Params | Tokens/sec | Total time (s) |
|-----------------|--------|------------|---------------|
| LLaVA-7B        | 7 B    | 19.06      | 1090.3        |
| MobileVLM-1.7B  | 1.7 B  | 919.25     |  579.9        |
| **Xmodel-VLM**  | 1.1 B  | 415.69     | 1360.3        |

Xmodel-VLM runs approximately 20× faster than a 7B model but is slower than heavily optimized mobile models.

### Ablation Studies

- The two-layer projector (XDP) approach yields accuracy comparable to a single-layer MLP while efficiently reducing visual token count.
- Reducing tokens from 576 to 144 results in a manageable 2–3% drop in performance.
- Increasing LLM size from 1.1B to 4B or 8B produces expected improvements in VLM task accuracy.

## 5. Deployment and Open-Source Availability

The complete codebase, pre-trained checkpoints, and configuration files for Xmodel-VLM are available at https://github.com/XiaoduoAILab/XmodelVLM.

**Quick start (Linux, CUDA, PyTorch ≥1.13):**
```python
import torch
from xmodelvlm import CLIPVisionEncoder, XmodelLM, Projector, XmodelVLM

# Load components
vision = CLIPVisionEncoder.from_pretrained("clip-vit-l-14-336")
lm     = XmodelLM.from_pretrained("xmodel-lm-1.1b")
proj   = Projector.from_pretrained("xdp-144")
model  = XmodelVLM(vision, proj, lm)
model.eval().cuda()

# Single-image QA
img = load_image("test.jpg")
q   = "What color is the bird?"
a   = model.generate(img, q)
print(a)
```
Additional examples and API documentation are provided in the repository’s `examples/` directory.

## 6. Technical Impact and Practical Applications

**Key Strengths:**
- Xmodel-VLM demonstrates that a compact (1.1B parameters) VLM can rival models 7–13× larger on several vision–language tasks.
- The two-stage LLaVA recipe achieves efficient modal alignment with no costly contrastive or LoRA-based alignment.
- Fully open-source implementation tailored for consumer GPU hardware.

**Noted Limitations:**
- Inference latency, while improved, is not as low as that of specialized mobile VLMs (e.g., MobileVLM).
- Performance on fine-grained tasks such as TextVQA is modestly lower compared to 13B-parameter references.

**Deployment Scenarios:**
- On-premise image–language assistants (e.g., kiosks, chatbots) deployed on commodity RTX 3060/3090.
- Integration into customer-service robotics.
- Mobile-edge scenarios enabled via further quantization or pruning.

In summary, Xmodel-VLM embodies a design trade-off between model scale and operational cost within VLM research, showing that efficient, scalable vision–language interaction is possible without large-scale infrastructure [2405.09215].

Source: https://www.emergentmind.com/topics/xmodel-vlm