Xmodel-VLM: Compact Multimodal Vision-Language Model

• Xmodel-VLM is a multimodal vision-language model that integrates a frozen CLIP vision encoder, a two-layer MLP projector, and a 1.1B-parameter language model.
• It employs a two-stage LLaVA-style instruction-tuning process to efficiently align visual and textual modalities without additional contrastive losses.
• Benchmark results reveal that despite its compact size, Xmodel-VLM delivers competitive performance and fast inference on consumer GPUs for various deployment scenarios.

Xmodel-VLM is a multimodal vision-LLM (VLM) engineered for efficient deployment on consumer GPU servers, targeting the prohibitive computational and service costs that hinder the adoption of large-scale multimodal systems. With approximately one billion parameters and an architecture distilled from the LLaVA paradigm for modal alignment, Xmodel-VLM achieves a balance between model compactness, inference speed, and performance on classic multimodal benchmarks. The approach comprises a bespoke LLM, a CLIP-based vision encoder, and a two-layer projector, trained via a two-step instruction-tuning framework. Xmodel-VLM, code, and checkpoints are publicly released by the Xiaoduo AI Lab (Xu et al., 2024).

1. Model Architecture

Xmodel-VLM employs a streamlined “vision encoder → projector → language decoder” architecture containing roughly 1.1 billion total parameters:

• Vision Encoder: Utilizes a frozen CLIP ViT-L/14 backbone with 336×336 input resolution. The encoder produces 576 patch embeddings $Z_v \in \mathbb{R}^{576 \times 1024}$.
• Projector ("XDP"): Implements a two-layer MLP with Mish activations, converting $d_v=1024$ to $d_\ell=2048$ (the LLM’s embedding dimension) and down-sampling tokens from 576 to 144.

$$H_v = P(Z_v) = Mish(Z_v W_1 + b_1) W_2 + b_2 \in \mathbb{R}^{144 \times 2048}$$

• LLM (Xmodel-LM 1.1B): Comprises 24 transformer layers, each with hidden size 2048 and 32 attention heads, context length 4096 tokens, and total parameters ≈1.1B. The architecture matches LLaMA for compatibility.

Schematic:

$d_v=1024$6 Autoregressive Generation Objective:

For image $X_v$, question tokens $X_q$, and previous answer tokens $X_{a,<i}$, the objective is:

$$p( X_a | X_v, X_q ) = \prod_{i=1}^L p_\theta( x_i | H_v, X_q, X_{a,<i} )$$

with standard cross-entropy loss:

$$L_{CE} = -\sum_{i=1}^L \log p_\theta( x_i | \cdot )$$

2. LLaVA-Style Modal Alignment

Xmodel-VLM adopts the two-stage LLaVA instruction-tuning paradigm to align visual and textual modalities without resorting to additional contrastive losses or parameter-efficient fine-tuning:

• Stage 1: Feature Alignment (Projector-Only)
  • Data: 595K image–caption pairs from Conceptual Captions 3M.
  • Trainable parameters: Only the projector; CLIP encoder and LLM are frozen.
  • Loss: Standard autoregressive cross-entropy on the multimodal instruction prompt.
  • Goal: Ensure $P(Z_v)$ resides in the same representation manifold as LLM word embeddings.
• Stage 2: End-to-End Fine-Tuning
  • Data: 150K GPT-4–augmented instruction-following triples (LLaVA-Instruct-150K).
  • Trainable parameters: Projector and full Xmodel-LM; visual encoder remains frozen.
  • Loss: Cross-entropy for all answer tokens.

Mathematical Objective:

$d_v=1024$0

Only AdamW weight decay (0.0) is used as regularization.

3. Training Setup

• Datasets
  • Pretrain: CC-595K (595K image–caption pairs).
  • Instruction-tune: LLaVA-Instruct-150K (150K GPT-4–augmented triples).
• Optimization Protocol
  • Stage 1 (Projector): AdamW, learning rate $d_v=1024$1, batch size 64, one epoch.
  • Stage 2 (Projector + LLM): AdamW, learning rate $d_v=1024$2, batch size 32, one epoch.
  • AdamW settings: $d_v=1024$3, $d_v=1024$4, $d_v=1024$5.
• Compute Requirements
  • Trained on commodity GPUs such as the NVIDIA RTX 3090.
  • Approximately 750K samples processed.
  • Estimated wall-clock time: less than 24 hours on a single high-end GPU.

4. Benchmarking and Empirical Performance

Xmodel-VLM’s performance is assessed on nine standard vision-language benchmarks and compared to open-source baselines:

Model	LLM (size)	VizWiz (%)	SciQA-IMG	TextVQA (%)	POPE (%)	GQA (%)	MMB (%)	MMB-CN (%)	MM-Vet (%)	MME (Σ tokens)
InstructBLIP (7B)	Vicuna-7B	34.5	60.5	50.1	—	49.2	36.0	23.7	26.2	—
LLaVA-1.5 (13B)	Vicuna-13B	53.6	71.6	61.3	85.9	63.3	67.7	63.6	35.4	1531.3
MobileVLM (1.7B)	MobileLLaMA	26.3	54.7	41.5	84.5	56.1	53.2	16.7	21.7	1196.2
Xmodel-VLM (1.1B)	Xmodel-LM 1.1B	31.9	54.4	38.9	86.1	57.4	48.5	44.2	23.4	1251.5

Despite a smaller parameter count, Xmodel-VLM matches or surpasses larger baselines on tasks such as POPE and GQA.

Inference Latency (SQA-IMG, RTX 3090)

Model	Params	Tokens/sec	Total time (s)
LLaVA-7B	7 B	19.06	1090.3
MobileVLM-1.7B	1.7 B	919.25	579.9
Xmodel-VLM	1.1 B	415.69	1360.3

Xmodel-VLM runs approximately 20× faster than a 7B model but is slower than heavily optimized mobile models.

Ablation Studies

• The two-layer projector (XDP) approach yields accuracy comparable to a single-layer MLP while efficiently reducing visual token count.
• Reducing tokens from 576 to 144 results in a manageable 2–3% drop in performance.
• Increasing LLM size from 1.1B to 4B or 8B produces expected improvements in VLM task accuracy.

5. Deployment and Open-Source Availability

The complete codebase, pre-trained checkpoints, and configuration files for Xmodel-VLM are available at https://github.com/XiaoduoAILab/XmodelVLM.

Quick start (Linux, CUDA, PyTorch ≥1.13):

$d_v=1024$7 Additional examples and API documentation are provided in the repository’s examples/ directory.

6. Technical Impact and Practical Applications

Key Strengths:

• Xmodel-VLM demonstrates that a compact (1.1B parameters) VLM can rival models 7–13× larger on several vision–language tasks.
• The two-stage LLaVA recipe achieves efficient modal alignment with no costly contrastive or LoRA-based alignment.
• Fully open-source implementation tailored for consumer GPU hardware.

Noted Limitations:

• Inference latency, while improved, is not as low as that of specialized mobile VLMs (e.g., MobileVLM).
• Performance on fine-grained tasks such as TextVQA is modestly lower compared to 13B-parameter references.

Deployment Scenarios:

• On-premise image–language assistants (e.g., kiosks, chatbots) deployed on commodity RTX 3060/3090.
• Integration into customer-service robotics.
• Mobile-edge scenarios enabled via further quantization or pruning.

In summary, Xmodel-VLM embodies a design trade-off between model scale and operational cost within VLM research, showing that efficient, scalable vision–language interaction is possible without large-scale infrastructure (Xu et al., 2024).