InternVL 8B: Multimodal Vision–Language Model

• InternVL 8B is a scalable multimodal vision–language model that fuses a 6B-parameter vision transformer with an 8B-parameter language alignment module using progressive cross-modal strategies.
• It employs a two-stage training workflow on massive web-scale image–text data, utilizing both contrastive and generative losses to achieve robust zero-shot performance and effective transferability.
• The architecture supports versatile applications ranging from image recognition to multimodal dialogue, and demonstrates competitive performance across 32 visual-linguistic benchmarks.

InternVL 8B is a large-scale multimodal vision–language foundation model designed to achieve strong performance on generic visual-linguistic benchmarks by scaling up the vision backbone and aligning it progressively with LLMs. The model integrates a 6B-parameter vision transformer (InternViT) and an 8B-parameter language alignment module (QLLaMA), jointly trained using web-scale image–text data to support tasks ranging from image-level recognition to multi-modal dialogue systems. This architecture emphasizes efficient cross-modal alignment, robust zero-shot capabilities, and transferability to downstream applications.

1. Model Architecture and Design

InternVL 8B employs a two-component architecture:

• Vision Encoder (InternViT, 6B parameters): The visual backbone follows a transformer-based design (similar to ViT), partitioning input images $I \in \mathbb{R}^{H \times W \times 3}$ into fixed-size patches, which are tokenized to produce a feature map $F \in \mathbb{R}^{N \times D}$. In dense prediction tasks, representations for [CLS] and individual patches ([PATCH$_i$]) can be managed via global average pooling or spatial preservation:

$$F = [f_{cls}; f_{patch_1}, f_{patch_2}, \ldots, f_{patch_N}]$$

• Language Middleware (QLLaMA, ~8B parameters): QLLaMA bridges vision and language modalities using learnable query tokens and cross-attention layers. Initialized with multilingual LLaMA weights where appropriate, it transforms image features into a representation aligned with LLMs, facilitating bidirectional information flow.
• Joint Vision–Language Interface: The overall system can function as an image encoder for perception tasks or, when using QLLaMA as an interface, connect to an LLM decoder for generative and dialogue tasks.

Training objectives include both contrastive losses (as in CLIP) and generative losses (BLIP-2-inspired), for example:

$$L = -\log\left[ \frac{\exp(\text{sim}(I_f, T_f))}{\sum_j \exp(\text{sim}(I_f, T_j))} \right]$$

where $I_f$ and $T_f$ are image and textual features from QLLaMA.

2. Scaling and Progressive Alignment with LLMs

• Vision Backbone Scaling: InternViT is scaled up through systematic variation of model depth, parameter width, attention heads, and MLP expansion ratio, resulting in a 6B-parameter model that matches the scale of contemporary LLMs.
• Progressive Alignment Strategy: Initially, contrastive learning aligns vision and text representations over billions of noisy image–text pairs. In a subsequent stage, the system freezes major networks, introducing trainable queries and cross-attention for tight coupling, employing composite losses (contrastive, image–text matching, and image-grounded generation).

This alignment ensures consistency between image embeddings and text embeddings, supporting high cross-modal transferability.

3. Training Data and Pre-Training Workflow

InternVL 8B leverages diverse, web-scale datasets, including LAION-en, LAION-multi, LAION-COCO, COYO, and Wukong:

• Stage 1 Filtering: Minimal filtering yields ~4.98 billion image–text pairs for large-scale contrastive pretraining.
• Stage 2 Filtering: Stringent caption filtering yields ~1.03 billion high-quality pairs for supervised generative alignment.

A two-stage workflow with massive data is critical for generalization and robust multimodal representation. The progressive approach enables efficient transfer and generalization across multiple downstream visual and vision-language tasks.

4. Benchmark Performance and Evaluation

InternVL 8B is evaluated on 32 generic visual-linguistic benchmarks:

Task Type	Representative Datasets	Performance Characteristics
Image Classification	ImageNet variants	Strong zero-shot robustness
Video Classification	Kinetics 400/600/700	State-of-the-art performance
Retrieval	Image/video-text retrieval, COCO, Wukong	Competitive with and often superior to larger models
Captioning & QA	Visual QA, ADE20K, semantic segmentation	High out-of-box performance via aligned embeddings

InternVL 8B consistently attains state-of-the-art or competitive results in zero-shot, retrieval, and visual QA tasks, demonstrating the efficacy of scaled vision transformers and progressive modal alignment.

5. Applications and System Integration

InternVL 8B supports a broad range of use cases:

• Standalone Vision Backbone: Suitable for perception tasks (classification, segmentation) as a pure image encoder.
• Vision–Language Pipeline: When connected to an LLM decoder through QLLaMA, enables multi-modal dialogue, captioning, and VQA.
• Cross-Modal Reasoning: Architecture facilitates tasks requiring joint reasoning over vision and language, including knowledge-based reasoning and complex scene understanding.
• Out-of-box Transferability: Robust across heterogeneous benchmarks and domains due to diverse training data and normalized cross-modal features.

This versatility is attributed to explicit design choices supporting vision–language alignment and scalable multimodal modeling.

6. Comparative Analysis: InternVL 8B versus ViT-22B

Model	Vision Params	Language Alignment	Multimodal Benchmarks	Key Differentiators
ViT-22B	~22B	Not directly LLM-aligned	Primarily vision tasks	Larger, vision-only
InternVL 8B	6B	QLLaMA (8B); LLM-friendly	Vision and vision-language	Efficient cross-modal alignment; competitive or superior performance

ViT-22B exemplifies large-scale pure vision models but lacks direct LLM compatibility for generative or dialogue tasks. InternVL 8B is specifically engineered for compatibility with LLMs, supporting multimodal reasoning and generative tasks with a more efficient parameter footprint.

7. Future Directions and Broader Impact

• Scaling Strategies: Further increases in backbone scale, refinement of progressive alignment techniques, and higher-quality web data are identified as promising directions for enhancing general-purpose multimodal models.
• Unified Multimodal AGI Frameworks: InternVL 8B’s design principles set foundations for future unified AGI systems that seamlessly couple vision and language modalities.
• Continued Development: Open availability of models and implementation resources expedites scientific progress in multimodal AI, promoting reproducible research and practical adoption across domains.

The InternVL series exemplifies the benefits of vision–language co-design at scale, efficient cross-modal training strategies, and openness as drivers for advancement in multimodal artificial intelligence research (Chen et al., 2023).