---
title: 'InternVL3: Open-Source Multimodal LLM'
url: https://www.emergentmind.com/topics/internvl3
type: topic
---

# InternVL3: Open-Source Multimodal LLM

InternVL3 is a family of open-source multimodal large language models (MLLMs) that unifies vision and language modeling within a native pre-training paradigm. Diverging from the incremental multimodal adaptation strategies seen in earlier models, InternVL3’s architecture, training objectives, and empirical optimization methods address the limitations of modular and post-hoc approaches. Noteworthy for its single-stage multimodal pretraining, scalable backbone, advanced preference optimization, and variable visual position encoding (V2PE), InternVL3 sets a state-of-the-art benchmark for open-source vision–language models, achieving competitive performance against leading proprietary systems across a range of multimodal, reasoning, and language tasks.

## 1. Model Architecture and Multimodal Pre-Training

InternVL3 implements a “ViT-MLP-LLM” backbone. The vision encoder (InternViT, 300M or 6B parameters) extracts patch embeddings from visual input, which are projected and merged with language tokens by a random-initialized two-layer MLP. The language model is typically a pre-trained LLM such as Qwen2.5 or InternLM3. Pixel-unshuffle compresses high-resolution images (e.g., 448×448 pixels) into 256 visual tokens. All modules—ViT, MLP, LLM—are trainable from the outset of pre-training, enabling cohesive multimodal learning.

A training sample $\mathbf{x}=(x_1, \dots, x_L)$ integrates both text and visual tokens, with position indices $p_i$ provided by the V2PE scheme (see Section 2). The unified training procedure exposes InternVL3 to $\sim$50B pure-text and $\sim$150B multimodal tokens in a single pre-training stage, with no intermediary text-only adaptation. The core autoregressive loss is
\[
\mathcal{L}_{\mathrm{full}}(\theta)
= -\sum_{i=2}^L w_i\;\log p_\theta(x_i\mid x_{<i}),
\]
with $w_i = 1/\ell^{0.5}$ for sample length $\ell$, which balances for sequence length bias. The optimization goal is
\[
\theta^* = \arg\min_\theta \mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\mathrm{multi}}}
\left[\mathcal{L}_\mathrm{LM}(\theta)\right].
\]
This architecture allows InternVL3 to natively align cross-modal representations, resulting in improved multimodal reasoning proficiency [2504.10479].

## 2. Variable Visual Position Encoding (V2PE) and Long-Context Processing

Integrating dense visual tokens alongside language tokens can rapidly exhaust a model’s context window due to fixed-step positional encoding. InternVL3 addresses this with V2PE, a recursive scheme that assigns for each token $x_i$:
\[
p_1=0,\quad
p_i = p_{i-1} +
\begin{cases}
1 & \text{if } x_i \text{ is textual} \\
\delta & \text{if } x_i \text{ is visual}
\end{cases}
\]
with $\delta<1$ drawn from $\Delta = \{1, \frac{1}{2}, \frac{1}{4}, ..., \frac{1}{256}\}$, chosen per image. This compresses the position space of visual tokens and preserves room for long text sequences within the same window. During inference, $\delta$ is adaptively tuned to fit window constraints. Visual position embeddings are computed as sinusoidal functions over $p_i$. V2PE is directly invoked in model self-attention and fusion layers, allowing robust processing of extended multimodal contexts [2504.10479].

## 3. Post-Training, Optimization, and Test-Time Strategies

InternVL3 applies advanced post-training strategies to augment base capabilities:

- **Supervised Fine-Tuning (SFT):** InternVL3 uses $\sim$21.7M curated, high-quality multimodal instruction samples, leveraging data augmentation (random JPEG, multimodal packing) to improve generalization.
- **Mixed Preference Optimization (MPO):** This combines three loss terms—preference (DPO), quality (BCO), and generation (next-token LM)—to optimize both preference alignment and task accuracy. For each model scale, gains range from +2.9 to +4.5 points in overall multimodal reasoning [2504.10479].
- **Test-Time Best-of-N and VisualPRM Critic:** At inference, InternVL3 can generate $N$ candidate responses, scoring them with a pretrained Visual Process Reward Model (VisualPRM-8B). The final prediction is chosen as the highest-scoring chain. This procedure significantly improves mathematical and reasoning accuracy, e.g., up to +6 points on MathVerse for 38B-scale models.

Training and inference are orchestrated using the InternEVO framework, which extends ZeRO-style sharding to allow fully decoupled data, tensor, sequence, and pipeline parallelism. This supports context lengths up to $32,000$ tokens and results in $50\%-200\%$ faster training compared to previous InternVL pipelines.

## 4. Empirical Performance Across Benchmarks

InternVL3 establishes competitive or best-in-class results across standard multimodal and language understanding benchmarks:

| Model                  | MMMU (%) | Overall Multi-Bench (%) | Notes             |
|------------------------|----------|------------------------|-------------------|
| InternVL3-78B (open)   | 72.2     | 54.6                   | Best open-source  |
| ChatGPT-4o             | 70.7     | 47.9                   | Closed-source     |
| Claude 3.5 Sonnet      | 75.0     | 53.9                   | Closed-source     |
| Gemini 2.5 Pro         | 69.9     | 58.5                   | Closed-source edge|

Additionally, InternVL3-8B outperforms Qwen2.5-7B on various language milestones, e.g., reaching 88.4% GSM8K (zero-shot), 89.0% HumanEval (pass@1), and an overall average of 78.9% [2504.10479].

In scientific visual QA, InternVL3-8B attains 0.740 ROUGE-1/ROUGE-L and 0.983 BERTScore on the SciVQA 2025 test split, outperforming next-best single models (Bespoke, Qwen2.5-VL) by margins of +0.031 ROUGE-1 F1 and +0.004 BERTScore. Error analysis identifies strengths in cross-chart grounding and numerical precision, with weaknesses in subfigure scaling misalignments and logical errors on multi-hop questions [2507.06183].

## 5. Data, Preprocessing, and Prompt Engineering

Input preprocessing involves resizing figures to a 1024px maximum dimension, applying center-cropping/padding to $1024\times1024$, and contrast normalization for textual/numerical clarity. Prompt construction concatenates captions, metadata, and questions with special tokens:
```
<FIG_CAP> [caption] </FIG_CAP> <TYPE> [figure_type] </TYPE> <Q> [question] </Q>
```
Tokenization uses a SentencePiece vocabulary shared across modalities. Alignment between visual patches and textual tokens is maintained with cross-modal position embeddings.

InternVL3’s high SciVQA performance is heavily dependent on prompt optimization and two-stage Chain-of-Thought (CoT) prompting: an “Initial Analysis” followed by structured answer extraction, leveraging XML tags such as `<reasoning>...</reasoning>` and `<answer>...</answer>`. This approach enforces concise, context-minimal output (numerical value or single word/phrase) and elicits improved intermediate reasoning and verifiability [2507.06183].

## 6. Subsequent Extensions: InternVL3.5 and Beyond

InternVL3.5 introduces significant architectural and algorithmic extensions:

- **Cascade RL:** Two-stage reinforcement learning (offline MPO, then online GSPO) adds +16% reasoning accuracy compared to InternVL3, with monotonic performance improvements at all model scales.
- **Visual Resolution Router (ViR):** Enables dynamic per-patch compression (4× or 16×), mediated by a lightweight router head trained via cross-entropy, achieving ~50% reduction in visual tokens with under 1% performance loss.
- **Decoupled Vision–Language Deployment (DvD):** Partitioning inference between vision and language GPU servers increases throughput up to 4.05×, reducing serialization bottlenecks.
- **Agentic/GUI Capabilities:** InternVL3.5 supports embodied and GUI-interactive tasks, with advances in performance on ScreenSpot-v2, OSWorld-G, WebArena-Lite-v2, VSI-Bench, and more, notably closing the gap with GPT-5 and related proprietary models [2508.18265].

| Model                       | Multimodal (35 bench) | Reasoning (9) | Text (8) | Agentic (6) | Notes                               |
|-----------------------------|----------------------|---------------|----------|-------------|-------------------------------------|
| InternVL3.5-241B-A28B (MoE) | 74.1                 | 66.9          | 85.3     | 66.2        | Open, ties or nears GPT-5           |
| InternVL3-78B               | 67.9                 | 54.6          | 79.4     | ≈60.6       |                                   |
| GPT-5                       | 74.0                 | 74.1          | 91.3     | 77.5        | Closed-source                       |

The introduction of Cascade RL, ViR, and DvD collectively enables both substantial reasoning improvement and real-world inference speedups, ushering in a new phase of openness and efficacy for MLLMs.

## 7. Limitations and Failure Modes

Despite strong empirical results, InternVL3 exhibits several notable failure modes:

- **Visual Misinterpretations:** Particularly when subfigures differ in scale or when visual elements (text, marks) overlap.
- **Numerical Misalignments:** Manifesting as rounding errors or off-by-one mistakes under degraded visual resolution.
- **Flawed Multi-Hop Reasoning:** Errors when intermediate steps require domain-specific or non-visual reasoning.

Current model variants are not fine-tuned for specific downstream domains such as SciVQA; performance relies primarily on inference-time prompt engineering. This suggests opportunities for further gains through domain-adaptive fine-tuning or higher-resolution visual processing.

A plausible implication is that, as model and infrastructure scaling continue, practitioner emphasis may shift toward optimizing context adaptation (e.g., V2PE strategies), advanced preference-based learning, and efficient multimodal deployment frameworks.

Source: https://www.emergentmind.com/topics/internvl3