InternVL 2.5: Open-Source MLLM Evolution

• InternVL 2.5 is an open‐source multimodal LLM series that refines the ViT-MLP-LLM architecture with improved backbones and dynamic test-time scaling.
• It employs dynamic high-resolution tiling with pixel unshuffle token reduction and a three-stage training pipeline for efficient multimodal learning.
• The model achieves a state-of-the-art MMMU validation score exceeding 70%, positioning it competitively against leading proprietary systems.

InternVL 2.5 is an open-source multimodal LLM (MLLM) series in the InternVL family that extends InternVL 2.0 while preserving the same high-level ViT-MLP-LLM architecture and emphasizing coordinated scaling of model, data, and test-time compute rather than architectural novelty alone (Chen et al., 2024). The series spans models from 0.9B to 78.4B parameters, targets image, multi-image, video, OCR, document, multilingual, reasoning, grounding, and pure-language tasks, and is presented as narrowing the gap between open-source systems and proprietary multimodal models such as GPT-4o and Claude-3.5-Sonnet. Its central headline is that InternVL2.5-78B is reported as the first open-source MLLM to exceed 70% on MMMU validation, scoring 70.1 under the paper’s protocol that takes the maximum accuracy across direct-answer and Chain-of-Thought settings (Chen et al., 2024).

1. Lineage and design position

InternVL 2.5 is explicitly framed as an upgraded successor to InternVL 2.0 that maintains its core model architecture while introducing stronger backbones, better data curation, a more formalized training pipeline, and explicit test-time scaling (Chen et al., 2024). It is therefore best understood as a continuation of the modular InternVL line rather than a break from it.

Within the broader family, several adjacent works clarify what is stable across InternVL generations. Mini-InternVL shows that compact descendants preserve the same basic ViT–MLP–LLM recipe, inherit the dynamic-resolution strategy associated with InternVL 1.5, and treat large InternVL vision encoders as sources of transferable visual knowledge (Gao et al., 2024). Mono-InternVL explores a distinct, monolithic direction in which visual encoding and language decoding are integrated into a single LLM; that work is explicitly not the InternVL 2.5 line, but it reveals that the InternVL ecosystem was simultaneously investigating latency, OCR, high-resolution handling, and language-retention tradeoffs beyond the standard modular recipe (Luo et al., 2024). InternVL3 is then presented as the next-generation successor to InternVL 2.5, preserving the same broad ViT-MLP-LLM backbone while replacing the earlier post-hoc adaptation pipeline with native multimodal pre-training and adding Variable Visual Position Encoding (V2PE), Mixed Preference Optimization (MPO), and new infrastructure (Zhu et al., 14 Apr 2025).

This lineage makes clear that InternVL 2.5 occupies a transitional but central position. It consolidates the family’s durable engineering principles—strong InternViT backbones, simple MLP bridging, pixel-unshuffle token reduction, dynamic high-resolution tiling, staged training, and broad multimodal scope—before the training-paradigm change introduced by InternVL3 (Chen et al., 2024).

2. Architecture and model lineup

InternVL 2.5 retains the standard “ViT-MLP-LLM” design used in earlier InternVL releases: a vision encoder, a LLM, and a randomly initialized 2-layer MLP projector that maps visual features into the LLM token space (Chen et al., 2024). The architecture does not introduce a new fusion mechanism. Its reported gains are attributed instead to upgraded backbones, stronger data, and revised training and inference procedures.

A key implementation detail is pixel unshuffle. After the ViT processes each image tile, pixel unshuffle reduces the visual token count by 4×: a 448×448 tile initially yields 1024 visual tokens and is reduced to 256 visual tokens (Chen et al., 2024). This token compression is central to InternVL-style high-resolution processing and persists into later family members such as InternVL3 (Zhu et al., 14 Apr 2025).

InternVL 2.5 employs dynamic high-resolution processing with a fixed tile size of 448 × 448. The strategy extends beyond single-image inputs to multi-image and video settings. With the common threshold of 48 tiles, images can reach 12,288 visual tokens. For video, no tiling is used; each frame is a single 448×448 image, so 32 frames correspond to 8,192 visual tokens and 64 frames correspond to 16,384 visual tokens (Chen et al., 2024).

The family uses explicit multimodal markup. Visual content is enclosed by <img> and </img>; multi-image samples use tags such as Image-1; video frames use tags such as Frame-1; and training data is formatted in a ChatML-style conversation format (Chen et al., 2024).

The released lineup is as follows.

Model	Total Params	Backbone configuration
InternVL2.5-1B	0.9B	InternViT-300M-448px-V2.5 + Qwen2.5-0.5B-Instruct
InternVL2.5-2B	2.2B	InternViT-300M-448px-V2.5 + internlm2_5-1_8b-chat
InternVL2.5-4B	3.7B	InternViT-300M-448px-V2.5 + Qwen2.5-3B-Instruct
InternVL2.5-8B	8.1B	InternViT-300M-448px-V2.5 + internlm2_5-7b-chat
InternVL2.5-26B	25.5B	InternViT-6B-448px-V2.5 + internlm2_5-20b-chat
InternVL2.5-38B	38.4B	InternViT-6B-448px-V2.5 + Qwen2.5-32B-Instruct
InternVL2.5-78B	78.4B	InternViT-6B-448px-V2.5 + Qwen2.5-72B-Instruct

The two vision encoders reflect an explicit scaling strategy. InternViT-6B-448px-V2.5 has 5.5B parameters, 45 layers, hidden size 3200, 25 heads, uses QK-Norm and RMSNorm, and is trained with NTP after earlier CLIP-era pretraining lineage. InternViT-300M-448px-V2.5 has 0.3B parameters, 24 layers, hidden size 1024, 16 heads, uses standard LayerNorm, and is described as being distilled from InternViT-6B-V1.5 before incremental improvement with NTP (Chen et al., 2024).

The dynamic tiling policy is formalized through aspect-ratio-aware tile selection. The paper gives candidate aspect ratios as

$$\mathcal{R} = \left\{i / j \mid 1 \leq i, j \leq n, \, i \times j \in [n_{\min}, n_{\max}] \right\},$$

with the target grid selected by minimizing the difference between the original image aspect ratio and candidate tiling ratios. The resized dimensions are then multiples of the tile size $S=448$, and for multi-image samples the tile budget is divided approximately evenly across images (Chen et al., 2024). This same tile-based reasoning is visible in the autonomous-driving adaptation of InternVL-1.5, where six camera views are resized, concatenated into a 2688×896 canvas, and decomposed into twelve 448×448 sub-images plus a 448×448 thumbnail (Li et al., 2024).

3. Training pipeline, scaling strategy, and data organization

InternVL 2.5 uses a three-stage training pipeline (Chen et al., 2024):

1. Stage 1: MLP warmup
2. Stage 1.5: ViT incremental learning
3. Stage 2: Full-model instruction tuning

In Stage 1, only the MLP projector is trainable; both the vision encoder and LLM are frozen. The objective is to align vision features to the LLM token space using a pre-training data mixture formatted as ChatML-style conversations and optimized with next-token prediction (NTP). Dynamic high resolution is already used at this stage despite the additional cost (Chen et al., 2024).

In Stage 1.5, the ViT and MLP are trainable while the LLM remains frozen. The stage uses the same pretraining data mixture and NTP loss, but with a lower learning rate intended to avoid catastrophic forgetting and to improve the vision encoder in underrepresented domains such as multilingual OCR, mathematical charts, and other rare visual domains. The paper emphasizes that this stage need only be executed once for a given vision encoder and can then be reused when scaling to larger LLMs (Chen et al., 2024).

In Stage 2, the full model is instruction-tuned on high-quality data. The paper stresses strict data filtering, noting that even a few thousand bad samples can damage behavior, especially for CoT and long-form generation (Chen et al., 2024).

A major practical contribution is the progressive scaling strategy. The basic idea is to optimize InternViT with a smaller LLM during Stage 1.5, then reuse that optimized ViT with larger LLMs and skip Stage 1.5 at larger scales. This is presented as a mechanism for lowering training cost. The paper contrasts Qwen2-VL, reported as processing 1.4T tokens, with InternVL2.5-78B, which is stated to use only about 120B tokens while benefiting from a substantially larger 6B vision encoder (Chen et al., 2024).

The training hyperparameters differ by size, but several settings are global across the series: context length = 16384, and image tile threshold = 48 (Chen et al., 2024). Examples include:

• 1B Stage 1: batch 512, LR 2e-4, tokens ~191B
• 1B Stage 2: batch 512, LR 4e-5, epochs 4, tokens ~176B
• 8B Stage 1.5: batch 1024, LR 1e-5, tokens ~76B
• 78B Stage 1: batch 512, LR 2e-4, tokens ~76B
• 78B Stage 2: batch 512, LR 2e-5, epochs 1, tokens ~44B

For larger models, weight decay is generally 0.05, and ViT drop path in Stage 2 is often 0.4 (Chen et al., 2024).

The paper also introduces a unified loss reweighting scheme for response-length normalization. It presents the weighting as

$$\mathcal{L} = \frac{w_i}{\sum_j w_j} \cdot \mathcal{L}_i, \quad w_i = \begin{cases} \frac{1}{x^0}, & \text{for token averaging} \ \frac{1}{x^1}, & \text{for sample averaging}, \end{cases}$$

and then states that InternVL 2.5 uses square averaging with

$w_i = \frac{1}{x^{0.5}}.$

The stated rationale is to avoid the bias of token averaging toward longer responses and the bias of sample averaging toward shorter responses (Chen et al., 2024). This formulation is later inherited explicitly by InternVL3 (Zhu et al., 14 Apr 2025).

Data organization is another defining feature. Fine-tuning data scale grows from 5.1M samples in InternVL 1.5 to 7.3M in InternVL 2.0 and 16.3M in InternVL 2.5 (Chen et al., 2024). The InternVL 2.5 fine-tuning token composition is reported as:

• single-image: 45.92%
• multi-image: 9.37%
• video: 39.79%
• text-only: 4.92%

The pretraining mixture is conversation-format instruction data spanning captioning, general QA, mathematics, chart understanding, OCR, knowledge, grounding, document understanding, conversation, medical, GUI, video captioning, and video QA. Named sources include COCO-Caption, OpenImages-Caption, Objects365-Caption, LAION-ZH/EN/COCO, ShareGPT4V, GQA, OKVQA, A-OKVQA, MAVIS, GeomVerse, ChartQA, PlotQA, FigureQA, SynthDoG-EN/ZH/RU/JP/KO, TextVQA, InfoVQA, Objects365, GRIT, RefCOCO, DocReason25K, DocVQA, ALLaVA, SVIT, PMC-VQA, Screen2Words, WebSight, Mementos, ShareGPT4Video, VideoChat2-IT, EgoTaskQA, and others (Chen et al., 2024).

The paper is unusually explicit about data filtering. For pure text, it uses LLM-based quality scoring from 0 to 10, repetition detection, heuristic rule-based filtering, and manual checking. For multimodal data, it uses repetition detection, heuristic filtering, and manual review, while exempting some high-quality academic datasets from aggressive filtering. The paper directly links this filtering pipeline to improved CoT behavior and reduced repetitive or deadlocked generations relative to InternVL 2.0 (Chen et al., 2024).

4. Test-time scaling and evaluation methodology

InternVL 2.5 makes test-time scaling a first-class design axis. The paper’s central claim is not only that better training improves performance, but also that difficult multimodal reasoning benefits from additional inference-time computation, especially through Chain-of-Thought prompting and majority voting / self-consistency (Chen et al., 2024).

The most prominent example is MMMU validation. For InternVL2.5-78B, the paper implies a direct-response MMMU score of 66.4, and states that CoT raises it to 70.1, a gain of +3.7 points (Chen et al., 2024). The reported evaluation protocol takes the maximum of direct-answer and CoT performance. For an earlier model, InternVL2-Llama3-76B, the paper reports improvement on MMMU from 62.7 to 65.3 with CoT plus majority voting (Chen et al., 2024).

The paper argues that many open-source MLLMs underperform with CoT because of repetitive or degenerate output behavior, whereas InternVL 2.5 benefits from CoT because stricter data filtering reduced such failure modes (Chen et al., 2024). This suggests that test-time scaling in InternVL 2.5 is not merely a prompting trick, but a design choice coupled to data curation.

The evaluation configuration also depends on modality. For images, tile limits and aspect-ratio-aware tiling control the visual context budget. For multi-image or high-resolution document settings, the paper states that $n_{\max}=24$ or 36 is used; for many low-resolution image datasets, $n_{\max}=6$ or 12; for video datasets, $n_{\max}=1$ (Chen et al., 2024). Video frame sampling in training was increased from 4–24 frames in InternVL 2.0 to 8–32 frames in InternVL 2.5. During evaluation, the paper reports sweeps over 16, 32, 48, and 64 frames on several video benchmarks and uses benchmark-specific settings such as 16 frames for MVBench and 32 frames for CG-Bench (Chen et al., 2024).

Later developments in the family sharpen the meaning of InternVL 2.5’s test-time strategy. InternVL3 adopts Best-of-N evaluation with VisualPRM-8B rather than relying only on CoT and majority voting, and explicitly treats this as a successor recipe rather than a restatement of InternVL 2.5 practice (Zhu et al., 14 Apr 2025). A plausible implication is that InternVL 2.5 represents the stage at which the InternVL line established test-time scaling as a durable principle, even though the later implementation became more specialized.

5. Benchmark profile and reported capabilities

InternVL 2.5 is evaluated across multimodal reasoning, mathematics, OCR, chart and document understanding, multi-image understanding, real-world perception, hallucination benchmarks, visual grounding, multilingual multimodal evaluation, video understanding, and pure-language tasks (Chen et al., 2024). Its strongest public headline is the MMMU result: InternVL2.5-78B = 70.1, compared with GPT-4o-20240513 = 69.1, Claude-3.5-Sonnet = 68.3, Qwen2-VL-72B = 64.5, and InternVL2-Llama3-76B = 62.7 under the paper’s reporting protocol (Chen et al., 2024).

On other reasoning and math-style evaluations, InternVL2.5-78B is reported at 61.8 on MMMU test, 48.6 on MMMU-Pro overall, 72.3 on MathVista, 34.9 / 32.2 on MATH-Vision (testmini / full), and 51.7 on MathVerse (Chen et al., 2024). On OlympiadBench, however, the result remains comparatively modest: 11.6 for InternVL2.5-78B, versus 25.9 for GPT-4o (Chen et al., 2024). This is one of the clearest places where the paper itself presents remaining distance from top proprietary systems.

On OCR-, chart-, and document-heavy evaluations, the model is consistently strong. InternVL2.5-78B scores 89.1 / 95.7 on AI2D (mask / no-mask), 88.3 on ChartQA, 83.4 on TextVQA, 95.1 on DocVQA, 84.1 on InfoVQA, and 854 on OCRBench (Chen et al., 2024). It exceeds GPT-4o on AI2D, TextVQA, DocVQA, and InfoVQA, but trails Qwen2-VL-72B on TextVQA, DocVQA, InfoVQA, and OCRBench (Chen et al., 2024). The paper explicitly acknowledges the MMVet family as a remaining gap even when aggregate multimodal performance is high.

On multi-image understanding, InternVL2.5-78B is reported at 63.8 on BLINK, 77.0 on Mantis-Eval, 55.8 on MMIU, 63.5 on MuirBench, 70.8 on MMT-Bench, and 61.1 on MIRB (Chen et al., 2024). The paper notes that it surpasses GPT-4o on MMIU (55.8 vs 55.7) and MMT-Bench (70.8 vs 65.4), while trailing on BLINK and MuirBench (Chen et al., 2024).

On real-world perception and robustness, InternVL2.5-78B reaches 78.7 on RealWorldQA, 62.9 on MME-RealWorld, 71.4 on WildVision, and 77.2 on R-Bench (Chen et al., 2024). The paper interprets the lower WildVision score relative to GPT-4o (80.6) as evidence that longer, preference-sensitive outputs remain a weakness (Chen et al., 2024).

On broader multimodal benchmarks, the same model scores 2494.5 on MME, 88.3 / 88.5 on MMBench, 87.4 on MMBench-Video, 72.3 on MMVet, 65.5 on MMVet v2, and 69.5 on MMStar (Chen et al., 2024). It is reported as beating GPT-4o on MME, MMBench, and MMStar, while trailing on MMVet v2 and trailing Qwen2-VL-72B on MMVet and MMVet v2 (Chen et al., 2024).

On hallucination-related evaluations, InternVL2.5-78B is reported at 57.4 on HallusionBench, 3.89 on MMHal, 78.8 on CRPE, and 90.8 on POPE (Chen et al., 2024). These numbers are described as competitive and improved over InternVL 2.0, but the paper explicitly states that hallucinations remain, especially in longer responses (Chen et al., 2024).

On visual grounding, InternVL 2.5 is one of the family’s clearest strengths. InternVL2.5-78B reports 93.7 / 95.6 / 92.5 on RefCOCO, 90.4 / 94.7 / 86.9 on RefCOCO+, 92.7 / 92.2 on RefCOCOg, for an average of 92.3 (Chen et al., 2024). The paper calls this state of the art among the compared MLLMs, above Qwen2-VL-72B average 91.1 and InternVL2-Llama3-76B average 90.0 (Chen et al., 2024).

On multilingual multimodal evaluation, InternVL2.5-78B scores 86.3 (en), 85.6 (zh), 85.1 (pt), 84.8 (ar), 83.1 (tr), and 85.4 (ru) on MMMB; 90.0 (en), 89.7 (zh), 87.4 (pt), 83.3 (ar), 84.9 (tr), and 86.3 (ru) on Multilingual MMBench; and 31.9 average on MTVQA (Chen et al., 2024). The authors conclude that multilingual ability appears to be largely inherited from the underlying LLM, since both InternVL2.5-78B and Qwen2-VL-72B use Qwen 2.5 and perform similarly (Chen et al., 2024).

On video understanding, InternVL2.5-78B reaches 72.1 / 74.0 on Video-MME (without / with subtitle), 76.4 on MVBench, 1.97 on MMBench-Video, 75.7 on MLVU, 63.6 on LongVideoBench, and 42.2 / 58.5 on CG-Bench v1.1 (Chen et al., 2024). A key finding is that InternVL 2.5 benefits from increasing frame counts, whereas InternVL 2.0 often degraded when frame count increased (Chen et al., 2024).

Finally, the paper emphasizes preservation of pure-language ability. It reports that InternVL 2.5 mitigates the pure-language degradation observed in InternVL 2.0: for example, InternVL2.5-2B averages 48.4, which is +0.8 over its base LLM, whereas InternVL2-2B averages 39.2, -2.1 below its base LLM; InternVL2.5-78B averages 72.9, +1.4 over its base LLM (Chen et al., 2024). This claim is directly tied to stronger text instruction data and stricter filtering.

6. Limitations, transfer, and historical significance within the InternVL family

InternVL 2.5’s limitations are stated with unusual clarity for a model-release paper. The system remains weaker than GPT-4o on WildVision, which the authors interpret as reflecting weaker performance on longer, preference-sensitive outputs (Chen et al., 2024). It also trails leading proprietary systems on OlympiadBench, and the paper explicitly notes that MMVet v2 remains a gap (Chen et al., 2024). Although CoT behavior is improved, the paper says data filtering alone cannot fully remove repetitive behavior, possibly because some noise is inherited from base-LLM pretraining (Chen et al., 2024). Hallucinations also remain, especially in longer outputs (Chen et al., 2024).

At the same time, adjacent InternVL-family papers show how InternVL 2.5-era design principles transfer beyond the base benchmarks. The Mini-InternVL work demonstrates that a compact derivative using only 1B–4B parameters can preserve InternVL-family behavior through a strong InternViT-derived vision backbone, dynamic resolution, a simple MLP projector, and staged alignment plus instruction tuning (Gao et al., 2024). Its Mini-InternVL-4B variant is reported as retaining about 90% of the average benchmark performance of InternVL2-Llama3-76B while using about 5% of the parameters, with performance defined over MMMU (val), MathVista (testmini), AI2D, ChartQA, DocVQA, InfoVQA, OCRBench, MMBench-EN, and MMBench-CN (Gao et al., 2024). This suggests that InternVL-family capabilities are not solely a product of scale, but also of architectural continuity and training discipline.

The autonomous-driving report based on InternVL-1.5 is similarly informative because it illustrates how the family’s high-resolution tiling and conversational formatting can be specialized without changing the core model. In that work, six camera images are resized to 896×448, concatenated into a 2688×896 canvas, paired with a system prompt describing the camera layout, and full-parameter fine-tuned on DriveLM-nuScenes, producing a final single-model leaderboard score of 0.6002 (Li et al., 2024). Later Mini-InternVL results on autonomous driving, medical imaging, and remote sensing extend the same philosophy: domain adaptation is treated primarily as data reformulation plus supervised fine-tuning, not architectural branching (Gao et al., 2024).

Historically, InternVL 2.5 is also the immediate baseline against which InternVL3 defines itself. InternVL3 keeps the same ViT-MLP-LLM macro-architecture and pixel unshuffle, reuses InternVL 2.5 corpora, inherits techniques such as random JPEG compression, square loss re-weighting, and multimodal packing, but replaces the older multi-stage post-hoc adaptation emphasis with native multimodal pre-training and adds V2PE, MPO, and faster infrastructure (Zhu et al., 14 Apr 2025). In benchmark terms, InternVL3 generally improves on InternVL 2.5—for example, MMMU 72.2 vs 70.0 at 78B—but not uniformly: on visual grounding, InternVL2.5-78B remains slightly stronger than InternVL3-78B (92.3 vs 91.4) (Zhu et al., 14 Apr 2025). This confirms that InternVL 2.5 should not be seen as a superseded anomaly. Rather, it is the mature culmination of the post-hoc modular InternVL paradigm and the direct predecessor against which the family’s next training philosophy was measured.

In that sense, InternVL 2.5’s lasting significance lies in the specific combination it stabilized: large InternViT backbones, simple projector-based fusion, dynamic 448-pixel tiling, response-length-aware loss weighting, aggressive data filtering, broad multimodal instruction mixtures, and test-time CoT scaling (Chen et al., 2024). These features persisted either directly or by explicit contrast in subsequent InternVL-family work, making InternVL 2.5 a pivotal reference point for understanding the modern InternVL line.