InternVL 3.5 4B: Efficient Multimodal Model

• InternVL 3.5 4B is a compact multimodal model built on InternViT-300M and Qwen3-4B, integrating vision and language with 4.7B parameters.
• It employs a two-stage cascade reinforcement learning strategy that significantly boosts reasoning performance while reducing GPU hours.
• The model features dynamic visual token compression and decoupled vision-language deployment, achieving up to a 4.05× inference speedup.

Searching arXiv for the cited InternVL papers to ground the article in current preprints. InternVL3.5-4B is an open-source multimodal LLM in the InternVL 3.5 series, built from an InternViT-300M vision backbone and a Qwen3-4B LLM for a total of approximately 4.7B parameters. Its defining contributions are a two-stage Cascade Reinforcement Learning procedure for reasoning-oriented post-training, a Visual Resolution Router for dynamic visual token compression, and a Decoupled Vision-Language Deployment strategy for higher-throughput inference. In the reported evaluation, the model achieves MMMU val 66.6, overall reasoning 57.4, general multimodal overall performance of approximately 80.0%, and a measured 4.05× inference speedup at 896×896 when DvD and ViR are combined; it also supports GUI interaction and embodied agency (Wang et al., 25 Aug 2025).

1. Architectural composition

InternVL3.5-4B follows a “ViT–MLP–LLM” paradigm. The vision backbone is InternViT-300M, with approximately 300M parameters, and the LLM is Qwen3-4B, with approximately 4.4B parameters, yielding a total model size of about 4.7B parameters. The architecture inherits a pixel-shuffle module that compresses 1,024 tokens to 256 tokens per image patch. At the 4B scale, the model does not use MoE. In the Flash variant, InternVL3.5-4B-Flash, a second pixel-shuffle path compresses to 64 tokens under the control of the Visual Resolution Router (Wang et al., 25 Aug 2025).

Component	Specification	Function
Vision backbone	InternViT-300M	Visual encoding
LLM	Qwen3-4B	Multimodal language reasoning
Token compression	1,024 $\rightarrow$ 256 via pixel-shuffle	Visual token reduction

This configuration places the 4B model at the intersection of compact parameterization and full multimodal capability. A plausible implication is that the design prioritizes deployment practicality while preserving a sufficiently large language backbone to support post-training for reasoning and agentic tasks.

2. Cascade reinforcement learning

InternVL3.5 applies reinforcement-learning fine-tuning on top of supervised fine-tuning through a two-stage Cascade RL procedure. The offline stage uses Mixed Preference Optimization (MPO), which jointly minimizes preference, quality, and generation losses:

$$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$

The preference term uses Direct Preference Optimization:

$$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$

with

$$s_\theta(x,y)=\log \pi_\theta(y\mid x)-\log \pi_{\mathrm{ref}}(y\mid x).$$

The quality term $L_q$ is implemented through Behavior Cloning with Optimization on high-quality rollouts, and the generation term is

$$L_g=-\mathbb{E}_{(x,y)}\sum_t \log \pi_\theta(y_t\mid x,y_{<t}).$$

The online stage uses Geometric-mean Sampled-PPO (GSPO). For each prompt $x$, the model generates $G$ responses $\{y_i\}$ and computes normalized advantages

$\hat{A}_i=\frac{r(x,y_i)-\mu_r}{\sigma_r},$

where $$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$0 and $$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$1 are the mean and standard deviation of rewards across the sampled responses. The clipped policy objective is

$$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$2

with

$$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$3

The stated rationale is coarse-to-fine: MPO warms up the policy using existing rollouts, which is described as stable and cheap, and GSPO then refines the policy using self-generated rollouts for a stronger performance ceiling. For the 4B model, the reported effect is an overall gain of approximately +8–12 percentage points on reasoning benchmarks at about 5.8K GPU-hours, compared with about 11K GPU-hours for PPO alone (Wang et al., 25 Aug 2025).

3. Dynamic visual routing and deployment

InternVL3.5-Flash introduces the Visual Resolution Router (ViR) and Visual Consistency Learning (ViCO) to adapt visual-token resolution on a per-patch basis. A binary gate selects a compression factor $$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$4. ViCO first freezes a reference model $$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$5 and trains a policy model $$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$6 to match outputs across compression settings:

$$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$7

Router training then computes

$$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$8

uses the $$L_{\mathrm{MPO}}(\theta)=w_p L_p(\theta)+w_q L_q(\theta)+w_g L_g(\theta).$$9-th percentile $$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$0 of historical $$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$1 to define binary targets, and optimizes the router $$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$2 with binary cross-entropy. At inference, each patch is routed as follows: if $$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$3, it is compressed to 256 tokens; otherwise, it is compressed to 64 tokens (Wang et al., 25 Aug 2025).

The deployment-side complement is Decoupled Vision-Language Deployment (DvD). In this setup, a vision server on GPU Group A hosts the ViT, MLP, and, for Flash, the router; it batches images and computes features $$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$4. A language server on GPU Group B hosts only the LLM, receives $$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$5 in BF16 over TCP or RDMA, and performs prefilling and decoding. The asynchronous three-stage pipeline overlaps vision, transfer, and language execution, with the reported speedup approximated by

$$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$6

Measured on InternVL3.5-4B at 896×896, the baseline serial throughput is 2.71 req/s, DvD alone reaches 5.06 req/s (×1.87), and DvD combined with ViR reaches 10.97 req/s (×4.05) (Wang et al., 25 Aug 2025).

4. Evaluation profile

The 4B model was evaluated without test-time scaling unless noted. On multimodal reasoning and mathematics, the reported seven-task average is 57.4. The constituent results are MMMU val 66.6, representing a +14.8 percentage-point improvement over InternVL3-4B, MathVista 77.1, MathVision 54.4, MathVerse (vision-only) 61.7, DynaMath 35.7, WeMath 50.1, and LogicVista 56.4 (Wang et al., 25 Aug 2025).

On general multimodal evaluation across eight tasks, the model records DocVQA 92.4%, ChartQA 86.0%, InfoVQA 78.0%, TextVQA 77.9%, OCRBench 82.2%, AI2D 82.6%, and MMStar 80.3%, with an overall score of approximately 80.0%. Taken together with the reasoning benchmarks, these results position the model as a compact system whose reported gains are concentrated not only in OCR-heavy and document-centric tasks but also in multimodal mathematical and logical reasoning (Wang et al., 25 Aug 2025).

The evaluation profile also clarifies the intended meaning of “efficiency” in InternVL3.5-4B. It is not restricted to lower parameter count; rather, it is defined jointly by post-training effectiveness on reasoning benchmarks and by throughput gains from DvD and ViR.

5. GUI interaction and embodied agency

InternVL3.5-4B is reported to support two novel capability classes: GUI interaction and embodied agency. On GUI grounding and agentic tests, the model achieves ScreenSpot 83.6%, ScreenSpot-v2 85.1%, OSWorld-G 33.9%, WindowsAgentArena 9.7%, and WebArena-Lite-v2 7.8%. The model is described as being able to ground textual instructions on arbitrary interface screenshots, click buttons, fill forms, and issue OS commands. An internally used example prompt is: “Please open the ‘Settings’ icon in this screenshot, navigate to ‘Display → Brightness’, and set it to 80 %” (Wang et al., 25 Aug 2025).

On embodied spatial reasoning benchmarks, the model records VSI-Bench 54.9%, ERQA 38.5%, SpaCE-10 35.5%, and OmniSpatial 45.8%, for an overall score of 43.7%. The reported capability is reasoning about 3D layouts, paths, and object relations in simulated environments (Wang et al., 25 Aug 2025).

These results are significant because they extend the model’s scope beyond static image-question answering. A plausible implication is that the architecture and post-training regime were designed to support action-conditioned or instruction-grounded multimodal behavior rather than only passive perception.

6. Mechanistic interpretation and causal analysis

A later study on conflict adaptation in vision-LLMs uses InternVL 3.5 4B as the basis for sparse-feature analysis and causal ablation in a sequential Stroop task (Hu, 28 Oct 2025). In that setup, the model is shown a two-word image on each trial, with each word drawn from six color names and rendered in one of the same six font colors. The prompt instructs the model to ignore word identity and name the ink colors in two tokens, first for the left or top item and then for the right or bottom item. Performance is analyzed primarily through the log-probability assigned to the correct second color token, alongside plain accuracy.

Conflict adaptation is operationalized as improved performance on an incongruent trial when it follows another incongruent trial rather than a congruent one, which can be written as

$$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$7

The paper reports that 12 of 13 tested VLMs exhibit behavior consistent with conflict adaptation, and InternVL 3.5 4B is the model on which the sparse-autoencoder-based circuit analysis is performed. Using the transcoder method of Dunefsky et al. (2024) and Huang et al. (2023), followed by coactivation networks as in Deng et al. (2025), the study identifies partially overlapping text and color supernodes. In early layers, the text supernode is larger than the color supernode in layers 3–15 versus 8–11; in late layers, both occur in layers 19–33, with the color supernode larger than the text supernode. The authors interpret this as mirroring the human asymmetry between reading automaticity and the relative difficulty of color naming.

The same study isolates a conflict-modulated supernode in layers 24–25 by contrasting mean activations on II and CI trials. Zeroing all activations in that supernode during the forward pass causes second-word error rates to rise from 17.5% to 59.2% on CI trials and from 2.5% to 20.8% on II trials, with minimal change on CC and IC trials; a new post-ablation failure mode appears in which the model outputs “ink” instead of a color on the second token. At the same time, the paper does not specify the total number of transformer layers, the hidden embedding dimension, or any novel cross-modal attention mechanism for InternVL 3.5 4B, and it does not report formal $$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$8-values, $$L_p=-\mathbb{E}_{(x,y^+,y^-)}\left[\log \sigma\big(s_\theta(x,y^+)-s_\theta(x,y^-)\big)\right],$$9-tests, or confidence intervals for the InternVL-specific log-probability differences or ablation effects (Hu, 28 Oct 2025).

7. Subsequent incorporation into unified multimodal systems

InternVL 3.5 also serves as a foundation for later unified multimodal modeling. In InternVL-U, a 4B-parameter unified multimodal model for understanding, reasoning, generation, and editing, the “InternVL 3.5 multimodal LLM” is used as the multimodal context backbone. That backbone is described as having 28 transformer blocks, hidden size $$s_\theta(x,y)=\log \pi_\theta(y\mid x)-\log \pi_{\mathrm{ref}}(y\mid x).$$0, 16 Q-heads and 8 KV-heads, and FFN size 6,144, and it is coupled with a specialized MMDiT-based visual generation head under a “backbone + head” design with modality-specific stems and decoupled visual representations (Tian et al., 10 Mar 2026).

This later reuse suggests that InternVL 3.5 is not merely a benchmarked MLLM family but also a reusable substrate for broader unified multimodal systems. In that lineage, InternVL3.5-4B is best understood as a compact reasoning- and deployment-oriented model whose architecture, post-training strategy, and systems optimizations made it suitable both for standalone multimodal inference and for incorporation into subsequent frameworks that combine understanding with generation and editing.