OpenVLThinkerV2: Multimodal RL Framework

• OpenVLThinkerV2 is a generalist multimodal reasoning model that stabilizes reinforcement learning across tasks with heterogeneous reward structures.
• It employs Gaussian GRPO (G²RPO) to non-linearly map task rewards to a standard normal distribution, ensuring inter-task gradient equity and robust handling of outliers.
• Task-level shaping with response length and entropy adjustments optimizes both fine-grained perception and extended reasoning across diverse domains.

OpenVLThinkerV2 is a generalist multimodal reasoning model for multi-domain visual tasks that is presented as a multi-task reinforcement learning post-training framework for open-source multimodal LLMs (MLLMs). Its defining objective is to stabilize and equalize reinforcement learning across heterogeneous visual tasks whose rewards have markedly different topologies, while preserving both fine-grained perception and multi-step reasoning. The model is introduced together with Gaussian GRPO, or G$^2$RPO, which replaces linear reward scaling with non-linear distributional matching to a standard normal advantage distribution, and with two task-level shaping mechanisms—response length shaping and entropy shaping—intended to balance concision, extended reasoning, and exploration across domains such as mathematics, chart understanding, document understanding, spatial reasoning, and visual grounding (Hu et al., 9 Apr 2026).

1. Problem setting and motivating constraints

The paper frames OpenVLThinkerV2 around two coupled problems in open-source multimodal post-training. The first is extreme reward-topology variance across tasks. In the reported formulation, math and logic VQA often produce sparse binary rewards, grounding produces dense continuous rewards such as IoU, format and structure rewards impose strong constraints, and some tasks are noisier or more heavy-tailed than others. This creates both intra-task and inter-task imbalance during reinforcement learning. The second problem is the difficulty of balancing fine-grained perception against multi-step reasoning. The paper explicitly argues that a multimodal model should not simply think longer everywhere: reasoning-centric tasks benefit from longer chains, whereas perception-heavy tasks benefit from short, direct outputs that reduce hallucinated reasoning (Hu et al., 9 Apr 2026).

Within that framing, OpenVLThinkerV2 is positioned not merely as a larger or more heavily optimized MLLM, but as a training framework that attempts to make multimodal RL stable and fair across tasks with different reward shapes. This suggests that the work targets a systems-level bottleneck in multimodal post-training rather than a single-domain capability gap. A plausible implication is that the model’s reported breadth depends as much on reward normalization and shaping strategy as on base-model scale.

2. Gaussian GRPO and distributional advantage matching

OpenVLThinkerV2 starts from Group Relative Policy Optimization (GRPO), but the paper argues that standard GRPO is inadequate in multimodal multi-task settings. The reported critique has two parts. First, group-local standard deviation normalization can favor low-variance rollouts and become unstable when reward values are irregular, producing intra-task imbalance. Second, if local standardization is removed, high-variance tasks dominate gradients while low-variance tasks are suppressed, producing inter-task imbalance. The paper further argues that linear scaling only matches mean and variance while preserving the original distribution shape, and therefore cannot equalize tasks whose reward distributions differ in higher-order structure (Hu et al., 9 Apr 2026).

The proposed remedy is Gaussian GRPO (G$^2$RPO), which maps each task’s empirical reward distribution to a standard normal distribution, $\mathcal{N}(0,1)$. The core mapping is

$$\widehat{A}_i = \Phi^{-1}(F_{\mathcal{R}_\tau}(R_i)),$$

with the rank-based probability

$p_i = \frac{\text{rank}(R_i)-0.5}{N},$

and the equivalent form

$$\Psi(R_i,\mathcal{R}_\tau) = \Phi^{-1}(p_i) = \sqrt{2}\,\operatorname{erfinv}(2p_i-1).$$

When multiple responses have identical rewards, the paper states that their assigned quantiles are averaged so that identical behaviors receive identical learning signals. The transport perspective is formalized through a Wasserstein-2 objective in one dimension, where the paper states that the empirical reward distribution is forced to converge to a standard normal target.

The claimed consequences of this Gaussian topology are threefold. First, it provides inter-task gradient equity, because tasks with different raw reward scales are mapped to comparable advantage magnitudes. Second, it improves robustness to heavy-tail outliers, because the mapping is rank-based and therefore “mathematically caps outliers.” Third, it provides symmetric updates for positive and negative rewards, because the target distribution is centered at zero and has balanced tails. The derivation section reportedly summarizes the resulting advantage statistics as approximately distributionally standardized, with $\sum_i \widehat{A}_i \approx 0$ and $\sum_i (\widehat{A}_i)^2 \approx 1$. In the paper’s terminology, this is standardization in a distributional, not merely moment-based, sense.

3. Task-level shaping: response length and entropy

Beyond G$^2$RPO, the paper introduces two task-level shaping mechanisms to balance perception and reasoning. The first is response length shaping. The authors report that, during training, reasoning-centric tasks initially shorten and then eventually need longer chains, while vision-centric tasks tend to become more concise and overthinking is harmful for grounding. To regulate this, the paper defines a trapezoidal reward envelope over response length $|y|$, with task-specific bounds $^2$0, $^2$1, $^2$2, and $^2$3 (Hu et al., 9 Apr 2026).

The reported purpose of this length reward is deliberately bidirectional. It encourages longer reasoning traces when needed, discourages excessively short outputs that omit reasoning, and also discourages excessively long outputs that may hallucinate or waste tokens. The paper’s plain-language summary is explicit: reasoning tasks are nudged to think more; perception tasks are nudged to answer directly. This addresses a recurring misconception in multimodal RL that longer generations are uniformly beneficial. In the reported design, longer outputs are advantageous only under task-specific length targets.

The second mechanism is entropy shaping. The paper reports divergent entropy pathologies across task types: reasoning tasks can exhibit entropy explosion, yielding incoherent outputs from excessive exploration, while perception tasks can exhibit entropy collapse, yielding overly deterministic token choices. The proposed regularizer constrains average task entropy $^2$4 within a safe band:

$^2$5

and this term is added to the final objective with weight $^2$6. The paper characterizes the effect as maintaining enough exploration for reasoning while preventing both collapse and explosion. This suggests a training regime in which exploration control is treated as task-conditional rather than globally optimal.

4. Model initialization and training pipeline

OpenVLThinkerV2 is initialized from Qwen3-VL-Instruct-8B and trained on a filtered subset of OneThinker-600k. The reported optimization setup uses AWS Trainium instances of type Trn1.32xlarge, one epoch, AdamW, batch size 128, learning rate $^2$7, and max generation length 4096. KL regularization is disabled. The pipeline also applies dynamic data filtering that removes rollouts that are uniformly correct or uniformly incorrect, with the stated purpose of keeping gradient signals informative. Training is reported to take about 3 days (Hu et al., 9 Apr 2026).

The paper summarizes the training workflow in four stages: start from Qwen3-VL-Instruct-8B; train with G$^2$8RPO on filtered OneThinker-600k samples; apply task-level length shaping and entropy shaping; and evaluate on diverse multimodal benchmarks. In this formulation, OpenVLThinkerV2 is both a named model and the endpoint of a post-training recipe. A plausible implication is that reproducibility depends not only on the RL objective but also on data filtering and task-conditioned shaping, since those elements are presented as integrated parts of the system rather than optional refinements.

5. Evaluation protocol and reported benchmark performance

The paper evaluates OpenVLThinkerV2 on 18 benchmarks spanning six major domains: general science knowledge and general VQA, mathematics, chart understanding, document understanding, spatial reasoning, and visual grounding. The explicitly listed benchmarks are MMMU, MMBench, MMStar, MathVista, MathVerse, MathVision, AI2D, ChartQA, CharXiv(RQ), DocVQA, OCRBench, InfoVQA, EmbSpatial, RefSpatial, RoboSpatial, RefCOCO, RefCOCO+, and RefCOCOg (Hu et al., 9 Apr 2026).

The reported benchmark scores are as follows.

Domain	Benchmark	Score
General VQA / multimodal understanding	MMMU	71.6
General VQA / multimodal understanding	MMBench	88.2
General VQA / multimodal understanding	MMStar	73.8
Mathematics	MathVista	79.5
Mathematics	MathVerse	65.8
Mathematics	MathVision	53.4
Chart / diagram understanding	AI2D	87.5
Chart / diagram understanding	ChartQA	87.4
Chart / diagram understanding	CharXiv(RQ)	53.0
Document understanding	DocVQA	96.7
Document understanding	OCRBench	911
Document understanding	InfoVQA	86.4
Spatial reasoning	EmbSpatial	83.1
Spatial reasoning	RefSpatial	44.6
Spatial reasoning	RoboSpatial	63.2
Visual grounding	RefCOCO	93.4
Visual grounding	RefCOCO+	88.2
Visual grounding	RefCOCOg	90.4

The paper claims these results establish new SOTA among open-source models and also surpass several proprietary frontier systems on multiple tasks. It explicitly highlights 71.6% on MMMU and 79.5% on MathVista, stating that these surpass GPT-4o, and 87.4% on ChartQA, stating that this exceeds Gemini 2.5 Pro. For document understanding, the paper states that DocVQA: 96.7, OCRBench: 911, and InfoVQA: 86.4 outperform strong proprietary models such as GPT-5 and Gemini 2.5 Pro. In spatial reasoning, the paper states that the model achieves the highest score on EmbSpatial, is close to specialized spatial experts on RoboSpatial, and exceeds GPT-5 and Gemini 2.5 Pro. On visual grounding, the reported RefCOCO: 93.4, RefCOCO+: 88.2, and RefCOCOg: 90.4 are described as state-of-the-art and as exceeding Grounding DINO.

6. Ablation results, interpretation, and relation to adjacent work

The ablation study reports a cumulative improvement pattern across six aggregate domains. Starting from Qwen3-VL-Instruct-8B, the reported scores are 71.3 on General VQA, 59.2 on Math VQA, 69.9 on Chart VQA, 87.1 on Grounding, 86.8 on Document Understanding, and 60.9 on Spatial Reasoning. Adding G$^2$9RPO changes these to 76.9, 64.8, 74.5, 90.2, 90.6, and 62.3. Adding entropy loss yields 77.0, 65.1, 75.3, 90.4, 90.8, and 62.8. Adding length reward yields 77.4, 65.7, 75.4, 90.5, 91.1, and 63.2. The final OpenVLThinkerV2 reaches 77.9, 66.2, 76.0, 90.7, 91.4, and 63.6 (Hu et al., 9 Apr 2026).

The paper interprets these results as showing that G$\mathcal{N}(0,1)$0RPO provides the largest single gain, entropy shaping improves stability especially on reasoning tasks, length shaping provides broader gains and appears stronger than entropy shaping alone, and the combination yields the best overall model. The appendix training curves are further described as showing earlier convergence, more stable accuracy reward, and stronger performance than GRPO and GDPO on length, format, and structure rewards. Taken at face value, this places the work’s primary methodological weight on reward-topology normalization rather than on architectural novelty.

A useful contextual distinction is between OpenVLThinkerV2 and the earlier multimodal reasoning framework “VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Search” (Wang et al., 12 Apr 2025). VisuoThink is a training-free test-time search method for large vision-LLMs built around vision-text interleaved expansion, rollout simulation, and selection, with iterative Thought → Action → Observation cycles and predictive rollout over multimodal reasoning trajectories. OpenVLThinkerV2 addresses a different layer of the stack: it is a multi-task reinforcement learning post-training framework for a generalist multimodal model. This suggests that the two works are complementary rather than interchangeable. One emphasizes inference-time multimodal search without fine-tuning; the other emphasizes stable and equitable RL across heterogeneous training tasks. A common simplification is to treat OpenVLThinkerV2 as merely another multimodal RL model; the paper instead presents it as a framework for making multimodal RL fairer and more stable across diverse visual tasks.