OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks

Abstract: Group Relative Policy Optimization (GRPO) has emerged as the de facto Reinforcement Learning (RL) objective driving recent advancements in Multimodal LLMs. However, extending this success to open-source multimodal generalist models remains heavily constrained by two primary challenges: the extreme variance in reward topologies across diverse visual tasks, and the inherent difficulty of balancing fine-grained perception with multi-step reasoning capabilities. To address these issues, we introduce Gaussian GRPO (G$^2$RPO), a novel RL training objective that replaces standard linear scaling with non-linear distributional matching. By mathematically forcing the advantage distribution of any given task to strictly converge to a standard normal distribution, $\mathcal{N}(0,1)$, G$^2$RPO theoretically ensures inter-task gradient equity, mitigates vulnerabilities to heavy-tail outliers, and offers symmetric update for positive and negative rewards. Leveraging the enhanced training stability provided by G$^2$RPO, we introduce two task-level shaping mechanisms to seamlessly balance perception and reasoning. First, response length shaping dynamically elicits extended reasoning chains for complex queries while enforce direct outputs to bolster visual grounding. Second, entropy shaping tightly bounds the model's exploration zone, effectively preventing both entropy collapse and entropy explosion. Integrating these methodologies, we present OpenVLThinkerV2, a highly robust, general-purpose multimodal model. Extensive evaluations across 18 diverse benchmarks demonstrate its superior performance over strong open-source and leading proprietary frontier models.

• The paper presents a novel Gaussian GRPO method for robust, distributional reward normalization to stabilize multi-domain RL training.
• It integrates task-specific response length and entropy shaping to balance fine-grained perception with multi-step visual reasoning.
• Experimental results show substantial improvements on benchmarks including VQA, mathematical reasoning, chart understanding, and spatial grounding.

OpenVLThinkerV2: Generalist Multimodal Reasoning via Gaussian Group Relative Policy Optimization

Introduction and Motivation

The proliferation of multimodal LLMs (MLLMs) has enabled progress across domains such as visual reasoning, chart and document understanding, OCR, visual grounding, and scientific VQA. However, reinforcement learning (RL) alignment for these models remains challenging due to extreme variance in reward topologies—a mixture of task reward sparsity, heavy-tailed distributions, and incommensurable reward scales. Standard RL protocols—specifically Group Relative Policy Optimization (GRPO)—suffer from gradient instability and optimization pathology when simultaneously training across divergent domains. The paper introduces OpenVLThinkerV2 (2604.08539), a generalist multimodal model specifically optimized to address these issues. Central to its method is Gaussian GRPO (G$^2$RPO), a non-linear, distributional matching technique that enforces inter-task gradient equity and is robust to outlier rewards.

Gaussian GRPO: Distributional Matching for Advantage Normalization

The key innovation is the replacement of linear moment-based REINFORCE normalization mechanisms with non-linear distributional alignment of the reward/advantage. In standard GRPO and EMA-GRPO, normalization either introduces inter-task or intra-task imbalances or leaves the optimization vulnerable to distributional pathologies due to higher-order reward moments and outliers.

G$^2$RPO leverages 1D optimal transport using quantile mapping: for each task, empirical reward distributions are mapped to a strict $\mathcal{N}(0, 1)$ target via inverse CDF transformations. This ensures that outliers are mathematically capped, heavy tails are neutralized, and advantages are distributed symmetrically across reward sign. Importantly, the approach inherently produces advantages with unit variance and zero mean per task, harmonizing the scale of learning signals across diverse domains.

Figure 1: G$^2$RPO ensures outlier robustness, symmetric reward handling, and uniform variance across tasks by enforcing Gaussian distribution topology.

Task-level Shaping: Balancing Reasoning and Perception

A persistent challenge in MLLM RL post-training is the simultaneous enhancement of fine-grained perception and multi-step visual reasoning. The optimization trajectories of response length and entropy diverge dramatically between vision-centric (e.g., grounding, OCR) and reasoning-centric (e.g., Math VQA, scientific QA) tasks. OpenVLThinkerV2 proposes two domain-specific shaping schemes:

• Response Length Shaping: Implements a task-dependent trapezoidal reward envelope on generation length, incentivizing concise completions for vision-centric benchmarks (reducing hallucination and spurious reasoning) and extended chains for reasoning-centric tasks.
• Entropy Shaping: Regularizes token entropy within a task-specific margin, simultaneously preventing entropy explosion (rambling, incoherence) and collapse (over-confident, rigid outputs).

  Figure 2: G$^2$RPO enables early convergence, scales up reasoning length for complex queries, and mitigates overthinking/hallucination for perception-heavy inputs.

  

  Figure 3: Task-level entropy shaping stabilizes model exploration behavior, mitigating both entropy explosion and collapse in a domain-aware fashion.

These shaping approaches are complementary: while length rewards are effective as a regularizer across most domains, entropy shaping is particularly impactful for complex/ambiguous tasks or OOD data, highlighting the difficulty-aware regularization properties of the methodology.

Experimental Results

The OpenVLThinkerV2 model, anchored on the Qwen3-VL-Instruct-8B backbone and trained via G$^2$RPO and task-level shaping, demonstrates superior and consistently state-of-the-art results on 18 benchmarks, covering general VQA, mathematical reasoning, chart/document understanding, visual grounding, and spatial logic.

Figure 4: Performance improvement (relative) of OpenVLThinkerV2 over its baseline Qwen3-VL-Instruct-8B across diverse visual tasks.

The magnitude of improvement is particularly notable on:

• General Scientific VQA: 71.6% MMMU (outperforming GPT-4o and Gemini 2.5 Pro).
• Math: 79.5% MathVista.
• Chart Understanding: 87.4% ChartQA.
• Visual Grounding: 93.4%/88.2%/90.4% RefCOCO/RefCOCO+/RefCOCOg, surpassing domain-specific detectors like Grounding DINO.
• Spatial Reasoning: 83.1% on EmbSpatial and robust performance on RefSpatial and RoboSpatial, exceeding several expert models and proprietary LLMs (e.g., GPT-5, Gemini).

Ablation analysis confirms that G$^2$RPO is the dominant source of gain, and each shaping component (length, entropy) provides additional, orthogonal improvement.

Training Stability and Reward Evolution

The training process for OpenVLThinkerV2 is markedly more stable under G$^2$RPO compared to GRPO and GDPO baselines. The model demonstrates faster convergence, less oscillation, and higher final rewards on multiple axes, including accuracy, output format, and output structure.

Figure 5: G$^2$RPO yields stable and superior accuracy reward during validation.

Figure 6: Stable and high length reward evolution under G$^2$RPO.

Figure 7: G$^2$0RPO maintains optimal output format reward across training.

Figure 8: G$^2$1RPO achieves optimal structure reward for format-sensitive tasks.

Theoretical and Practical Implications

By reformulating multi-task RL alignment in MLLMs as a problem of distributional matching (rather than scalar normalization), G$^2$2RPO offers a theoretically principled and empirically validated approach to harmonizing update signals in environments with heterogeneous reward structure. The connection between OT-based advantage normalization and inter-task gradient equity generalizes to any multitask RL context that faces reward incomparability or heavy-tail outliers, including code generation, GUI tasks, and interactive agentic reasoning.

Task-level shaping (length and entropy) introduces an additional axis of optimization regularization and can be readily incorporated to expedite convergence and mitigate overfitting/hallucination phenomena in largescale multimodal models.

Conclusion

OpenVLThinkerV2 establishes a new standard in generalist multimodal reasoning through robust multitask RL post-training. The introduction of Gaussian GRPO (G$^2$3RPO) addresses foundational obstacles in joint task optimization, and its success is amplified by domain-aware length and entropy shaping. These techniques collectively deliver strong empirical gains, stable training, and improved sample efficiency across a wide array of visual-linguistic AI domains. Future research can extend these concepts to broader RL-for-LLM applications facing similar reward variance and scale heterogeneity, and further explore automated/shaped regularization schedules to optimize task-specific behavioral trends (2604.08539).