Visionary-R1: Visual Reasoning RL Framework

• Visionary-R1 is a reinforcement learning framework that enhances visual reasoning in vision–language models by enforcing a structured caption–reason–answer output.
• It employs a novel GRPO-based training regime with a caption grounding reward to mitigate shortcut learning and improve answer accuracy.
• Empirical results show Visionary-R1 attains new state-of-the-art performance on multiple visual reasoning benchmarks using efficient, CoT-free data.

Visionary-R1 is a reinforcement learning (RL) framework designed to enhance visual reasoning abilities in large vision–LLMs (VLMs), specifically targeting the longstanding challenge of learning general-purpose, robust reasoning from image–question data without reliance on expensive, proprietary chain-of-thought (CoT) supervision. The core innovation of Visionary-R1 is the enforcement of a caption–reason–answer output structure in models trained purely from question–answer (QA) pairs and the introduction of a caption-grounding reward within a Group Relative Policy Optimization (GRPO) regime. This approach mitigates shortcut learning, where models exploit spurious textual cues rather than visually grounded inference, and establishes new state-of-the-art (SOTA) results on major visual reasoning benchmarks with efficient, scalable data and compute regimes (Xia et al., 20 May 2025).

1. Motivation and Shortcut Learning in Visual Reasoning

Vision–LLMs traditionally struggle with general-purpose reasoning due to two major factors: (1) The lack of abundant, high-quality stepwise supervision such as explicit chain-of-thought annotations, and (2) the tendency of RL-trained models to develop “shortcuts,” where superficial correlations in the question dominate over real visual understanding (Xia et al., 20 May 2025). Supervised fine-tuning on QA pairs is insufficient, as it provides no incentive for the model to utilize image features, leading to poor generalization on out-of-distribution (OOD), complex, or ambiguous examples. Moreover, intensive CoT supervision—often distilled from proprietary models such as GPT-4o—is prohibitively expensive and restricts openness.

Visionary-R1’s central hypothesis is that enforcing a structured output format consisting of a detailed caption, an explicit reasoning trace, and a final answer compels the model to ground its reasoning in concrete visual content, thus mitigating shortcut behaviors and enabling robust transfer.

2. Model Architecture and Output Formatting

Visionary-R1 utilizes Qwen2.5-VL-3B as the VLM backbone, which is pretrained on broad image–text corpora but has no explicit post-training for multi-step reasoning. Each output is formatted as a triplet:

<info> [Detailed caption: objects, text, spatial relations, etc.] </info> <think> [Stepwise reasoning chain using caption content] </think> <answer> [Final answer] </answer>

A strict binary format reward is given only if all segments are present. The caption is not a free-form element; it must comprehensively describe visible scene information relevant to the subsequent reasoning (e.g., object identities, notable text, relative positions, and numerical attributes). This explicitly forces the model’s decoding policy to attend to image regions and features in a manner legible for downstream reasoning (Xia et al., 20 May 2025).

3. Reinforcement Learning Objective and Reward Structure

Visionary-R1 is trained solely via RL on 272.6K CoT-free visual QA pairs sourced from eleven major vision reasoning datasets, with no additional stepwise supervision. The RL schedule, based on an extension of the Group Relative Policy Optimization (GRPO) methodology, balances three reward components:

• Answer Accuracy Reward ($r_a$): Binary indicator of final answer correctness.
• Format Compliance Reward ($r_f$): Binary indicator for successful emission of the designated <info>–> –<answer> structure.

  • Caption Grounding Reward ($r_c$): Binary reward, where only the caption is provided as model input, and the predicted answer must still be correct; this assesses the informativeness and relevance of the generated caption.

  The total reward for each generated sample $i$ is: $$R_i = r_a + r_f + \alpha \, r_c, \quad \alpha = 0.1$$ Standardized advantage estimates within each sample group normalize the reward to stabilize learning: $$A_i = \frac{R_i - \mathrm{mean}(R)}{\mathrm{std}(R)}$$

  Optimization proceeds by maximizing the clipped, KL-regularized GRPO objective: $$\mathcal{J}(\theta) = \mathbb{E}\left[ \frac{1}{n} \sum_{i=1}^n \min \left( \frac{\pi_\theta(o_i|q)}{\pi_{\theta_\mathrm{old}}(o_i|q)} A_i,\, \mathrm{clip}\left(\frac{\pi_\theta}{\pi_{\theta_\mathrm{old}}}, 1-\varepsilon, 1+\varepsilon\right) A_i \right) - \hat\beta D_{KL}(\pi_\theta \| \pi_\mathrm{ref}) \right]$$ The KL penalty coefficient is annealed via a cosine schedule to encourage initial exploration and eventual stabilization of policy updates.

  4. Training Corpus and Implementation Details

  Visionary-R1 is trained on a diverse aggregation of QA pairs (no CoT labels) covering general scenes, text documents, scientific charts, diagrams, mathematical problems, and multimodal comprehension tasks. This composition is summarized below.

  Dataset	Size (K)	Format	Answer Type
  A-OKVQA	17.1	General Scene	Multiple-choice
  ChartQA	28.3	Charts	Text/Number
  AI2D	15.5	Diagrams	Multiple-choice
  DocVQA	39.5	Documents	Text
  CLEVR-Math	32.6	3D Scenes	Number
  ...	...	...	...

  The learning rate is $5 \times 10^{-7}$ with group size $n=8$, temperature $0.9$, and 1,500 A800-80G GPU-hours. No CoT annotation, data filtering, or augmentation is applied.

  5. Empirical Results and Ablation Analysis

  Visionary-R1 achieves new SOTA on three out of four major visual reasoning benchmarks, outperforming both closed-source (GPT-4o, Gemini-1.5-Pro, Claude3.5) and strong open-source VLMs. Notably, in the absence of CoT labels or data augmentation:

  Model	MathVista	MathVision	MMStar	MMBench
  GPT-4o*	63.8	31.2	65.1	84.3
  Qwen2.5-VL-3B	62.3	21.2	55.9	79.1
  SFT baseline	54.6	7.0	61.9	80.7
  GRPO baseline	61.8	20.3	54.3	78.6
  Visionary-R1	69.4	24.7	66.5	84.1

  *Results for GPT-4o, etc., from system cards.

  Ablations reveal:

  • Enforcing the caption–reason–answer format yields substantial gains over vanilla RL.

  • The caption reward further enhances both in-domain and transfer accuracy.
  • Cosine-annealed KL regularization outperforms static or linearly decayed alternatives, preventing mode collapse and reward hacking.
  • Qualitative outputs reflect long, visually grounded captions and coherent reasoning traces, in contrast to shortcut-prone, unstructured outputs generated by baseline RL models (Xia et al., 20 May 2025).

  6. Interpretation, Limitations, and Extensions

  Enforcing a caption stage before reasoning is the critical intervention to suppress shortcut learning in RL-trained VLMs. The caption grounding reward prevents the model from emitting vacuous or purely decorative captions. This combination, with careful RL policy update scheduling, enables scalable, data- and computation-efficient attainment of high visual reasoning performance without dependence on proprietary data.

  Limitations include: restriction to a 3B model size; binary caption reward signal (potentially suboptimal vs. learned or continuous visual rewards); all experimentation on single-turn QA (multi-turn dialog, video, and interactive scenarios remain open). Future work may explore richer caption discriminators, larger backbone models, unsupervised alignment, and applicability to multimodal dialogue or video domains.

  7. Comparative and Broader Impact

  Visionary-R1’s paradigm—RL with structure-enforcing, visually grounded rewards on CoT-free data—achieves or exceeds the performance of SFT- or CoT-augmented models, including those with proprietary data. It sets a precedent for open, scalable visual reasoning model development where stepwise supervision is infeasible or undesirable. The methodology and code structure directly influence other recent R1-style VLM frameworks (Shen et al., 10 Apr 2025, Zhan et al., 23 Mar 2025, Yang et al., 13 Mar 2025), and inform practical RL protocols for vision–language alignment, robust reasoning, and generalization in emerging vision-centric AI systems (Xia et al., 20 May 2025).