Vision-SR1: Self-Rewarding VLM Framework

• Vision-SR1 is a vision-language model training framework that mitigates visual hallucinations and language shortcuts by decomposing reasoning into distinct visual perception and language reasoning stages.
• The framework employs a novel self-rewarding mechanism with reinforcement learning, applying a composite reward function to supervise both intermediate and final outputs.
• Experimental validations show improved visual reasoning accuracy and reduced false content generation compared to existing models on diverse multimodal benchmarks.

Vision-SR1 is a vision-LLM (VLM) training framework designed to mitigate two principal failure modes—visual hallucination and reliance on language shortcuts—by introducing a self-rewarding, reasoning-decomposed learning paradigm. Unlike previous VLM post-training algorithms that supervise only the final output and frequently require external visual annotations, Vision-SR1 decomposes reasoning into visual perception and language reasoning stages and applies a novel self-reward mechanism based on reinforcement learning to adaptively supervise both components. This method is intended to encourage grounded image understanding and robust language generation by providing balanced training signals at intermediate and final steps.

1. Motivation and Problem Statement

Vision-LLMs such as recent large multimodal transformers commonly exhibit the following deficiencies:

• Visual Hallucinations: The tendency to generate descriptions or answers containing content not present in the visual input, often exacerbated by strong language priors.
• Language Shortcuts: The practice of sidestepping the intended visual reasoning process and instead exploiting linguistic regularities or patterns in the training data.

These problems are rooted in sparse or misdirected visual supervision—i.e., post-training methods primarily optimize for answer correctness and neglect explicit guidance at the intermediate stages of visual reasoning. Attempts to correct this have included adding auxiliary rewards based on human annotations or distilled labels from teacher models; however, these approaches are costly, lack adaptability, and can cause policy drift or reward hacking under evolving model policies (Li et al., 27 Aug 2025).

2. Reasoning Decomposition

Vision-SR1 enforces a strict separation between "visual perception" and "language reasoning" during the answer generation process. The protocol is:

1. Visual Perception Stage: Given a question $q$ and an image $I$, the model produces a text chunk $c$ encapsulated within dedicated tags (e.g., <visual perception>$c$</visual perception>) containing all visual information necessary to answer $q$.
2. Language Reasoning Stage: The model receives only the question $q$ and the generated visual perception $c$ and proceeds to generate chain-of-thought reasoning $t$ (> $t$</think>) and a final answer $a$ (<answer>$a$</answer>), without referring back to $I$.

   The output format is explicitly structured:

   Stage	Format	Description
   Visual Perception	<visual perception>$c$</visual perception>	All necessary visual details
   Reasoning	<think>$t$</think>	Textual intermediate reasoning
   Answer	<answer>$a$</answer>	Final answer

   This separation allows the system to assess whether the intermediate perception $c$ is sufficient for correct downstream reasoning without implicit reference to the image.

   3. Self-Rewarding Mechanism via Reinforcement Learning

   Vision-SR1 utilizes reinforcement learning to supervise both intermediate and final outputs by defining a composite reward function:

   $$r(Q, s) = r_{\mathrm{visual}}(Q, c) + r_{\mathrm{ans}}(Q, a) + \alpha r_{\mathrm{fmt}}(s)$$

   • $Q = (q, I)$ is the original vision-language query.

   • $s$ is the output sequence.
   • $r_{\mathrm{ans}}(Q, a)$ is a binary reward for answer correctness (matches ground truth).
   • $r_{\mathrm{visual}}(Q, c)$ is a binary reward based on secondary rollout: the model is re-prompted with $q$ and $c$ only, and if the derived answer matches the ground truth, $c$ is judged "self-contained."
   • $r_{\mathrm{fmt}}(s)$ is a formatting reward enforcing output structure.
   • $\alpha$ is a tunable hyperparameter for format reward scaling.

   The policy optimization is performed via group relative policy optimization (GRPO):

   $$\hat{A}^{\mathrm{grp}}(Q, s_k) = r(Q, s_k) - \frac{1}{K} \sum_{j=1}^K r(Q, s_j)$$

   $$\mathcal{L}_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{Q}\left[\sum_{k=1}^{K} \hat{A}^{\mathrm{grp}}(Q, s_k) \log \pi_{\theta}(s_k|Q) - \beta \cdot \mathrm{KL}( \pi_{\theta}(\cdot|Q)\|\pi_{\theta_0}(\cdot|Q))\right]$$

   Here, $K$ is the batch size per query, $\beta$ is the KL regularization factor, and $\pi_{\theta}$ is the current policy.

   This framework supplies targeted, variance-controlled gradients to both visual perception and answer-generation modules, encouraging the model to ground outputs in the image rather than purely in linguistic patterns.

   4. Experimental Validation

   Vision-SR1 is tested across diverse multimodal tasks, including general image understanding, multimodal math, and hallucination detection. Empirical results indicate:

   • Visual Reasoning Accuracy: On benchmarks such as MMMU, MM-Vet, and RealWorldQA, Vision-SR1 achieves higher scores compared to established baselines. For example, a Qwen2.5-VL-7B backbone obtained an average score of 58.8 across eight benchmarks, surpassing Vision-R1 (≈57.4).
   • Reduction in Visual Hallucinations: The system demonstrates quantifiable improvements in reducing the rate of false or ungrounded content in responses.
   • Shortcut Avoidance: Language Shortcut Rate (LSR) metrics decrease, indicating improved reliance on visual grounding rather than linguistic guesswork.

   These improvements stem from the explicit supervision at both visual and reasoning stages, which calibrates the learned representations away from shortcut-prone behavior.

   5. Comparison with Existing Approaches

   Prior methods address visual grounding mainly via final answer supervision, optionally supplemented with external annotation-based or teacher-model distilled rewards. Such external signals are inflexible, scale poorly, and may shift distributionally as policy adapts, leading to reward hacking. Vision-SR1's self-contained, adaptive protocol:

   • Requires no external visual annotation or model distillation.
   • Produces supervisory signals dynamically by validating the sufficiency of visual perception via internal dual-rollout.
   • Ensures mutual reinforcement of visual detail extraction and language reasoning.
   • Results in enhanced model robustness and interpretability as all answer rationales are explicitly represented in structured outputs.

   6. Implications and Future Directions

   The Vision-SR1 paradigm suggests new possibilities for scalable, annotation-free, multimodal VLM training. Notable implications include:

   • Increased trustworthiness and interpretability by requiring "explanatory" intermediate outputs before answers.
   • Scalability to new domains without dependence on annotated external visual rewards.
   • Foundations for more rigorous disentanglement of visual grounding from linguistic priors.

   Future work is expected to address enhancements such as direct supervision on visual embeddings (to minimize textual conversion information loss) and systematic quantification of true visual reasoning gains. Comprehensive evaluation of RL-derived improvements in grounding and generalization remains an open challenge.

   7. Summary Table: Vision-SR1 Workflow

   Step	Input	Output	Reward
   Visual Perception	(image, question)	<visual perception>$c$</visual perception>	$r_{\mathrm{visual}}$ (if self-contained)
   Language Reasoning	(question, perception)	<think>$t$ <answer>$a$</answer>	$r_{\mathrm{ans}}$ (answer correct)
   All stages	(question/image)	Structured tags (<visual perception>, <think>, <answer>)	Format adherence $r_{\mathrm{fmt}}$

Vision-SR1 provides robust, decomposed RL supervision for vision-LLMs and demonstrates marked improvements in both visual grounding and answer fidelity on standard benchmarks (Li et al., 27 Aug 2025).