PeBR-R1: Two-Stage RL for Visual Reasoning

• The paper introduces a two-stage RL framework that separates perception optimization from reasoning to address visual neglect in VLMs.
• It employs a supervised warm-up followed by GRPO-based RL, routing data into Easy and Medium cases for stable perception and effective reasoning.
• Empirical results on benchmarks like MathVista demonstrate significant performance gains over baseline models through this staged training strategy.

Searching arXiv for the target paper and closely related work on RL for VLM visual reasoning. PeBR-R1 is a vision-LLM trained with a deliberately staged reinforcement learning pipeline that treats visual perception and reasoning as distinct but dependent capabilities. The model is introduced in “Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-LLMs” as a response to the claim that reinforcement learning recipes that work well for LLMs do not transfer cleanly to vision-LLMs, because VLMs must first accurately perceive and understand visual inputs before reasoning can be effectively performed. In this framework, PeBR-R1 is not a new backbone architecture; it is a training procedure applied to pretrained Qwen2.5-VL base models, followed by supervised warm-up and then two-stage GRPO-based reinforcement learning (Chen et al., 16 Sep 2025).

1. Conceptual basis

PeBR-R1—short for a “Perception Before Reasoning” model trained with RL—is defined by the thesis that reasoning-only optimization is insufficient for visual reasoning. The paper argues that VLMs face an extra bottleneck relative to LLMs: before they can reason, they must first correctly perceive the image. In the authors’ framing, directly transplanting reasoning-centric RL from language-only settings is suboptimal because VLMs can fall into “visual neglect,” relying on language priors or superficial textual patterns instead of grounding their reasoning in the image (Chen et al., 16 Sep 2025).

The resulting design principle is explicit: first improve what the model sees, then improve how it reasons over what it sees. The two stages are therefore separated into a perception RL stage and a reasoning RL stage. The stated purpose of this separation is to avoid interference between the two learning problems and to ensure that reasoning is built on a stronger perceptual substrate. A common misconception is that PeBR-R1 is primarily a reasoning model with an auxiliary perception term; the paper instead presents perception optimization as the prerequisite stage.

Another misconception is that PeBR-R1 introduces architectural modifications to the underlying VLM. The paper states that PeBR-R1 is built from Qwen2.5-VL base models in two sizes, 3B and 7B, and does not report architectural modifications to the underlying VLM. Initialization comes from pretrained Qwen2.5-VL, after which the method applies warm-up SFT and then two-stage RL (Chen et al., 16 Sep 2025).

2. Training pipeline and data routing

Before RL begins, the model is warmed up with supervised fine-tuning. The warm-up stage uses the Mulberry-260K dataset as the main SFT source, but the authors report that SFT on Mulberry alone degraded performance on some reasoning tasks. They therefore augment the warm-up data with roughly 39K additional examples drawn from CLEVR-Math, DVQA, TabMWP, VQA2.0, Super-CLEVR, GeoQA+, and a subset of sampled Hard cases. The appendix breakdown given in the paper is 9,730 from CLEVR-Math, 8,676 from DVQA, 4,908 from TabMWP, 3,298 from VQA2.0, 2,398 from Super-CLEVR, 1,336 from GeoQA+, and 8,786 Hard cases. On MathVista with Qwen2.5-VL-7B, the baseline is 68.2, SFT on the original Mulberry data drops to 65.2, and the expanded SFT data brings the warm-up model to 69.8 (Chen et al., 16 Sep 2025).

After warm-up, the authors perform dataset-level sampling by querying the current model with 8 independent rollouts per question. Each question is categorized by how many of those 8 outputs are correct: Easy if all 8 are correct, Medium if 1 to 7 are correct, and Hard if all 8 are incorrect. This partition is central to the method. Hard cases are used partly in warm-up, because RL is ineffective when the model gets everything wrong. Easy cases are used for perception RL, because they are more likely to contain reliable perception traces. Medium cases are used for reasoning RL, because they yield mixed rewards and therefore informative, non-degenerate advantages (Chen et al., 16 Sep 2025).

The RL dataset is described as a 110K-example set produced from various open-source vision-language datasets, with candidate responses generated using Seed1.5-VL and then filtered by keyword-based filtering and manual verification. Of these examples, 36K filtered Easy cases are used in Stage 1 Perception RL and 49K Medium cases are used in Stage 2 Reasoning RL. The paper does not provide a complete split table for all 110K examples, and it does not fully account for the remaining examples (Chen et al., 16 Sep 2025).

3. GRPO and the “vanishing advantage” problem

The two-stage RL framework is built on GRPO. The paper identifies a specific failure mode, termed the “vanishing advantage” problem, as a reason to route data by difficulty rather than train on the full dataset indiscriminately. The group-relative advantage is defined as

$$A_i = \frac{R_i - \mathrm{mean}(\{R_1, R_2, \ldots, R_G\})}{\mathrm{std}(\{R_1, R_2, \ldots, R_G\})}$$

where $R_i$ is the reward for candidate $o_i$ in a group of size $G$. The GRPO objective is written as

$$J_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{q,\, \{o_i\}} \left[ \frac{1}{G} \sum_{i=1}^G \min\big( r_\theta(o_i \mid q) A_i,\, \mathrm{clip}(r_\theta(o_i \mid q), 1-\varepsilon, 1+\varepsilon) A_i \big) - \beta D_{\mathrm{KL}}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$

The paper then notes that if all rewards in a group are identical, then $A_i = 0$ for every sample, and the gradient collapses to the KL term alone. In that regime, RL no longer improves the policy and may instead pull the model back toward the reference policy, causing degradation (Chen et al., 16 Sep 2025).

This diagnosis motivates the Easy/Medium/Hard split as more than a curriculum heuristic. It is also a reward-variance control mechanism. Easy cases are suitable for perception shaping because they are already mostly solved and can provide stable perception supervision. Medium cases are suitable for reasoning shaping because partial correctness yields reward variance. This suggests that PeBR-R1 should be understood as a data-routing strategy as much as a reward-design strategy.

The model is trained to produce a structured response format, including sections such as:

$o_i$0

This structure is operationally important because Stage 1 extracts the Image Description section for perception rewards, whereas Stage 2 scores overall answering and format adherence (Chen et al., 16 Sep 2025).

4. Stage 1: perception reinforcement learning

Stage 1 is designed to improve visual perception rather than final-answer correctness. The paper focuses on the Image Description section of the model’s response and defines rewards that target both coarse-grained and fine-grained visual understanding. For coarse-grained perception, it uses FG-CLIP, a fine-grained image-text alignment model. Let $I_m$ be the image and $I_D$ the model-generated image description. The similarity is

$\text{clipscore} = \text{sim}(I_m, I_D),$

and the reward is

$$r_{\text{clip}} = \begin{cases} 1, & \text{if } \text{clipscore} \geq \tau_{\text{clip}} \ \text{clipscore} / \tau_{\text{clip}}, & \text{otherwise} \end{cases}$$

with $R_i$0 in the appendix (Chen et al., 16 Sep 2025).

For fine-grained perception, the paper uses a teacher-guided keyword reward. A pretrained VLM, Seed1.5-VL, generates structured responses on Easy cases. From the teacher’s Image Description section, the authors extract semantically critical keywords covering object recognition, numerical understanding, attribute comprehension, and spatial relation modeling. If $R_i$1 is the reference set and $R_i$2 is the keyword set extracted from the policy model’s image description, then

$R_i$3

Stage 1 also includes a format reward and a length penalty. The length term is included because the model tends to maximize perception rewards by generating excessively long descriptions, creating loops or redundancy that hurt later reasoning training. The length reward is

$R_i$4

with $R_i$5 (Chen et al., 16 Sep 2025).

The full Stage 1 reward is

$R_i$6

with appendix weights

$R_i$7

The reward composition shows that Stage 1 is primarily a perception-alignment stage, with global alignment and fine-grained keyword matching given the highest weights. The actual final answer is not the main reward target at this stage (Chen et al., 16 Sep 2025).

5. Stage 2: reasoning reinforcement learning

Stage 2 starts from the Stage 1 perception-enhanced policy and shifts the training set to the Medium-case subset. The reward becomes deliberately simpler and more task-oriented:

$R_i$8

with appendix weights

$R_i$9

The paper describes this as a rule-based reward strategy that encourages well-structured and accurate solutions. Final-answer correctness dominates Stage 2, while format remains a small auxiliary reward. The method does not provide a more granular step-level reasoning reward, a separate learned reward model, or a curriculum beyond the transition from Easy to Medium cases (Chen et al., 16 Sep 2025).

The dependence of Stage 2 on Stage 1 is a central claim. Conceptually, the paper argues that better object recognition, counting, attribute understanding, and spatial relation modeling yield more reliable premises for later reasoning. Empirically, the ablations are used to support that claim. On MathVista with the warm-up Qwen2.5-VL-7B baseline, the reported results are 69.8 for the warm-up model, 73.9 for Reasoning RL only, 72.7 for single-stage Perception+Reasoning RL, and 76.0 for two-stage Perception RL + Reasoning RL. A second stage-wise ablation reports 68.2 for no warm-up and no RL, 69.8 for warm-up only, 71.4 for warm-up + Perception RL, 74.3 for warm-up + Reasoning RL, and 76.0 for warm-up + Perception RL + Reasoning RL (Chen et al., 16 Sep 2025).

These numbers are used in the paper to argue against two alternatives: first, that reasoning-only RL is sufficient; and second, that a single RL stage jointly optimizing perception and reasoning is equivalent to staged training. In the reported experiments, neither alternative matches the two-stage pipeline.

6. Empirical performance, limitations, and significance

The main empirical evaluation covers seven benchmarks: MathVista (testmini), MathVision (full), DynaMath, MMStar, AI2D (with the table qualifier “w.M.”), HallusionBench (average), and ChartQA. For PeBR-R1-3B, the reported scores are 71.2 on MathVista, 28.1 on MathVision, 52.0 on DynaMath, 60.5 on MMStar, 81.3 on AI2D, 46.8 on HallBench, and 86.4 on ChartQA. For PeBR-R1-7B, the reported scores are 76.0, 32.7, 56.9, 67.1, 83.3, 56.5, and 89.6 respectively (Chen et al., 16 Sep 2025).

Against same-family baselines, the paper emphasizes large MathVista gains: Qwen2.5-VL-3B scores 62.3, so PeBR-R1-3B improves by +8.9; Qwen2.5-VL-7B scores 68.2, so PeBR-R1-7B improves by +7.8. The paper also highlights that on MathVista, PeBR-R1-3B exceeds InternVL2.5-8B by 6.8 points, and PeBR-R1-7B exceeds Qwen2.5-VL-72B by 1.2 points and InternVL2.5-78B by 3.7 points. On the supplementary MathVerse testmini benchmark for 7B models, PeBR-R1-7B scores 53.6 (Chen et al., 16 Sep 2025).

The length-penalty ablation exposes one of the clearest failure modes in the method. Without the length penalty, mean output length is 485.6 and MathVista is 69.1; with the length penalty, mean output length is 359.2 and MathVista is 71.4. The appendix also compares GRPO with SFT on the same RL dataset, starting from the warm-up model: 69.8 for the warm-up baseline, 76.0 for GRPO, and 73.7 for SFT (Chen et al., 16 Sep 2025).

The paper is explicit about several limitations. Many implementation details are omitted, so exact reproducibility is incomplete. The method relies on a strong teacher model, Seed1.5-VL, for keyword-bearing image descriptions and on manual verification to remove examples with incorrect answers or redundant keywords. The staged data routing depends on repeated rollouts and answer checking, which adds precomputation cost. The authors do not provide extensive robustness or out-of-distribution analyses beyond the benchmark suite and DynaMath. They also do not deeply explore whether the Easy/Medium/Hard thresholds generalize across tasks or models, nor do they report sensitivity to the choice of 8 rollouts (Chen et al., 16 Sep 2025).

Within the literature on RL for multimodal reasoning, PeBR-R1 is therefore best understood not as a new generic RL algorithm but as a task-structured training framework for VLMs. Its technical contribution lies in separating perception optimization from reasoning optimization, routing data according to rollout-based difficulty to avoid vanishing advantages, and defining a perception reward that combines FG-CLIP-based image-text alignment with teacher-guided fine-grained keyword matching. The paper presents this combination as evidence that, in multimodal RL, better reasoning can emerge from better perception objectives delivered first (Chen et al., 16 Sep 2025).