Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning

Abstract: Recent multimodal LLMs have shown great promise in clinical image reasoning, but existing post-training pipelines remain predominantly outcome-centric, relying on final answer correctness or sequence-level preferences. This suffers from sparse credit assignment, making it difficult to optimize the reasoning process essential for clinical applications. Our analysis reveals that cascading errors from early-stage reasoning failures are a leading cause of incorrect predictions in medical visual question answering (VQA) benchmarks. Motivated by this, we propose Medical Reasoning-aware Policy Optimization (MRPO), an RL algorithm that incorporates step-wise process rewards. When the final answer is incorrect, MRPO assigns exponentially larger penalties to tokens in earlier invalid reasoning steps, breaking failure cascades without compromising successful paths. Across three multimodal LLM backbones, MRPO consistently outperforms standard GRPO and a recent RL baseline, and on Qwen3-VL-8B-Instruct even surpasses substantially larger medical MLLMs such as HuatuoGPT-Vision-34B by 2.79 points. Moreover, MRPO reduces early-stage reasoning failures from 64.0% to 13.0%, showing that targeted mitigation of cascading failures improves both reasoning quality and final answer accuracy. Our code is available at https://github.com/dmis-lab/MRPO

• MRPO achieves higher accuracy in correct answers and reduces early-stage failures by selectively reweighting tokens based on failure points in medical multimodal reasoning tasks, outperforming baseline models.
• The First Failure Point (FFP) and Failure Accumulation Rate (FAR) metrics reveal that early failures in reasoning steps significantly impact overall performance, justifying the need for targeted intervention in the training process.
• The MRPO algorithm integrates step-wise reasoning rewards and an exponential penalty structure for early invalid steps to improve answer accuracy and reasoning quality.

Motivation and problem statement

This paper addresses a structural weakness in the post-training of multimodal LLMs (MLLMs) for medical visual question answering (VQA): outcome-centric reinforcement learning. Most GRPO-style pipelines supervise only through final-answer correctness or sequence-level preferences, distributing learning signals uniformly across all tokens. The authors identify two consequences. First, sparse credit assignment makes it impossible for the training signal to localize which intermediate reasoning step caused a failure, which is especially severe in free-form generation where rewards are delayed until the full response is emitted. Second, they document an empirical failure mode—cascading errors—in which an early invalid reasoning step propagates through subsequent steps and drives final prediction errors.

The paper's central diagnostic finding comes from sentence-level analysis of reasoning traces on open-ended VQA-RAD, SLAKE, and PathVQA using four models (Qwen3-VL-8B-Instruct, InternVL3-8B-Instruct, HuatuoGPT-Vision-7B, Lingshu-7B). Two metrics are defined: the First Failure Point (FFP), the relative position $k/K$ of the first invalid step, and the Failure Accumulation Rate (FAR), the fraction of failed steps after that first failure. Earlier FFPs correlate monotonically with higher incorrect rates, and incorrect instances exhibit substantially higher FAR than correct ones, particularly when failures begin early. This motivates targeting the location of failure rather than merely its existence.

Step validity is judged by GPT-5-mini against two complementary criteria: Gold Alignment (consistency with MedThink gold rationales) and Answer Contribution (whether the step supports deriving the ground-truth answer), with a step counted as valid if it passes either criterion—an acknowledgment that multiple valid diagnostic pathways can lead to the same conclusion. Human–LLM alignment studies support this protocol: overall Cohen's $\kappa$ is 0.717 for answer correctness and 0.712 for the composite process reward, both indicating substantial agreement.

The MRPO algorithm

MRPO builds on GRPO and reshapes advantages at token level based on where reasoning fails. The total reward combines three components: an answer reward (weighted ROUGE-1/BLEU-1 plus BERTScore computed with BiomedBERT), a step-wise reasoning process reward from the LLM judge over sentence-segmented steps, and a length reward penalizing traces outside 4–10 steps.

The core mechanism is selective, position-dependent advantage shaping. When the final answer is judged incorrect ($R_{\mathrm{ans}} \le \tau = 0.6$) and a step is invalid, tokens in that step receive advantage $$\hat{A}_{i,t} = -\exp\!\left(1 - \frac{k-1}{K-1}\right)|A_i|$$, so earlier failed steps are penalized exponentially more strongly; tokens in valid steps of incorrect trajectories receive $+|A_i|$. Crucially, when the answer is correct, no reweighting occurs—the method deliberately avoids disrupting successful trajectories. This selectivity matters: ablations show Full Reweighting (applying reshaping to all instances) costs 1.75–1.80 average points versus selective reshaping on Qwen3-VL-8B, confirming that indiscriminate correction degrades already-correct reasoning patterns.

Two further design choices are validated by ablation. Among uniform, linear, quadratic, and exponential shaping functions, exponential yields the largest gains (+2.63 and +3.90 points over uniform on Qwen2.5-VL-7B and Qwen3-VL-8B respectively), consistent with the hypothesis that the earliest failures deserve the steepest penalties. Additionally, SFT cold-start initialization before RL improves in-distribution performance but consistently degrades out-of-distribution transfer—for example, dropping Qwen3-VL's OOD average from 28.94 to 25.05—so the main configuration trains RL directly without cold-start.

Empirical results

Training uses only 13,381 open-ended QA instances aligned one-to-one with MedThink rationales. Evaluation covers three in-distribution test splits and five out-of-distribution benchmarks (PMC-VQA, VQA-Med-2021, Quilt-VQA, RadImageNet-VQA, MIMIC-Ext-MIMIC-CXR-VQA), scored via binary LLM-as-judge accuracy.

Backbone	Base	GRPO	GDPO	MRPO
Qwen2.5-VL-7B-Instruct	23.36	26.06	25.92	26.79
Qwen3-VL-8B-Instruct	26.83	28.69	29.01	29.09
InternVL3-8B-Instruct	28.07	30.84	30.11	31.94

MRPO achieves the highest average across all eight benchmarks on all three backbones, with improvements of 0.73, 0.40, and 1.10 points over GRPO respectively. On Qwen3-VL-8B-Instruct, MRPO reaches 28.94 average on out-of-distribution benchmarks alone—a 7.05-point gain over its base model on RadImageNet-VQA—and surpasses the much larger HuatuoGPT-Vision-34B (26.15) by 2.79 points, as well as the reasoning-tuned Qwen3-VL-8B-Thinking (26.73) by 2.21 points. The authors interpret this as evidence that targeted step-level supervision can substitute for large-scale medical instruction tuning, though this claim rests on comparisons at differing scales and data recipes rather than a controlled scaling study.

Reasoning-quality analyses confirm the causal mechanism. Early-stage (FFP 0.0–0.4) failures fall from 64.0% in the baseline to 13.0% under MRPO, versus 21.2% for GRPO and 21.4% for GDPO, with failures redistributed toward late stages (47.0%, up from 9.6%). FAR in the earliest FFP bin drops to 43.3% under MRPO versus 64.6% baseline, 62.9% GRPO, and 58.4% GDPO, indicating improved recovery even when early failures occur. An instance-level paired comparison between GRPO and MRPO sharpens this picture: of instances failing early under GRPO that MRPO subsequently answers correctly, 92.9% move out of the early stage under MRPO (30.5% resolved entirely); conversely, failures newly introduced by MRPO concentrate in the late stage (25.0%) or involve no detectable failure point (47.4%), consistent with residual errors being terminal terminology mismatches rather than cascading reasoning corruption. Qualitative case studies reinforce this asymmetry.

Robustness checks

Because GPT-5-mini serves both as process-reward judge during training and as evaluator, evaluator-aligned overfitting is a legitimate concern. Cross-judge evaluation with GPT-5.4 and Claude-4.5-haiku preserves the relative ordering: MRPO beats GRPO on every backbone under all three judges, and the monotonic reduction of early-stage failures holds throughout. Judge quality does govern absolute performance, however—Med-PRM, which receives no image input, produces notably degraded results, underscoring that accurate identification of invalid visual reasoning steps is prerequisite for the method to work. Training overhead is modest: roughly $215 versus $193 for GRPO per epoch (~12%), attributable mainly to longer generated traces rather than additional judge calls.

Limitations and open questions

Three limitations are conceded directly. First, MRPO depends on a frontier API judge (GPT-5-mini) for step-wise rewards, introducing cost and dependency; weaker judges yield notable degradation, so the effectiveness of the approach is conditional on judge strength. Training or distilling dedicated medical process reward models matching frontier-judge quality locally remains open. Second, the framework requires gold reasoning annotations from MedThink; extension to annotation-free settings is unexplored. Third, evaluation is confined to medical VQA—generalization of cascading-failure mitigation to other multi-step domains (scientific QA, legal reasoning) is asserted as plausible but not verified. One may also note that the correctness threshold $\tau$ is calibrated on the same judge used during training, and the paired analysis suggests remaining headroom lies in terminal answer grounding rather than reasoning itself, a direction the paper identifies but does not address.

Conclusion

The paper contributes both an empirical characterization of cascading reasoning failures in medical multimodal VQA and a targeted algorithmic response. MRPO demonstrates that redistributing GRPO's learning signal according to failure position—with exponential penalties concentrated on the earliest invalid steps of incorrect trajectories—consistently improves answer accuracy and reasoning quality across three backbones, reduces early-stage failures from 64.0% to 13.0%, and narrows the gap between 8B-scale generalist models and substantially larger medical MLLMs. Its principal dependencies—an external frontier judge and gold rationales—define the clearest boundaries of the contribution and the most concrete questions left unresolved.