Reinforcing Multimodal Reasoning Against Visual Degradation

Abstract: Reinforcement Learning has significantly advanced the reasoning capabilities of Multimodal LLMs (MLLMs), yet the resulting policies remain brittle against real-world visual degradations such as blur, compression artifacts, and low-resolution scans. Prior robustness techniques from vision and deep RL rely on static data augmentation or value-based regularization, neither of which transfers cleanly to critic-free RL fine-tuning of autoregressive MLLMs. Reinforcing reasoning against such corruptions is non-trivial: naively injecting degraded views during rollout induces reward poisoning, where perceptual occlusions trigger hallucinated trajectories and destabilize optimization. We propose ROMA, an RL fine-tuning framework that modifies the optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance. A dual-forward-pass strategy uses teacher forcing to evaluate corrupted views against clean-image trajectories, avoiding new rollouts on degraded inputs. For distributional consistency, we apply a token-level surrogate KL penalty against the worst-case augmentation; to prevent policy collapse under regularization, an auxiliary policy gradient loss anchored to clean-image advantages preserves a reliable reward signal; and to avoid systematically incorrect invariance, correctness-conditioned regularization restricts enforcement to successful trajectories. On Qwen3-VL 4B/8B across seven multimodal reasoning benchmarks, our method improves robustness by +2.4% on seen and +2.3% on unseen corruptions over GRPO while matching clean accuracy.

• The paper introduces ROMA, a reinforcement learning framework that enforces robustness against visual degradations in multimodal reasoning.
• It employs a dual-forward pass strategy with multi-view sampling and worst-case token-level KL regularization to prevent reward poisoning.
• Empirical results show ROMA maintains parity on clean inputs while significantly improving accuracy under severe and out-of-distribution degradations.

Reinforcement Learning-Based Robustness for Multimodal Reasoning in Degraded Visual Contexts

Problem Context and Motivation

Autoregressive Multimodal LLMs (MLLMs), fine-tuned with Reinforcement Learning (RL), have achieved substantial advances in multimodal reasoning. However, their performance is critically brittle when confronted with real-world visual degradations—such as blur, compression artifacts, and low-resolution scans—despite strong accuracy on curated, clean datasets. Conventional robustness techniques grounded in static data augmentation or value-based RL regularization are architecturally incompatible with critic-free RL schemes (e.g., GRPO) in large autoregressive MLLMs. Further, naive approaches to robustness via degraded rollouts are hampered by reward poisoning, which destabilizes optimization and induces policy collapse under hallucinated or occluded observations.

ROMA: Robust Optimization Framework

The paper introduces ROMA, a reinforcement learning fine-tuning framework explicitly designed to enforce reasoning robustness against visual degradation in critic-free, autoregressive multimodal generative models (2605.09262). Rather than traditional value-based regularization or direct data augmentation, ROMA modifies optimization dynamics with a dual-forward pass: standard RL rollouts produce trajectories on clean images, followed by teacher-forced re-evaluation on multiple degraded views. Crucially, new rollouts are never sampled from degraded inputs, preventing reward drift from perceptual occlusions.

Regularization Schemes

Three central regularization mechanisms are integrated:

• Token-Level Surrogate KL Penalty (Worst-Case Invariance): For each clean trajectory, the most adversarial degraded view (highest divergence) receives a token-level KL regularizer, enforcing distributional consistency and maximizing resilience against perturbations.
• Auxiliary Policy Gradient Loss: An additional policy gradient is computed on a randomly sampled degraded view, but critically anchored to the clean-image advantage signal. This prevents policy collapse and maintains a reliable reward structure.
• Correctness-Conditioned Regularization: All invariance penalties are gated—only applied to trajectories validated as correct under the clean input—so robustness learning is not propagated to systematically erroneous reasoning.

Optimization Objective

The consolidated objective $J_{\text{total}}$ combines the main RL objective, auxiliary PG loss for robustness, and correctness-conditioned KL penalty against worst-case visual augmentation. The multi-view sampling (typically $K=3$) ensures meaningful adversarial supervision. Hyperparameter sensitivity studies confirm the necessity and effectiveness of balanced weighting between robustness and learning signals, with $\alpha=\beta=0.10$ as optimal coefficients.

Empirical Evaluation

ROMA was fine-tuned on Qwen3-VL 4B and 8B models using the MMRL30k dataset. Evaluation spanned seven multimodal benchmarks encompassing mathematical reasoning (Math Vista, WeMath), visual logic inference, chart understanding (ChartQA), and general visual puzzles, with degraded inputs drawn from both seen (e.g., Gaussian noise, JPEG, blur) and strictly OOD (motion blur, pixelation, speckle, posterization) pools at severity levels exceeding those encountered during training.

Strong numerical results include:

• Clean Input Performance: ROMA achieves parity with GRPO (68.7% for 8B; 68.2% for 4B), demonstrating no deterioration of core reasoning capability.
• Seen Degradation Robustness: ROMA improves average accuracy by +2.4% (61.6% vs 59.2% for GRPO in 8B) under severe degradation.
• Unseen (OOD) Degradation Robustness: ROMA yields +2.3% higher accuracy (56.3% vs 54.0% for GRPO in 8B) and a smaller clean-to-OOD drop (12.4% vs 14.9%).
• Performance Across Degradation Magnitudes: ROMA consistently minimizes accuracy degradation at the highest severity level, outperforming all baselines and achieving superior resilience.
• Ablation and Sensitivity: Removing worst-case multi-view optimization, auxiliary PG loss, or correctness conditioning each significantly degrades performance (by 1.6–2.2%), confirming their necessity.

Theoretical and Practical Implications

ROMA demonstrates that adversarial worst-case invariance, correctness gating, and anchored advantage signals are synergistic for robustness in RL-fine-tuned multimodal models, even without value networks. By never sampling degraded rollouts, the framework avoids reward poisoning and hallucinated reasoning, a key distinction from prior works (e.g., NoisyRollout).

ROMA’s methodology is transferrable to other critic-free RL settings, suggesting possible extension to video-based multimodal reasoning or adaptive penalty schemes sensitive to input degradation severity. The approach potentially sets a new baseline for practical MLLM deployment in visually noisy, real-world environments—where reasoning stability is non-negotiable.

Speculation on Future Directions

Temporal extension to video reasoning, dynamic regularization weighting contingent on estimated degradation, and adaptive adversarial augmentation are immediate avenues for investigation. Such research will likely yield advances in both robustness and interpretability for complex multimodal agents operating under real-world uncertainty.

Conclusion

ROMA addresses a critical gap in multimodal RL—robust reasoning under visual degradation—by integrating correctness-conditioned invariance penalties, worst-case optimization, and auxiliary PG regularization. This framework achieves improved accuracy and minimized degradation loss on both seen and OOD corruptions, without sacrificing clean input performance. The results substantiate the efficacy of token-level adversarial regularization and anchored reward signals for scalable, practical robustness in MLLMs (2605.09262).