MathVis-Fine: Aligning Visual Supervision with Necessity via Progressive Dependency-Guided Training for Multimodal Mathematical Reasoning

Abstract: Chain-of-Thought (CoT) reasoning has extended from purely linguistic domains to multimodal scenarios; however, existing approaches often treat visual inputs as homogeneous or auxiliary signals, failing to capture the intricate and sample-specific dependencies between text and images in mathematical problem-solving. This gives rise to two core issues: first, the supervisory signals for visual content are generalized and coarse-grained, lacking adaptation to the actual necessity of visual information in each sample; second, training feedback becomes inaccurate when visual rewards are uniformly applied without distinguishing the complementary relationships among inputs. These limitations hinder models from achieving precise multimodal reasoning. In this work, we propose a framework for modeling fine-grained visual dependencies in mathematical reasoning. We first construct the MathVis-Fine dataset, augmenting fine-grained visual annotations with visual dependency ratings. Building upon this dataset, we introduce a two-stage progressive visual enhancement training paradigm that balances answer correctness rewards and visual grounding rewards according to the intrinsic visual dependency level of each sample, thereby mitigating reward bias and improving supervision accuracy. Extensive experiments demonstrate that the MathVis-Fine framework effectively enhances visual perception progressively based on visual dependency, offering a more precise training framework for multimodal mathematical reasoning. We will release the dataset upon acceptance.

• The paper introduces MathVis-Fine, a framework that employs fine-grained visual dependency annotations and a two-stage training process (SFT and RL) to align visual supervision with problem necessity in multimodal math reasoning.
• It leverages dependency-aware loss functions and visual reward fusion to reduce hallucinations and enhance accuracy across benchmarks like MathVista and GeoQA.
• Experimental results show a strong correlation between visual retrieval and answer correctness, emphasizing the need for context-sensitive visual supervision in mathematical problems.

Progressive Dependency-Guided Visual Supervision for Multimodal Mathematical Reasoning

Motivation and Problem Formulation

Multimodal mathematical reasoning, which requires intricate combination of linguistic and visual modalities, presents domain-specific challenges not adequately addressed by existing MLLM pipelines. Prior approaches generally impose homogeneous or context-agnostic visual supervision, disregarding the heterogeneity in visual dependency among mathematical problems. This leads to two primary deficiencies: excessive or insufficient visual grounding depending on the actual relevance of visual material, and reward misalignment in policy optimization due to undifferentiated weighting of visual and logical signals.

To resolve these, the "MathVis-Fine" framework (2606.17888) couples dataset refinement with algorithmic innovation, targeting both data-centric and model-centric visual dependency alignment.

MathVis-Fine Dataset: Fine-Grained Annotation and Visual Dependency Modeling

Central to this work is the MathVis-Fine dataset, which contains approximately 5,400 visual math problems. Each problem is annotated with a (i) step-wise visual-text alignment and (ii) explicit visual dependency score $\lambda_v \in \{0.0, 0.5, 1.0\}$, capturing Low, Medium, and High visual necessity.

The distribution over these dependency categories ensures broad coverage, from purely text-based to visually indispensable mathematical problems.

Figure 1: Distribution of annotated visual dependency levels ($\lambda_v$) within MathVis-Fine, highlighting the proportion of high/medium/low dependency samples.

Visual dependency annotation follows a rule-based protocol refined with manual checks: $\lambda_v=1.0$ implies essential visual content for correct problem resolution; $\lambda_v=0.5$ identifies visual-material-enhanced but textually solvable problems; $\lambda_v=0.0$ marks decorative or redundant visual signals. This fine granularity is directly leveraged during model optimization, in both loss functions and reward mixing.

Two-Stage Progressive Training: Synergizing SFT and RL

The training methodology proceeds in two explicitly dependency-aware stages:

Stage 1: Retrieval-Perception Synergy in SFT

The initial phase uses supervised fine-tuning on MathVis-Fine, integrating a Retrieval-Perception Synergy Loss to explicitly couple vision grounding and reasoning. For high- and medium-dependency samples ($\lambda_v \neq 0.0$), the model must not only retrieve precise visual regions but also demonstrate sensitivity to occlusion of these key regions. This is achieved by supplementing the classic BCE loss for retrieval accuracy with a KL-divergence term measuring output consistency when critical visual input is masked.

For samples with $\lambda_v=0.0$, visual supervision is disregarded, focusing entirely on logical consistency.

Stage 2: Multi-Dimensional Visual-Dependent RL (MDVD-RL)

The RL stage uses the reference model from Stage 1 as a policy initializer and applies a customized Group Relative Policy Optimization (GRPO) objective. Here, answer correctness is blended with two orthogonal visual rewards:

1. Visual Indexing Reward ($r_{\text{idx}}$): Penalizes both hallucinated and omitted visual token localization.
2. Visual Content Reward ($r_{\text{con}}$): Uses a strong external judge to confirm semantic grounding of retrieved visual evidence in reasoning chains.

This visual reward is modulated by the sample’s ground-truth $\lambda_v$. For text-only problems, answer correctness solely guides optimization; high-dependency tasks intensify visual reward signals. This selective, dependency-adaptive reward fusion prevents reinforcement of spurious visual priors in linguistics-dominated problems and suppresses visual hallucinations.

Figure 2: Framework overview—dataset construction, synergy-based SFT, and dependency-adaptive, multi-reward RL via GRPO.

Experimental Analysis

The framework is evaluated using MINT-CoT-style interleaved-token architectures, with Qwen2-VL-7B and strong SFT/RL configurations. Evaluation benchmarks include MathVista, MMStar-Math, GeoQA, and the fine-grained HC-M3D, covering general and geometry-specific multimodal reasoning.

MathVis-Fine yields 77.26% accuracy on MathVista (open-source SOTA at 7B scale; +3.56% over MINT-CoT), and scores 66.45% on GeoQA (+1.73% over MINT-CoT). On MMStar-Math, it achieves 71.0%. In HC-M3D, MathVis-Fine attains an overall score of 50.6 with a hallucination (Attribute Generation) error rate of 34.8, the best among open models.

Ablation studies confirm substantial performance drops when either the Synergy Loss, visual reward fusion, or content/index reward is omitted, especially on tasks with high visual dependency. The model is sensitive to reward misalignment, with universal visual reward application in $\lambda_v$0 samples degrading performance due to hallucination and answer noise.

Correlative Analysis between Visual Extraction and Reasoning Accuracy

A core empirical finding is the increasing Pearson correlation between Visual Retrieval Recall and final answer correctness as $\lambda_v$1 increases—rising from near-zero at low dependency to $\lambda_v$20.65 for high dependency, validating the theoretical motivation for dependency-aware reward scaling.

Figure 3: Pearson correlation coefficient grows sharply with visual dependency, confirming that visual extraction accuracy is only predictive in visually grounded questions.

This supports the proposition that reward alignment must be strongly modulated by sample-level visual necessity, a mechanism not present in prior RL pipelines for multimodal LLMs.

Implications and Future Directions

The MathVis-Fine pipeline reifies the principle that visual supervision in MLLMs should be necessity-driven and contextually adaptive. By explicitly quantifying and leveraging problem-specific visual dependence, models (a) avoid visual reward dilution, (b) mitigate visual hallucinations, and (c) better allocate optimization capacity to compositional reasoning.

Practically, this approach is crucial for mathematical domains where visual-text relationships are neither uniform nor consistently critical, especially in subfields such as geometry, chart analysis, and spatial navigation. Theoretically, aligning reward and loss with input signal necessity has broader implications for general MLLM alignment and policy optimization, where extraneous or irrelevant modalities must be dynamically suppressed.

Future developments may explore attribute-specific reward granularity beyond global visual dependency, distilling not only whether but which visual facets (e.g., topology, coordinates, trends) are critical on a per-problem basis. Integration with vision-specialized priors and dynamic expert models will further strengthen this direction.

Conclusion

MathVis-Fine presents a comprehensive dataset and training pipeline to resolve reward misalignment in multimodal mathematical reasoning and achieves strong empirical improvements over general and specialized SFT/RL-based MLLMs. Adaptive, dependency-aware supervision, realized through both dataset annotation and multi-stage training, yields more robust, grounded, and interpretable mathematical reasoning in multimodal LLMs (2606.17888).