MathLens: Dissecting Multimodal Geometry

• MathLens is a benchmark that dissects multimodal reasoning in textbook-style geometry, categorizing performance into perception, reasoning, and integration.
• It uses symbolic specifications and controlled diagram perturbations to provide clear insights into visual extraction and abstract reasoning.
• Empirical findings indicate that while training boosts perception and reasoning, the integration of these skills remains a primary and persistent challenge.

MathLens is a benchmark designed to disentangle the subskills of multimodal reasoning while preserving the complexity of textbook-style geometry problems. It separates performance into three components—Perception, Reasoning, and Integration—and supports each test with visual diagrams, textual descriptions to evaluate reasoning in isolation, controlled questions that require both modalities, and probes for fine-grained perceptual skills, all derived from symbolic specifications of the problems to ensure consistency and robustness (Chung et al., 2 Oct 2025).

1. Diagnostic objective and research setting

The benchmark was introduced against a background in which evaluation of multimodal reasoning is dominated by aggregate accuracy. That single score is useful, but it does not reveal whether a gain comes from improved extraction of visual facts, improved symbolic inference once facts are available, or improved coordination between those two processes. MathLens therefore asks not only whether a model answers correctly, but why a model succeeds or fails. Its central research question is explicit: when multimodal LLMs improve on visual mathematics after fine-tuning, what exactly are they learning more of—better visual extraction, better abstract reasoning, or better use of visual evidence inside reasoning (Chung et al., 2 Oct 2025).

This diagnostic orientation distinguishes MathLens from benchmarks that vary modalities without fully isolating them. The appendix contrasts MathLens with MathVerse by noting that MathVerse text descriptions often fail to fully encode the geometric relations, so “text-only reasoning” is not a clean reasoning test. MathLens instead derives all benchmark views from a shared symbolic state, which makes its modality separation controlled rather than heuristic. The resulting benchmark is not merely a new leaderboard task; it is a causal-style decomposition of multimodal geometry performance (Chung et al., 2 Oct 2025).

MathLens was also validated as a genuine multimodal reasoning benchmark rather than a purely synthetic probe set. Fine-tuned models consistently outperform their backbones on the main geometry-solving task, and MathLens correlates with established multimodal math benchmarks: Spearman $\rho = 0.83$ with MathVista, $\rho = 0.86$ with MathVerse, and $\rho = 0.67$ with MathVision. The appendix further introduces MathLens-E, a version using original source diagrams instead of fresh renders, and reports that MathLens-E correlates more strongly with MathVision than MathLens does, suggesting that public benchmark performance may partly reflect visual familiarity with known diagram styles (Chung et al., 2 Oct 2025).

A common misconception is that improved aggregate accuracy can be read straightforwardly as improved multimodal reasoning. MathLens rejects that interpretation. Its design assumes that apparent progress may conceal uneven changes across perception, reasoning, and their coupling, and the empirical analysis confirms that these components often move differently under different training regimes (Chung et al., 2 Oct 2025).

2. Formal decomposition of multimodal geometry

MathLens formalizes each problem instance $k$ by a latent semantic state

$S_k=\{s_{k,1},\ldots,s_{k,m}\},$

where each $s_{k,i}$ is an atomic fact such as $\angle ABC=50^\circ$. The gold answer is then defined by

$A_k=f(\varphi_k,S_k),$

where $\varphi_k$ is the question operator. Visual context and textual context are both generated conditionally on the same $S_k$, so the image, the text description, the multimodal question, and the perception probes are aligned views of one semantic object rather than loosely matched annotations (Chung et al., 2 Oct 2025).

Within that structure, Perception is the ability to extract information from raw inputs, especially the diagram. Reasoning is the ability to operate correctly once all required facts are already available. Integration is the ability to select the relevant perceptual evidence and apply it correctly during reasoning. The text-only task operationalizes reasoning by replacing the diagram with a complete textual rendering of the semantic state. Perception is operationalized by atom-level probes. Integration is treated as the residual failure mode: if a model answers the relevant perception probes correctly and also solves the text-only version correctly, but still fails on the original multimodal problem, the failure is attributed to integration (Chung et al., 2 Oct 2025).

The text descriptions are generated clausewise from the symbolic state:

$\rho = 0.86$0

and the perception probes are templated from individual atoms:

$\rho = 0.86$1

MathLens also constructs context-forcing questions by removing facts from the question that overlap with the context. This prevents the question text from restating the needed geometric information and ensures that the benchmark actually requires use of the supplied context (Chung et al., 2 Oct 2025).

This formalization matters because it makes the three capacities testable on the same underlying problem. In ordinary multimodal evaluation, failures are often observationally ambiguous: a wrong answer could reflect bad diagram reading, bad inference, or bad coordination. In MathLens, those possibilities are separated by construction (Chung et al., 2 Oct 2025).

3. Benchmark construction and perturbation design

The main MathLens dataset is built from FormalGeo-7K, which provides symbolic annotations for geometry problems. Symbolic clauses such as $\rho = 0.86$2 are converted into coordinate constraints; the paper gives

$\rho = 0.86$3

as an example. The complete constraint set is solved with SLSQP via scipy.optimize.minimize, using random initializations and up to ten attempts per sample. If no feasible solution is found, the sample is discarded. The solution is then rendered with a matplotlib backend that draws points, segments, arcs, and labels (Chung et al., 2 Oct 2025).

The rendered outputs are manually filtered to remove overlapping vertices, occluded labels, extreme aspect ratios, and degenerate layouts. The benchmark also preserves olympiad-style rendering conventions: diagrams may be intentionally non-canonical and “not drawn to scale,” so an obtuse angle may be drawn acute if that prevents the problem from collapsing into visual estimation. This keeps the benchmark close to textbook and competition geometry rather than to purely perceptual shape recognition (Chung et al., 2 Oct 2025).

For the main geometry benchmark, the paper reports 926 geometry problems, 8 visual modifications per problem, and an average of $\rho = 0.86$4 visual probes per problem. Each problem has the original diagram from the source dataset, a newly rendered base diagram from the symbolic representation, and six semantic-preserving variants: add_shapes, add_lines, flip, rotate, merge, and rename. The merge transformation concatenates the diagram with another random diagram and revises labels; rename relabels points such as $\rho = 0.86$5 (Chung et al., 2 Oct 2025).

The perturbation regime is central to the benchmark’s robustness claims. These are not ordinary pixel augmentations. They are semantic-space perturbations generated from the symbolic specification, and the invariance condition is that the answer remains unchanged under admissible perturbations $\rho = 0.86$6:

$\rho = 0.86$7

This gives MathLens a stronger notion of robustness than blur or noise ablations, because the model is being tested on alternate yet valid realizations of the same geometry (Chung et al., 2 Oct 2025).

The paper also introduces MathLens-General, a complementary broader-domain set with 107 problems and an average of $\rho = 0.86$8 visual probes per problem. It is curated from BLINK, V*, SpatialEval-Real, MMMU-Pro, EMMA, and MathVista, excluding geometry items. The authors explicitly present it as a broader but less formally controlled complement to the main geometry benchmark, because those problems do not come with a formal semantic state like FormalGeo-7K (Chung et al., 2 Oct 2025).

4. Evaluation protocol and error taxonomy

MathLens evaluates the same underlying problem through aligned tasks rather than through one monolithic score. The multimodal task tests the full pipeline; the text-only task tests reasoning once perception is trivialized; the perception probes test recovery of specific visual facts; and the perturbation suite tests invariance under semantic diagram variation (Chung et al., 2 Oct 2025).

Capacity	Input view	Operational meaning
Perception	Diagram + atom-specific probe	Recover facts from raw inputs
Reasoning	Complete textual description + question	Solve when all required facts are explicit
Integration	Residual multimodal failure	Select relevant perceptual evidence and apply it during reasoning

The benchmark’s most distinctive analytical component is its automatic error taxonomy. Outcomes are partitioned into six groups: Perception + Reasoning, Perception, Reasoning, Integration, Trivial, and Rest. “Perception + Reasoning” denotes cases that fail both perception probes and text-only reasoning. “Perception” denotes cases that fail perception probes but succeed on text-only reasoning. “Reasoning” denotes cases that succeed on perception probes but fail text-only reasoning. “Integration” denotes cases that succeed on both perception probes and text-only reasoning, but fail on the multimodal task. “Trivial” covers correct cases solvable from text alone, and “Rest” covers the remaining correct cases (Chung et al., 2 Oct 2025).

Robustness is quantified by Consistency Rate (CR):

$\rho = 0.86$9

Here $\rho = 0.67$0 is the set of diagram variants for question $\rho = 0.67$1, and $\rho = 0.67$2 is the model. CR therefore measures expected agreement of predictions across semantically equivalent diagram variants, not raw accuracy under arbitrary perturbation (Chung et al., 2 Oct 2025).

This evaluation design addresses a second common misconception: that multimodal reasoning is adequately characterized by separate “vision” and “reasoning” scores. MathLens shows that even when those two isolated capabilities look strong, the multimodal pipeline can still fail because cross-modal selection and transfer remain unstable (Chung et al., 2 Oct 2025).

5. Empirical findings on training effects and robustness

MathLens reports that different training approaches have uneven effects. Reinforcement learning chiefly strengthens perception, especially when supported by textual supervision. Textual supervised fine-tuning indirectly improves perception through reflective reasoning. Reasoning improves only in tandem with perception. Integration remains the weakest capacity, with residual errors concentrated there once other skills advance. Robustness also diverges: RL improves consistency under diagram variation, whereas multimodal SFT reduces it through overfitting (Chung et al., 2 Oct 2025).

Concrete model trajectories make the asymmetry explicit. For Revisual-R1-7B, text-only accuracy moves from 62.3 to 63.8 after multimodal RL, while base-diagram accuracy moves from 39.2 to 45.8. For OVR-7B, the corresponding shift is 66.2 → 70.4 on text and 38.7 → 43.7 on the base diagram. These patterns support the authors’ interpretation that multimodal RL improves perception and grounding more strongly than abstract reasoning in isolation (Chung et al., 2 Oct 2025).

The robustness analysis shows an equally strong contrast. For Vision-R1-7B, consistency is 31.0 at the backbone stage, 21.1 after multimodal SFT, and 39.4 after multimodal RL. For R1-OneVision-7B, the sequence is 31.0, 24.0, and 28.5. By contrast, text-SFT followed by multimodal RL yields steadily increasing consistency for Revisual-R1-7B (31.0 → 41.0 → 46.9) and OVR-7B (31.0 → 42.9 → 49.9). The paper interprets this as evidence that multimodal SFT can overfit to familiar visual cues, whereas RL tends to improve semantic invariance (Chung et al., 2 Oct 2025).

Probe-level results further show that perception is not monolithic. The paper reports reliable improvements for cocircular, parallel, and collinear relations; improvements that mainly appear with textual SFT followed by multimodal RL for same_angle, val_angle, triangle, and quadrilateral; and persistent difficulty for same_length, val_length, and perpendicular. The proposed explanation is whether the cue is visually direct or spatially displaced: some relations are visually local, while others require linking offset symbols and distant structure (Chung et al., 2 Oct 2025).

The benchmark’s most consequential empirical claim is that integration remains the dominant residual failure mode. Once perception and text-only reasoning improve, the remaining errors are disproportionately cases where the model can recover relevant facts in isolation and can reason correctly when those facts are stated explicitly, yet still fails when it must coordinate the visual evidence with the reasoning process. This is the paper’s strongest argument that current multimodal geometry failures cannot be reduced to “bad vision” or “bad logic” alone (Chung et al., 2 Oct 2025).

6. Position within the broader math-evaluation ecosystem

MathLens belongs to a broader shift from coarse end-to-end scoring toward capability-specific analysis in mathematical AI. Within multimodal mathematics, MATHGLANCE isolates mathematical perception in MLLMs through four tasks—shape classification, object counting, relationship identification, and object grounding—and reports that current models are notably limited, particularly in fine-grained grounding tasks. It also introduces GeoPeP, a perception-oriented dataset of 200K structured geometry image-text pairs, and shows that stronger perceptual accuracy substantially improves mathematical reasoning (Sun et al., 26 Mar 2025). MathLens is narrower in domain but more explicit about the residual category of integration.

A related diagnostic trend appears in multilingual visual mathematics. The Kangaroo-based benchmark on visual mathematics reports that Gemini 2.0 Flash reaches 45.4% on image-based tasks and 75.9% on non-image questions, while some weaker models are nearly unchanged when images are removed, indicating underutilization of diagrammatic information (Sáez et al., 9 Jun 2025). That result complements MathLens: both argue that apparent multimodal competence often overstates actual diagram use, but MathLens supplies a formal mechanism for locating whether the bottleneck is perception, reasoning, or their coordination.

The same broad diagnosis appears in text-only mathematical reasoning. CogMath formalizes human reasoning into problem comprehension, problem solving, and solution summarization, develops 9 fine-grained evaluation dimensions, and reports that the mathematical capabilities of 7 mainstream LLMs are overestimated by 30\%-40\% under final-answer evaluation (Liu et al., 4 Jun 2025). MathLens can be read as the multimodal analogue of that move: it replaces one downstream score with a structured account of the subskills required by a geometry problem.

A different but equally important part of the ecosystem concerns evaluation fidelity itself. MARIO Eval reframes math answer evaluation as a typed equivalence problem rather than a string-matching problem, supports answer types such as Real, Set, Interval(s), Vector, Matrix, Expression, Function, Equation, and Inequality, and reports that its basic symbolic design reaches 97.23% equivalence accuracy on MATH and 96.74% on GK2023 (Zhang et al., 2024). This suggests a broader implication for MathLens-style benchmarks: once reasoning is decomposed into subskills, the scoring layer must also be mathematically typed and robust, or capability-specific conclusions can be distorted by evaluator brittleness.

Taken together, these neighboring efforts suggest that “MathLens” names not only a benchmark but a research direction. The shared premise is that mathematical AI should be analyzed through structured capacities—perception, reasoning, integration, comprehension, verification, or typed equivalence—rather than through aggregate accuracy alone. In that landscape, MathLens contributes a particularly strict result: even when perception improves and text-only reasoning improves, the coordination between them remains the weakest and most persistent bottleneck in multimodal geometry (Chung et al., 2 Oct 2025).