---
title: 'MathLens: Dissecting Multimodal Geometry'
url: https://www.emergentmind.com/topics/mathlens
type: topic
---

# MathLens: Dissecting Multimodal Geometry

MathLens is a benchmark designed to disentangle the subskills of multimodal reasoning while preserving the complexity of textbook-style geometry problems. It separates performance into three components—**Perception**, **Reasoning**, and **Integration**—and supports each test with visual diagrams, textual descriptions to evaluate reasoning in isolation, controlled questions that require both modalities, and probes for fine-grained perceptual skills, all derived from symbolic specifications of the problems to ensure consistency and robustness [2510.01719].

## 1. Diagnostic objective and research setting

The benchmark was introduced against a background in which evaluation of multimodal reasoning is dominated by aggregate accuracy. That single score is useful, but it does not reveal whether a gain comes from improved extraction of visual facts, improved symbolic inference once facts are available, or improved coordination between those two processes. MathLens therefore asks not only whether a model answers correctly, but why a model succeeds or fails. Its central research question is explicit: when multimodal large language models improve on visual mathematics after fine-tuning, what exactly are they learning more of—better visual extraction, better abstract reasoning, or better use of visual evidence inside reasoning [2510.01719].

This diagnostic orientation distinguishes MathLens from benchmarks that vary modalities without fully isolating them. The appendix contrasts MathLens with MathVerse by noting that MathVerse text descriptions often fail to fully encode the geometric relations, so “text-only reasoning” is not a clean reasoning test. MathLens instead derives all benchmark views from a shared symbolic state, which makes its modality separation controlled rather than heuristic. The resulting benchmark is not merely a new leaderboard task; it is a causal-style decomposition of multimodal geometry performance [2510.01719].

MathLens was also validated as a genuine multimodal reasoning benchmark rather than a purely synthetic probe set. Fine-tuned models consistently outperform their backbones on the main geometry-solving task, and MathLens correlates with established multimodal math benchmarks: Spearman $\rho = 0.83$ with MathVista, $\rho = 0.86$ with MathVerse, and $\rho = 0.67$ with MathVision. The appendix further introduces MathLens-E, a version using original source diagrams instead of fresh renders, and reports that MathLens-E correlates more strongly with MathVision than MathLens does, suggesting that public benchmark performance may partly reflect visual familiarity with known diagram styles [2510.01719].

A common misconception is that improved aggregate accuracy can be read straightforwardly as improved multimodal reasoning. MathLens rejects that interpretation. Its design assumes that apparent progress may conceal uneven changes across perception, reasoning, and their coupling, and the empirical analysis confirms that these components often move differently under different training regimes [2510.01719].

## 2. Formal decomposition of multimodal geometry

MathLens formalizes each problem instance $k$ by a latent semantic state
$$
S_k=\{s_{k,1},\ldots,s_{k,m}\},
$$
where each $s_{k,i}$ is an atomic fact such as $\angle ABC=50^\circ$. The gold answer is then defined by
$$
A_k=f(\varphi_k,S_k),
$$
where $\varphi_k$ is the question operator. Visual context and textual context are both generated conditionally on the same $S_k$, so the image, the text description, the multimodal question, and the perception probes are aligned views of one semantic object rather than loosely matched annotations [2510.01719].

Within that structure, **Perception** is the ability to extract information from raw inputs, especially the diagram. **Reasoning** is the ability to operate correctly once all required facts are already available. **Integration** is the ability to select the relevant perceptual evidence and apply it correctly during reasoning. The text-only task operationalizes reasoning by replacing the diagram with a complete textual rendering of the semantic state. Perception is operationalized by atom-level probes. Integration is treated as the residual failure mode: if a model answers the relevant perception probes correctly and also solves the text-only version correctly, but still fails on the original multimodal problem, the failure is attributed to integration [2510.01719].

The text descriptions are generated clausewise from the symbolic state:
$$
C_k^{\mathrm{txt}}=\text{concat}_i f_d(s_{k,i}),
$$
and the perception probes are templated from individual atoms:
$$
Q_k^{\mathrm{perc}}=\{f_p(s_{k,i})\}_i.
$$
MathLens also constructs context-forcing questions by removing facts from the question that overlap with the context. This prevents the question text from restating the needed geometric information and ensures that the benchmark actually requires use of the supplied context [2510.01719].

This formalization matters because it makes the three capacities testable on the same underlying problem. In ordinary multimodal evaluation, failures are often observationally ambiguous: a wrong answer could reflect bad diagram reading, bad inference, or bad coordination. In MathLens, those possibilities are separated by construction [2510.01719].

## 3. Benchmark construction and perturbation design

The main MathLens dataset is built from **FormalGeo-7K**, which provides symbolic annotations for geometry problems. Symbolic clauses such as $Perpendicular(AB,BC)$ are converted into coordinate constraints; the paper gives
$$
(x_a-x_b)(x_c-x_b)+(y_a-y_b)(y_c-y_b)=0
$$
as an example. The complete constraint set is solved with SLSQP via `scipy.optimize.minimize`, using random initializations and up to ten attempts per sample. If no feasible solution is found, the sample is discarded. The solution is then rendered with a matplotlib backend that draws points, segments, arcs, and labels [2510.01719].

The rendered outputs are manually filtered to remove overlapping vertices, occluded labels, extreme aspect ratios, and degenerate layouts. The benchmark also preserves olympiad-style rendering conventions: diagrams may be intentionally non-canonical and “not drawn to scale,” so an obtuse angle may be drawn acute if that prevents the problem from collapsing into visual estimation. This keeps the benchmark close to textbook and competition geometry rather than to purely perceptual shape recognition [2510.01719].

For the main geometry benchmark, the paper reports **926 geometry problems**, **8 visual modifications per problem**, and an average of **$\sim 7.03$ visual probes per problem**. Each problem has the original diagram from the source dataset, a newly rendered base diagram from the symbolic representation, and six semantic-preserving variants: `add_shapes`, `add_lines`, `flip`, `rotate`, `merge`, and `rename`. The `merge` transformation concatenates the diagram with another random diagram and revises labels; `rename` relabels points such as $A,B,C \to X,Y,Z$ [2510.01719].

The perturbation regime is central to the benchmark’s robustness claims. These are not ordinary pixel augmentations. They are semantic-space perturbations generated from the symbolic specification, and the invariance condition is that the answer remains unchanged under admissible perturbations $\tau \in \mathcal{T}_{\mathrm{AP}}$:
$$
S'_k = \tau(S_k), \qquad f(\varphi_k, S_k) = f(\varphi_k, S'_k).
$$
This gives MathLens a stronger notion of robustness than blur or noise ablations, because the model is being tested on alternate yet valid realizations of the same geometry [2510.01719].

The paper also introduces **MathLens-General**, a complementary broader-domain set with **107 problems** and an average of **$\sim 7.96$ visual probes per problem**. It is curated from BLINK, V*, SpatialEval-Real, MMMU-Pro, EMMA, and MathVista, excluding geometry items. The authors explicitly present it as a broader but less formally controlled complement to the main geometry benchmark, because those problems do not come with a formal semantic state like FormalGeo-7K [2510.01719].

## 4. Evaluation protocol and error taxonomy

MathLens evaluates the same underlying problem through aligned tasks rather than through one monolithic score. The multimodal task tests the full pipeline; the text-only task tests reasoning once perception is trivialized; the perception probes test recovery of specific visual facts; and the perturbation suite tests invariance under semantic diagram variation [2510.01719].

| Capacity | Input view | Operational meaning |
|---|---|---|
| Perception | Diagram + atom-specific probe | Recover facts from raw inputs |
| Reasoning | Complete textual description + question | Solve when all required facts are explicit |
| Integration | Residual multimodal failure | Select relevant perceptual evidence and apply it during reasoning |

The benchmark’s most distinctive analytical component is its automatic error taxonomy. Outcomes are partitioned into six groups: **Perception + Reasoning**, **Perception**, **Reasoning**, **Integration**, **Trivial**, and **Rest**. “Perception + Reasoning” denotes cases that fail both perception probes and text-only reasoning. “Perception” denotes cases that fail perception probes but succeed on text-only reasoning. “Reasoning” denotes cases that succeed on perception probes but fail text-only reasoning. “Integration” denotes cases that succeed on both perception probes and text-only reasoning, but fail on the multimodal task. “Trivial” covers correct cases solvable from text alone, and “Rest” covers the remaining correct cases [2510.01719].

Robustness is quantified by **Consistency Rate (CR)**:
$$
CR = \mathbb{E}_{Q'_k \sim Q'} \Big[\mathbb{E}_{\substack{\bar{C}_{k,i},\,\bar{C}_{k,j} \sim \bar{C}_k \\ i \neq j}} \mathbf{1}\!\left[M(Q'_k,\bar{C}_{k,i}) = M(Q'_k,\bar{C}_{k,j})\right]\Big].
$$
Here $\bar{C}_k$ is the set of diagram variants for question $Q'_k$, and $M$ is the model. CR therefore measures expected agreement of predictions across semantically equivalent diagram variants, not raw accuracy under arbitrary perturbation [2510.01719].

This evaluation design addresses a second common misconception: that multimodal reasoning is adequately characterized by separate “vision” and “reasoning” scores. MathLens shows that even when those two isolated capabilities look strong, the multimodal pipeline can still fail because cross-modal selection and transfer remain unstable [2510.01719].

## 5. Empirical findings on training effects and robustness

MathLens reports that different training approaches have uneven effects. Reinforcement learning chiefly strengthens perception, especially when supported by textual supervision. Textual supervised fine-tuning indirectly improves perception through reflective reasoning. Reasoning improves only in tandem with perception. Integration remains the weakest capacity, with residual errors concentrated there once other skills advance. Robustness also diverges: RL improves consistency under diagram variation, whereas multimodal SFT reduces it through overfitting [2510.01719].

Concrete model trajectories make the asymmetry explicit. For **Revisual-R1-7B**, text-only accuracy moves from **62.3** to **63.8** after multimodal RL, while base-diagram accuracy moves from **39.2** to **45.8**. For **OVR-7B**, the corresponding shift is **66.2 → 70.4** on text and **38.7 → 43.7** on the base diagram. These patterns support the authors’ interpretation that multimodal RL improves perception and grounding more strongly than abstract reasoning in isolation [2510.01719].

The robustness analysis shows an equally strong contrast. For **Vision-R1-7B**, consistency is **31.0** at the backbone stage, **21.1** after multimodal SFT, and **39.4** after multimodal RL. For **R1-OneVision-7B**, the sequence is **31.0**, **24.0**, and **28.5**. By contrast, text-SFT followed by multimodal RL yields steadily increasing consistency for **Revisual-R1-7B** (**31.0 → 41.0 → 46.9**) and **OVR-7B** (**31.0 → 42.9 → 49.9**). The paper interprets this as evidence that multimodal SFT can overfit to familiar visual cues, whereas RL tends to improve semantic invariance [2510.01719].

Probe-level results further show that perception is not monolithic. The paper reports reliable improvements for cocircular, parallel, and collinear relations; improvements that mainly appear with textual SFT followed by multimodal RL for same\_angle, val\_angle, triangle, and quadrilateral; and persistent difficulty for same\_length, val\_length, and perpendicular. The proposed explanation is whether the cue is visually direct or spatially displaced: some relations are visually local, while others require linking offset symbols and distant structure [2510.01719].

The benchmark’s most consequential empirical claim is that **integration remains the dominant residual failure mode**. Once perception and text-only reasoning improve, the remaining errors are disproportionately cases where the model can recover relevant facts in isolation and can reason correctly when those facts are stated explicitly, yet still fails when it must coordinate the visual evidence with the reasoning process. This is the paper’s strongest argument that current multimodal geometry failures cannot be reduced to “bad vision” or “bad logic” alone [2510.01719].

## 6. Position within the broader math-evaluation ecosystem

MathLens belongs to a broader shift from coarse end-to-end scoring toward capability-specific analysis in mathematical AI. Within multimodal mathematics, **MATHGLANCE** isolates mathematical perception in MLLMs through four tasks—shape classification, object counting, relationship identification, and object grounding—and reports that current models are notably limited, particularly in fine-grained grounding tasks. It also introduces **GeoPeP**, a perception-oriented dataset of **200K structured geometry image-text pairs**, and shows that stronger perceptual accuracy substantially improves mathematical reasoning [2503.20745]. MathLens is narrower in domain but more explicit about the residual category of integration.

A related diagnostic trend appears in multilingual visual mathematics. The Kangaroo-based benchmark on visual mathematics reports that **Gemini 2.0 Flash** reaches **45.4%** on image-based tasks and **75.9%** on non-image questions, while some weaker models are nearly unchanged when images are removed, indicating underutilization of diagrammatic information [2506.07418]. That result complements MathLens: both argue that apparent multimodal competence often overstates actual diagram use, but MathLens supplies a formal mechanism for locating whether the bottleneck is perception, reasoning, or their coordination.

The same broad diagnosis appears in text-only mathematical reasoning. **CogMath** formalizes human reasoning into **problem comprehension**, **problem solving**, and **solution summarization**, develops **9 fine-grained evaluation dimensions**, and reports that the mathematical capabilities of **7 mainstream LLMs** are overestimated by **30\%-40\%** under final-answer evaluation [2506.04481]. MathLens can be read as the multimodal analogue of that move: it replaces one downstream score with a structured account of the subskills required by a geometry problem.

A different but equally important part of the ecosystem concerns evaluation fidelity itself. **MARIO Eval** reframes math answer evaluation as a typed equivalence problem rather than a string-matching problem, supports answer types such as Real, Set, Interval(s), Vector, Matrix, Expression, Function, Equation, and Inequality, and reports that its basic symbolic design reaches **97.23%** equivalence accuracy on MATH and **96.74%** on GK2023 [2404.13925]. This suggests a broader implication for MathLens-style benchmarks: once reasoning is decomposed into subskills, the scoring layer must also be mathematically typed and robust, or capability-specific conclusions can be distorted by evaluator brittleness.

Taken together, these neighboring efforts suggest that “MathLens” names not only a benchmark but a research direction. The shared premise is that mathematical AI should be analyzed through structured capacities—perception, reasoning, integration, comprehension, verification, or typed equivalence—rather than through aggregate accuracy alone. In that landscape, MathLens contributes a particularly strict result: even when perception improves and text-only reasoning improves, the coordination between them remains the weakest and most persistent bottleneck in multimodal geometry [2510.01719].

Source: https://www.emergentmind.com/topics/mathlens