SpatialViz-Bench: Spatial Visualization Benchmark

• SpatialViz-Bench is a benchmark that isolates spatial visualization, emphasizing internal mental manipulation over simple pattern recognition.
• It features 1,180 automatically generated multiple-choice tasks across 12 tests grouped into four sub-abilities with controlled difficulty scaling.
• Empirical evaluations reveal steep performance drops in 3D tasks and unexpected model behaviors, underscoring current MLLMs’ limitations.

SpatialViz-Bench is a cognitive-science-inspired multimodal benchmark for evaluating spatial visualization in multimodal LLMs (MLLMs), where spatial visualization is treated as the ability to mentally manipulate visual images rather than merely recognize visible relations or retrieve memorized patterns. Introduced in "SpatialViz-Bench: Automatically Generated Spatial Visualization Reasoning Tasks for MLLMs" (Wang et al., 10 Jul 2025), it comprises 1,180 automatically generated multiple-choice problems organized into 12 tasks across 4 sub-abilities: Mental Rotation, Mental Folding, Visual Penetration, and Mental Animation. Its central methodological claim is that spatial visualization had been insufficiently evaluated because it was usually embedded within broader mathematical, logical, or IQ-style assessments, many of them web-sourced and therefore susceptible to training-data overlap.

1. Conceptual scope and motivation

SpatialViz-Bench was proposed to isolate spatial visualization as a first-class ability. The benchmark distinguishes this ability from broader spatial or visual reasoning by emphasizing mental transformation and internal visualization over simple object recognition, coarse relation labeling, or formulaic problem solving. Existing evaluations were characterized as problematic in five ways: spatial visualization was often buried inside broader reasoning benchmarks; many tasks were web-sourced; subskills were represented by too few items; formats were heterogeneous; and benchmarks lacked a cognitive-theoretic structure (Wang et al., 10 Jul 2025).

The benchmark adopts a process view of visual-spatial reasoning with two phases. The first is spatial perception, which concerns visible cues and explicit relations. The second is a hidden-inference phase, in which the solver alternates between spatial visualization and spatial memorization. The paper names three reasoning components explicitly: P1-1 Spatial Perception, P2-1 Spatial Visualization, and P2-2 Spatial Memorization. It also argues that visual-spatial reasoning is not purely visual, because language-guided logic remains involved in interpreting and solving the tasks.

A common misconception is to treat SpatialViz-Bench as a generic spatial VQA set. Its design is narrower and more diagnostic. The benchmark does not primarily test whether a model can say where an object is, identify depth, or follow ordinary scene descriptions; it tests whether a model can carry out latent operations such as rotation, folding, penetration, and animation in a way that resembles mental image manipulation.

2. Taxonomy of sub-abilities and tasks

The benchmark groups its 12 tasks into four sub-abilities, with three tasks per group (Wang et al., 10 Jul 2025).

Sub-ability	Tasks	Core operation
Mental Rotation	2D Rotation; 3D Rotation; Three-View Projection	Rotate or project structured objects
Mental Folding	Paper Folding; Cube Unfolding; Cube Reconstruction	Fold or unfold 2D and 3D forms
Visual Penetration	Cross-Section; Cube Counting; Cube Assembly	Infer hidden internal structure
Mental Animation	Arrow Moving; Block Moving; Mechanical System	Simulate motion and transformation over time

Mental Rotation covers both planar and volumetric transformation. 2D Rotation uses colored grid patterns with a red corner marker and requires reasoning over rotations of $90^\circ$, $180^\circ$, or $270^\circ$. 3D Rotation uses connected cube stacks rotated around the $x/y/z$ axes by the same angles. Three-View Projection asks for an unobserved view given other views; the benchmark includes both a cube-stack version and a DeepCAD engineering-model version.

Mental Folding targets fold/unfold reasoning. Paper Folding simulates sequential horizontal, vertical, or diagonal folds, followed by hole punching and unfolding. Cube Unfolding asks the solver to select the correct 2D net for a cube with six uniquely colored faces; the paper notes that there are 11 possible cube nets. Cube Reconstruction includes two variants: selecting the correct 3D vertex view from a cube net, and identifying the opposite face of a given face.

Visual Penetration addresses hidden structure. Cross-Section uses composite solids formed from basic geometric primitives and asks for cross-sections under axis-parallel or oblique cuts. Cube Counting requires inferring the number of cubes in a stack from orthogonal projections. Cube Assembly presents a pyramid-like cube stack split into connected parts and asks for the complementary piece.

Mental Animation evaluates dynamic internal simulation. Arrow Moving operates on a grid under egocentric movement rules, where forward, backward, left, and right are defined relative to the arrow’s current orientation. Block Moving introduces colored cubes, six-direction movement, gravity, and swap behavior under occupancy conflicts. Mechanical System uses open-source mechanical simulations to test understanding of motion propagation through linked components.

3. Automatic generation pipeline and difficulty control

A major contribution of SpatialViz-Bench is its emphasis on automatic generation rather than manual collection from web sources. The paper states that all tasks except the mechanical system task are generated automatically. The automated pipeline combines Python and FreeCAD, while Paper Folding and Arrow Moving use Python-only simulation logic. Mechanical System is the only non-fully-automated task; it is manually constructed from open-source simulation materials and contains 80 validated samples (Wang et al., 10 Jul 2025).

The general generation logic is consistent across tasks. For each item, the system creates a valid reference instance, generates one or more positive options that preserve the correct structure, produces plausible distractors that violate key spatial constraints, shuffles the answer positions, and stores the question, image or images, the correct option, and explanations for incorrect choices. This design supports controlled randomness, systematic distractor construction, and explicit difficulty scaling.

Difficulty is structured at the task level. Each task has 2 or 3 difficulty levels, with 40–50 cases per level, and the benchmark is designed for balanced coverage and controlled difficulty progression. In 2D Rotation, difficulty rises with non-centrally symmetric patterns and greater internal asymmetry. In 3D Rotation, it rises with stack dimensions and geometric complexity. In Paper Folding, it rises with additional folds, larger grids, and denser hole patterns. In Cross-Section, harder versions include three-solid composites, disconnected cross-sections, and oblique cuts at $45^\circ$ and $135^\circ$.

Several tasks are mathematically explicit. Cube Counting computes answer ranges directly from projections. For two-view counting, the paper gives the upper bound

$$\text{max\_2view} = \text{sum\_front\_col} \cdot \text{sum\_top\_col},$$

and for three views it computes an array by

$$\text{ans}[row][col] \gets \min(\text{sum\_front\_col}[col], \text{sum\_left\_col}[row]),$$

followed by

$\text{max\_3view} \gets \text{sum(ans)}.$

This explicit construction is characteristic of the benchmark’s attempt to make distractors systematic rather than ad hoc.

The generation process also reflects the benchmark’s cognitive orientation. Distractors are not merely visually similar alternatives; they are usually produced by operations such as mirroring, deleting internal lines, swapping visible or opposite-face colors, altering hole positions, removing cubes from an otherwise correct assembly, or applying incorrect transformations to an unobserved view. The result is a benchmark that attempts to separate genuine spatial transformation ability from superficial pattern matching.

4. Evaluation protocol

SpatialViz-Bench standardizes all tasks in a multiple-choice format with one correct answer, described in the paper as MCA (multiple-choice answers) (Wang et al., 10 Jul 2025). Option images and reference images are combined into a unified visual input, which reduces confounds from heterogeneous task presentation.

The evaluation covers 33 MLLMs, including 9 closed-source and 24 open-source systems. The closed-source group includes GPT-4o, o1, Gemini-1.5-pro, Gemini-2.5-flash, Gemini-2.5-pro, Claude-3.5-sonnet, Claude-3.7-sonnet, Qwen-VL-max, and Doubao-1.5-vision-pro. The open-source group includes families such as Qwen2.5-VL, QvQ, Qwen-Omni, InternVL-2.5, InternVL-3, Deepseek-VL2, SAIL-VL, Kimi-VL-A3B, Llama-4, and LLaVA-OneVision. Reported model sizes span 3B to 108B.

All experiments use a zero-shot prompt that explicitly requests a reasoning process followed by a final answer inside `and<answer></answer>tags. The primary metric is **accuracy**. Answer extraction is rule-based: it first looks for content inside<answer></answer>`, then falls back to patterns such as “Answer:” or “Final answer.” A prediction is scored as correct only if the extracted output contains exactly one uppercase option letter matching the ground truth.

This protocol is significant because it suppresses several common evaluation ambiguities. By converting all tasks to the same output schema and using rule-based extraction, the benchmark minimizes format variance as an alternative explanation for failure. At the same time, the use of explicit reasoning prompts makes it possible to inspect whether a model reaches correct answers by plausible visuospatial reasoning or by other, sometimes inappropriate, heuristics.

5. Empirical findings and diagnostic patterns

The benchmark reports that all evaluated models remain well below human level, and that SpatialViz-Bench has strong discriminative power across systems (Wang et al., 10 Jul 2025). The best overall result is Gemini-2.5-pro at 44.66%, followed by o1 at 41.36%. Among open-source systems, the best reported overall scores are LLaMA-4-Scout-17B-16E-Instruct at 34.24% and Qwen2.5-VL-72B-Instruct at 33.31%. The random baseline is described as being around the mid-20s overall, so the best systems are only roughly twice random.

One of the benchmark’s main findings is a set of counter-intuitive behaviors. The paper argues that models show difficulty perception that misaligns with human intuition. Harder levels do not always reduce accuracy, and some models perform better on more complex inputs than on simpler ones. A striking example comes from Three-View Projection: o1 rises from 40.0% to 58.0%, and Gemini-2.5-pro rises from 28.0% to 66.0% between two reported levels. This suggests that model competence is not organized along the same internal difficulty gradient that humans would ordinarily assume.

A second major result is the presence of dramatic 2D-to-3D performance cliffs. Models that are moderately competent on 2D Rotation often deteriorate sharply on 3D Rotation, cube tasks, and multi-view projection. The benchmark interprets this as evidence that current MLLMs still lack robust 3D spatial generalization.

A third finding is that models frequently default to formula derivation despite spatial tasks requiring visualization alone. This is most visible in the Mechanical System task, where models often attempt symbolic or formulaic reasoning rather than directly simulating the mechanism. The paper treats this as a misalignment between the intended cognitive demand of the task and the reasoning style induced by pretraining.

Task-group analysis further sharpens the picture. Mental Rotation shows a familiar split: 2D versions are easier than 3D ones. Mental Folding is difficult throughout, with Cube Unfolding and Cube Reconstruction especially challenging. Visual Penetration reveals instability under increased structural complexity; for instance, Gemini-2.5-pro reaches 80% on Cube Counting Level 0 but drops to 52.5% at Level 1 and 32.5% at Level 2. Mental Animation yields the widest gaps between simple and complex dynamics: Gemini-2.5-pro reaches 95% on Arrow Moving in the aggregate table, but performance is much weaker on Block Moving and on the more structurally demanding dynamics of Mechanical System.

The qualitative error analysis attributes most failures to perception and visualization, rather than to pure logical reasoning. Reported failure modes include color-recognition errors, inability to identify asymmetric patterns, difficulty with relative positions, failure to infer cube-net structure, confusion in stacking and 3D occupancy, and incorrect handling of spatial relations in 3D scenes. The paper also notes that models can sometimes arrive at the correct option with flawed reasoning, indicating that answer accuracy alone may conceal defective internal processes.

6. Position in the spatial-benchmark landscape and reported limitations

SpatialViz-Bench occupies a specific niche within the expanding literature on spatial evaluation. Later and broader frameworks organize spatial intelligence in ways that are more hierarchical or task-diverse. SIBench, for example, divides spatial intelligence into basic perception, spatial understanding, and spatial planning (Yu et al., 23 Sep 2025). SpatialBench expands the hierarchy to five levels from Observation to Planning (Xu et al., 26 Nov 2025). Spatial-DISE uses a cognitively grounded Intrinsic/Extrinsic × Static/Dynamic taxonomy (Huang et al., 15 Oct 2025). By contrast, SpatialViz-Bench concentrates on mental transformation and internal visualization, rather than on planning, embodied action selection, or broad scene-level spatial competence. This suggests that it is most useful as a diagnostic instrument for a narrow but fundamental cognitive component.

The benchmark also differs from datasets centered on viewpoint transformation and localization. ViewSpatial-Bench evaluates multi-perspective spatial localization across camera-perspective and human-perspective tasks (Li et al., 27 May 2025), whereas SpatialViz-Bench is primarily about transforming latent visual structure rather than switching reference frames. The distinction is important: a model may be competent at localization or egocentric/allocentric conversion while still performing poorly on rotation, folding, penetration, or animation.

The paper identifies several limitations of SpatialViz-Bench itself (Wang et al., 10 Jul 2025). Some generated cube-stacking scenes still required manual verification to avoid ambiguity. The dataset size is limited to 1,180 examples, a deliberate trade-off in favor of reliability. Mechanical System is manually curated rather than fully automated. More generally, the automatic generation pipeline is described as strong but not perfect, and some tasks still require human checking for ambiguity. These caveats do not negate the benchmark’s contribution; rather, they clarify that automatic generation in this domain remains partially dependent on human validation when spatial ambiguity is difficult to eliminate algorithmically.

SpatialViz-Bench therefore stands as an ability-centric benchmark whose main importance lies in showing that state-of-the-art MLLMs remain deficient in true spatial visualization. Its contribution is not only a scorecard but a particular decomposition of visuospatial reasoning into perception, visualization, and memorization, along with empirical evidence that current models often substitute symbolic heuristics for genuine mental image manipulation (Wang et al., 10 Jul 2025).