SpatialTunnel: Benchmark for Depth Reasoning

• SpatialTunnel is a synthetic benchmark suite designed to decouple vertical image cues from true 3D depth in controlled tunnel environments.
• It employs Blender-rendered scenes with two objects and a fixed tunnel geometry to remove natural perspective correlations and test VLM depth reasoning.
• Empirical findings reveal that models often rely on perspective shortcuts, highlighting the need for robust spatial representations in vision-language systems.

SpatialTunnel is a synthetic benchmark suite introduced to test whether vision-LLMs (VLMs) represent relative depth as a distinct spatial axis or instead rely on perspective-consistent shortcuts inherited from natural images, especially the heuristic that an object appearing higher in the image is farther from the camera (Min et al., 28 May 2026). Rendered in Blender and organized around binary depth comparison inside a controlled tunnel geometry, it isolates depth from common 2D correlates by varying image-plane position independently of true 3D distance. Within the broader study that introduced it, SpatialTunnel functions as a behavioral stress test for vertical-distance entanglement and as a benchmark-level complement to representation probing based on contrastive pairs and embedding-space geometry (Min et al., 28 May 2026).

1. Scientific rationale

SpatialTunnel was motivated by a specific critique of existing spatial benchmarks: many real-image evaluation sets are dominated by examples in which perspective heuristics are correct, so strong aggregate accuracy can be obtained without genuinely disentangled depth reasoning (Min et al., 28 May 2026). On depth-related questions from EmbSpatial-Bench, $80.9\%$ of examples are “consistent,” meaning the farther object appears higher in the image, while only $10.7\%$ are “counter” examples; in CV-Bench-3D, the corresponding numbers are $60.5\%$ consistent and $10.8\%$ counter (Min et al., 28 May 2026). The benchmark was therefore designed to remove the statistical regularities of ordinary photographs rather than merely rebalance labels.

The underlying claim is not that natural-image benchmarks are useless, but that they can contain spurious correlations and perspective-consistent shortcuts. SpatialTunnel addresses this by constructing scenes in which vertical image position can vary independently of true depth. In the authors’ formulation, the tunnel geometry “decouples vertical image position from depth,” so that an object near the top of the image and one near the bottom can be equidistant from the camera (Min et al., 28 May 2026). This makes the benchmark an intervention on the dataset geometry itself: if a model’s prediction changes when only the 2D arrangement changes while depth ordering is held fixed, the model is exhibiting shortcut sensitivity rather than robust depth reasoning.

A common misconception is that high benchmark accuracy alone implies structured 3D understanding. SpatialTunnel was created precisely to challenge that inference. The broader paper shows that models with similar top-line benchmark scores can differ markedly in internal spatial organization and in their robustness once perspective-consistent regularities are removed (Min et al., 28 May 2026).

2. Benchmark construction and scene geometry

SpatialTunnel is rendered in Blender as a set of RGB images containing exactly two objects at different depths inside a single-point-perspective corridor with symmetric walls, ceiling, and floor (Min et al., 28 May 2026). The tunnel has a square cross-section of $2\,\text{m} \times 2\,\text{m}$, and the two objects are parameterized by depth $z$ and angular position $\theta$ on the tunnel cross-section (Min et al., 28 May 2026). The farther object is always denoted $\text{obj}_1$, and the nearer one $\text{obj}_2$ (Min et al., 28 May 2026).

The core intervention is that depth ordering is fixed while image-plane placement changes (Min et al., 28 May 2026). Each object is swept independently over 16 discrete angular positions, yielding a $16 \times 16$ Cartesian grid over $10.7\%$0 (Min et al., 28 May 2026). Because the tunnel is symmetric about the optical axis, this sweep changes whether an object appears higher or lower in the image without changing its depth. This is the mechanism by which SpatialTunnel breaks the natural-image correlation between vertical position and distance.

The objects are deliberately simple. Each is either a sphere or a cube and is assigned one of seven colors—red, green, blue, yellow, cyan, magenta, or black—using a Principled BSDF shader; roughness is sampled uniformly from $10.7\%$1 (Min et al., 28 May 2026). The two objects are constrained to have distinct $10.7\%$2 combinations (Min et al., 28 May 2026). In the main “phase-variation” setting, base sizes are $10.7\%$3 for the farther object and $10.7\%$4 for the nearer object, each multiplied by an independent random scale factor in $10.7\%$5 (Min et al., 28 May 2026). Lighting is randomized via a Blender Nishita sky texture, with sun rotation sampled uniformly from $10.7\%$6, while background intensity is fixed at $10.7\%$7 (Min et al., 28 May 2026).

The resulting benchmark cardinalities are explicit.

Component	Value
Tunnel cross-section	$10.7\%$8
Angular positions per object	16
Angular configurations	$10.7\%$9
Scene instances per configuration	12
Images	$60.5\%$0
Question templates per image	4
Question-image pairs	$60.5\%$1

For each joint angular configuration, the benchmark renders 12 scene instances with independently randomized shape, color, size, and lighting, producing $60.5\%$2 images (Min et al., 28 May 2026). Each image is paired with four yes/no depth-comparison templates:

1. “Is the $60.5\%$3 closer to the camera than the $60.5\%$4?” — ground truth: No
2. “Is the $60.5\%$5 closer to the camera than the $60.5\%$6?” — ground truth: Yes
3. “Is the $60.5\%$7 farther from the camera than the $60.5\%$8?” — ground truth: No
4. “Is the $60.5\%$9 farther from the camera than the $10.8\%$0?” — ground truth: Yes (Min et al., 28 May 2026)

This yields $10.8\%$1 question-image pairs (Min et al., 28 May 2026).

3. Evaluation protocol and metrics

SpatialTunnel is evaluated primarily as a zero-shot binary VQA-style benchmark (Min et al., 28 May 2026). For open-source models, the score is derived from the logits of the first generated token for “Yes” and “No”:

$10.8\%$2

Single-query correctness is then defined as

$10.8\%$3

The benchmark reports four metrics: mean correctness score $10.8\%$4, consistent accuracy $10.8\%$5, counter accuracy $10.8\%$6, and the accuracy gap

$10.8\%$7

Here, “consistent” means that the farther object appears higher in the image, while “counter” means the opposite (Min et al., 28 May 2026). A model with no vertical-distance bias should have $10.8\%$8 (Min et al., 28 May 2026).

This scoring protocol matters because SpatialTunnel is intended not merely as a dataset of hard examples, but as a tool for measuring behavioral asymmetry under controlled geometric interventions. The benchmark therefore distinguishes average performance from directional bias. That distinction is important in interpreting results: a narrow consistent-counter gap can indicate robustness, but it can also arise from near-random performance, a caveat explicitly discussed in the paper for base NVILA-Lite-2B (Min et al., 28 May 2026).

The appendix reports a second evaluation setting for proprietary APIs—GPT-5.2, GPT-5.2 with reasoning enabled, and Gemini-2.5-Pro—using final yes/no outputs and exact-match accuracy, because logits were unavailable (Min et al., 28 May 2026). Those scores are explicitly stated to be not directly comparable to the logit-based $10.8\%$9 values (Min et al., 28 May 2026).

4. Empirical findings

The principal empirical result is that all open-source model families show a positive consistent-counter gap on SpatialTunnel, even though the benchmark was constructed so that vertical position does not determine depth (Min et al., 28 May 2026). This is the benchmark’s strongest evidence that vertical-distance entanglement is not merely an artifact of skewed natural-image evaluation sets.

Model	$2\,\text{m} \times 2\,\text{m}$0	$2\,\text{m} \times 2\,\text{m}$1	$2\,\text{m} \times 2\,\text{m}$2	$2\,\text{m} \times 2\,\text{m}$3
Molmo-7B base	0.528	0.565	0.487	+0.078
NVILA-Lite-2B base	0.488	0.504	0.471	+0.033
Qwen2.5-VL-3B base	0.570	0.776	0.360	+0.416
RoboRefer-2B-SFT	0.793	—	—	+0.046
Qwen3-VL-235B	0.908	—	—	+0.068

The base Qwen2.5-VL-3B result is especially severe: $2\,\text{m} \times 2\,\text{m}$4, $2\,\text{m} \times 2\,\text{m}$5, $2\,\text{m} \times 2\,\text{m}$6, and $2\,\text{m} \times 2\,\text{m}$7 (Min et al., 28 May 2026). By contrast, base NVILA-Lite-2B has a smaller gap, but the paper emphasizes that its overall accuracy is below $2\,\text{m} \times 2\,\text{m}$8, so the narrow gap does not indicate robustness; it reflects near-random performance (Min et al., 28 May 2026). This is one of SpatialTunnel’s most useful interpretive lessons: low bias metrics must be read together with overall correctness.

The benchmark also exposes a scaling effect already noted elsewhere in the paper: mean accuracy can improve without eliminating shortcut reliance, and in some cases the gap widens with additional fine-tuning data (Min et al., 28 May 2026). Molmo’s gap grows from $2\,\text{m} \times 2\,\text{m}$9 at 80k to $z$0 at 400k and $z$1 at 800k before dropping to $z$2 at 2M, while mean accuracy eventually rises to $z$3 (Min et al., 28 May 2026). NVILA reaches $z$4 at 2M but still has $z$5 (Min et al., 28 May 2026). Qwen2.5’s 2M model remains poor on SpatialTunnel, with $z$6 and $z$7 (Min et al., 28 May 2026).

Stronger models are those that combine high mean performance with smaller gaps. RoboRefer-2B-SFT obtains $z$8 with $z$9, the smallest gap among the above-chance open-source models discussed in the main text, while Qwen3-VL-235B reaches $\theta$0 with $\theta$1 (Min et al., 28 May 2026). The paper interprets this pattern as evidence that richer training or larger-scale pretraining can alleviate the bias, though not necessarily eliminate it (Min et al., 28 May 2026).

5. Representation-level connection

SpatialTunnel was introduced within a broader representation-analysis framework based on contrastive probing (Min et al., 28 May 2026). For a fixed image, the paper constructs minimal question pairs differing only by object-order reversal and extracts a final-token hidden state $\theta$2 at a chosen layer $\theta$3 (Min et al., 28 May 2026). For each pair $\theta$4, the delta vector

$\theta$5

is used to analyze how spatial relations are organized in embedding space (Min et al., 28 May 2026).

From these deltas, the paper defines axis coherence for horizontal, vertical, and distance relations, and then the VD-Entanglement Index (VD-EI) to measure the coupling between the vertical and distance axes (Min et al., 28 May 2026). The central claim is that SpatialTunnel is the behavioral complement to this internal analysis: models whose distance axis is weakly organized or entangled with the vertical axis are precisely those that remain brittle once perspective-consistent shortcuts are removed (Min et al., 28 May 2026).

The paper reports that $\theta$6 measured on SpatialTunnel correlates with counter-example accuracy on EmbSpatial-Bench and CV-Bench-3D with Spearman $\theta$7 and $\theta$8, respectively, both $\theta$9 (Min et al., 28 May 2026). This is one of the benchmark’s strongest validation results, because it links synthetic-benchmark behavior to robustness on real-image tasks rather than treating SpatialTunnel as an isolated toy problem.

The coherence values also support the same interpretation. Across model families, $\text{obj}_1$0 is consistently the weakest axis; for example, Molmo base has $\text{obj}_1$1, $\text{obj}_1$2, $\text{obj}_1$3, and $\text{obj}_1$4, while RoboRefer has $\text{obj}_1$5 and $\text{obj}_1$6 (Min et al., 28 May 2026). Qwen2.5 base has $\text{obj}_1$7 and $\text{obj}_1$8, and its 2M variant has $\text{obj}_1$9 and $\text{obj}_2$0 (Min et al., 28 May 2026). The paper’s PCA visualizations further show that stronger models such as RoboRefer and Qwen3 exhibit more clearly separated horizontal, vertical, and distance clusters, whereas weaker models show overlap between vertical and distance clusters (Min et al., 28 May 2026).

A plausible implication is that SpatialTunnel is best understood not as a standalone leaderboard, but as an assay for whether the distance axis is represented distinctly enough to support counter-heuristic reasoning. The paper’s own conclusion is close to this: models with well-separated spatial axes exhibit greater robustness, suggesting that well-structured spatial representations lead to more reliable spatial reasoning across diverse benchmarks (Min et al., 28 May 2026).

6. Extensions, limitations, and broader terminological context

The appendix extends SpatialTunnel to probe another shortcut cue: apparent object size (Min et al., 28 May 2026). In a size-controlled variant, object depths remain fixed while sizes are anti-correlated under $\text{obj}_2$1, sweeping $\text{obj}_2$2 over 11 values and setting $\text{obj}_2$3 (Min et al., 28 May 2026). The associated size-bias gap is

$\text{obj}_2$4

The reported pattern mirrors the vertical-distance results: Molmo 2M reaches $\text{obj}_2$5 with $\text{obj}_2$6, NVILA 2M reaches $\text{obj}_2$7 with $\text{obj}_2$8, and RoboRefer maintains $\text{obj}_2$9 with a much smaller $16 \times 16$0 (Min et al., 28 May 2026). This shows that the benchmark framework is adaptable to multiple depth-related shortcuts, even though the main paper focuses on vertical-distance entanglement.

The benchmark’s limitations are explicit. SpatialTunnel is synthetic; its objects are simple cubes and spheres; its geometry is a stylized corridor; and its main task scope is narrow, namely binary relative depth under controlled two-object arrangements (Min et al., 28 May 2026). The paper does not present train/test split details for SpatialTunnel, and it does not claim that the benchmark replaces natural-image evaluation (Min et al., 28 May 2026). Instead, SpatialTunnel is paired with real benchmarks, while the cross-domain coherence analysis is presented as only a partial answer to transfer concerns (Min et al., 28 May 2026).

In contemporary arXiv usage, SpatialTunnel most specifically denotes this benchmark. The phrase also sits within a wider family of “spatial tunneling” concepts in other research areas. In cold-atom physics, Raman pulses can induce coherent tunneling between Wannier-Stark states in a gravity-tilted optical lattice, with spectroscopic selectivity over site separation (Beaufils et al., 2011). In quantum chaos, dynamical tunneling can occur between regular torus regions and a chaotic sea, with tunneling strongly enhanced when the chaotic side delocalizes across an Anderson transition (Ishikawa et al., 2010). In metamaterials, two opaque interlaced metallic wire meshes can exhibit anomalous light tunneling through a low-frequency longitudinal mode and Fano-type destructive interference (Latioui et al., 2017). These are distinct from the VLM benchmark, but they illustrate that “spatial tunnel” language spans multiple technical traditions.

Within VLM evaluation specifically, however, SpatialTunnel denotes a controlled synthetic stress test for depth reasoning under manipulated 2D layout. Its enduring significance lies in making visible a failure mode that broad benchmark averages can conceal: a model may appear spatially competent while still encoding distance through vertical shortcut structure rather than a disentangled depth axis (Min et al., 28 May 2026).