Why Far Looks Up: Probing Spatial Representation in Vision-Language Models

Abstract: Vision-LLMs (VLMs) achieve strong performance on spatial reasoning benchmarks, yet it remains unclear whether this reflects structured 3D understanding or reliance on statistical shortcuts in natural images. We introduce a representation-level analysis framework that constructs minimal contrastive pairs to measure how spatial axes are organized and disentangled within VLM embeddings. Our analysis across multiple model families reveals a consistent vertical-distance entanglement: models conflate vertical image position with distance, mirroring the perspective bias of natural photographs. This bias produces a significant accuracy gap between perspective-consistent and counter-heuristic examples, and intensifies under data scaling even as overall benchmark accuracy improves. We further show that models with similar benchmark scores can exhibit different internal representations, and that these differences predict accuracy and robustness across diverse spatial reasoning benchmarks. To isolate this bias from evaluation-set skew, we introduce SpatialTunnel, a synthetic benchmark designed to expose spatial shortcut biases by removing common correlations present in natural images. Experiments confirm that the entanglement is model-intrinsic, and that models with well-separated spatial axes exhibit greater robustness, suggesting that well-structured spatial representations lead to more reliable spatial reasoning across diverse benchmarks. Code and benchmark are available on the project page: https://cheolhong0916.github.io/whyfarlooksup.github.io/.

• The paper introduces a novel analysis framework that uses minimal contrastive probing combined with a synthetic SpatialTunnel benchmark to diagnose spatial representation biases in VLMs.
• It reveals that vision-language models frequently conflate vertical image cues with depth, leading to perspective-driven shortcuts that undermine genuine 3D spatial reasoning.
• The study demonstrates that disentangled spatial axes in embedding space are predictive of robust spatial reasoning, emphasizing the need for targeted supervision and novel architectural interventions.

Probing and Diagnosing Spatial Representation Biases in Vision-LLMs

Introduction

The paper "Why Far Looks Up: Probing Spatial Representation in Vision-LLMs" (2605.30161) investigates the internal spatial reasoning mechanisms of contemporary Vision-LLMs (VLMs). It provides a technical diagnosis of the representational entanglement between 2D image-plane vertical position and 3D depth (vertical-distance entanglement) within VLMs, which results in the use of perspective-driven shortcuts rather than genuine 3D spatial reasoning. The authors introduce a systematic representation-level analysis framework combining minimal contrastive probing with synthetic, bias-controlled evaluation via the SpatialTunnel benchmark, decoupling prevalent cues in natural images. The study demonstrates that representational structure—specifically, disentangled spatial axes with high distance coherence—is tightly predictive of robust spatial reasoning, whereas benchmark accuracy alone can be confounded by shortcut exploitation and does not reliably reflect true 3D spatial understanding.

Figure 1: Many VLMs answer spatial questions via a perspective-driven shortcut, entangling vertical position with distance, leading to systemic failures on counterexamples; disentangled models show consistent correctness.

Perspective-driven Shortcuts and the Vertical-Distance Entanglement Problem

VLMs are increasingly deployed in scenarios that require compositional spatial reasoning (e.g., robotics, embodied agents, multimodal assistants), but their spatial reasoning training is predominantly anchored in 2D image–text pairs. While aggregate benchmark performance often appears strong, internal analysis and behavioral probes indicate that this strength is at least partially underwritten by exploitation of statistical regularities in image corpora—most notably, the correlation induced by single-view perspective: objects appearing higher in-ground-plane images are probabilistically farther from the camera.

This vertical-distance entanglement manifests as models systematically mapping "above" to "far" and "below" to "close" during reasoning, breaking down on spatial configurations that are counter to this heuristic (i.e., counter-examples where a farther object is lower in the image).

Synthetic Benchmarking: SpatialTunnel

To control and decouple the spatial cues prevalent in natural images, the authors introduce the SpatialTunnel benchmark, a synthetic Blender-based environment. Here, two objects can have arbitrary vertical and angular positions within a symmetric tunnel, but their actual 3D depths are held fixed and strictly controlled.

Figure 2: SpatialTunnel benchmark: two objects are placed at fixed depths, but swept in angular position, decoupling image-plane layout from actual depth ordering.

This construction isolates whether VLMs can perform genuine depth reasoning or fall back to heuristic shortcuts based on vertical position or apparent size.

Behavioral and Internal Representation Analysis

Shortcut Diagnosis on Benchmarks

Across real-world and synthetic benchmarks (e.g., EmbSpatial-Bench, CV-Bench, BLINK), all evaluated VLMs exhibit marked accuracy drops on vertically counter-heuristic (counter) samples compared to consistent ones. This holds across architectures, family, and scale, persisting even with substantial spatial fine-tuning. The distribution of natural images is heavily skewed toward perspective-consistent cases (up to 80%); thus, naive accuracy is an unreliable indicator of robustness.

SpatialTunnel Results

Contrastive evaluation on the unbiased, synthetic SpatialTunnel demonstrates that most models retain a significant performance gap ($\Delta$) favoring consistent cases even after scaling fine-tuning datasets. Only models trained on very large, unbiased datasets (e.g., RoboRefer-2B-SFT, Qwen3-VL-235B) or with targeted supervision begin to close this gap and attain both high absolute accuracy and low entanglement.

Figure 3: Results on consistent samples in the SpatialTunnel environment; consistent accuracy improves with fine-tuning, but counter cases remain challenging for most models.

Figure 4: Positive correlation between behavioral accuracy on counter examples and internal distance coherence in representations.

Representation-level Probing: Contrastive Delta Analysis

The paper uses minimal contrastive probing: for a given VQA task, swapping the object order flips the ground-truth relation but preserves all confounds; in the embedding space, the difference (delta) of the final hidden-state for each pair traces the encoding of spatial axes.

Figure 5: Contrastive probing process: constructing question pairs by object swap, extracting their embedding difference, and analyzing aggregated deltas to diagnose spatial cue entanglements.

Principal component analysis of these deltas reveals that:

• The horizontal axis (left/right) is robustly encoded as an unentangled direction.
• The vertical and distance axes are frequently conflated; only the best models obtain cleanly separated clusters for all three axes.
• In strong, well-supervised models (e.g., Qwen3-VL-235B, RoboRefer), all three axes are orthogonal at the embedding level, which coincides with both high counter-example accuracy and generalization across benchmarks.

  Figure 6: PCA of delta vectors: Qwen3 and RoboRefer families exhibit clean three-way separation in representation space, while others show only horizontal/vertical distinction or overlap of distance with vertical cues.

Metrics: Distance Coherence and VD-Entanglement Index

Two key metrics are introduced:

• Distance Coherence ($\mathrm{Coh}_D$): Mean cosine similarity among sign-aligned deltas on the depth axis—measures whether 'far' and 'close' form a stable, unique direction in embedding space. Strong correlation with generalization and counter-example accuracy: models with high $\mathrm{Coh}_D$ are robust outside the natural image regime.
• VD-Entanglement Index (VD-EI): Aggregate cosine similarity of perspective-aligned (above/far, below/close) and perspective-opposing (above/close, below/far) pairs; high positive VD-EI indicates vertical-depth entanglement.

  Figure 7: Cross-domain correlation: Distance coherence measured on synthetic (SpatialTunnel) and real datasets tracks counter accuracy across model families.

Additional Findings: Sensitivity to Apparent Size

Beyond vertical position, VLMs were also shown to be sensitive to apparent object size as a proxy for depth. When this cue is decoupled from true depth, models with high training accuracy exhibit reduced robustness, indicating that shortcut reliance goes beyond vertical position.

Figure 8: Synthetic object-size variation: sizes $s_1$, $s_2$ sweep for objects at fixed depth, enabling control over size-depth cue alignment or conflict.

Figure 9: Correctness plotted as a function of object size. Many models exhibit clear performance drops as size conflicts with depth, evidencing entanglement of size-based cues.

Implications and Future Directions

Theoretical Implications:

The findings highlight a critical divergence between output-level benchmark performance and the learned latent structure of spatial representations. Models might display strong accuracy on natural datasets while internally encoding spatial axes in an entangled manner, susceptible to shortcut exploitation and failing under distribution shift. The notion of representational coherence and axis disentanglement serves as a more granular, diagnostic criterion for genuine spatial reasoning, suggesting future evaluations should prioritize internal analysis over standard behavioral metrics.

Practical Implications:

For spatial reasoning tasks (robotics, AR/VR, embodied question answering), current VLMs may lack the compositional spatial understanding required for out-of-distribution robustness. Naive scaling of training sets does not guarantee axis disentanglement; instead, targeted supervision, counterfactual synthetic data, and explicit architectural biases may be necessary.

Future Research Directions:

• Systematic introduction of synthetic, counterfactual environments for training and evaluation across all spatial axes and confounding cues.
• Architectural interventions or loss functions to incentivize orthogonal and disentangled spatial representations, e.g., via explicit manifold regularization or auxiliary directionality objectives.
• Extending contrastive probing diagnostics to other multimodal reasoning axes, such as temporal, causal, and physical dynamics.
• Benchmarking emergent, larger-scale VLMs to verify if scale alone can eventually induce disentangled spatial representations, or if intrinsic limitations persist without change in data or supervision strategy.

Conclusion

This study provides robust evidence that many VLMs systematically confound 2D image-plane cues with actual 3D relations, most notably conflating vertical position with depth ("why far looks up"), and that high benchmark accuracy can mask fundamental representational shortcomings. Axis separation in embedding space (particularly the emergence of a distinct, coherent distance axis) is a necessary precondition for robust, generalizable spatial reasoning. These results inform both diagnostic evaluation and future model and dataset design for spatially grounded AI systems.