Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning

Abstract: The standard way to read latent knowledge out of a model, a linear probe confirmed by a steering recovery, can systematically overstate what a vision-LLM (VLM) actually grounds in the image. We show this on spatial reasoning, where the error is invisible to both probing and steering yet exposed by a one-line causal control: replacing the image with a gray blank. Probes decode the within-axis answer at 73--97% across axes, and a training-free projection lifts a near-chance axis from 59% to 79%, exactly the signature of unlocking latent knowledge. The blank-image arbiter refutes it, revealing three grounding regimes that probing conflates: an axis can be grounded (vision-dependent, correct), a prior (vision-independent, with its decode and its apparent recovery a directional default rather than perception), or, surprisingly, inverted: decodable, causally controllable, but deployed with the wrong sign, so the model scores below chance and the error requires looking. The taxonomy holds across the studied VLMs: in fourteen models spanning six language-model families and 2B--27B, horizontal is grounded, vertical is a prior, and depth is inverted, with the inversion emerging at scale within families. The decode-versus-deploy inversion replicates on seven of eight models across five families, and the minimal edit that re-deploys it varies with geometry: a training-free rotation matches a trained edit on the cleanest model, while distributed inversions need a trained low-rank edit, tracing a per-model correction-complexity spectrum. The cheap, self-calibrating arbiter cleanly separates grounded perception, inverted perception, and prior substitution; we argue it should be a default control for latent-knowledge and steering claims in VLMs.

• The paper introduces a vision ablation test to distinguish between grounded, prior-driven, and inverted spatial reasoning in VLMs.
• It uses probing, steering, and intervention techniques to show that high linear decodability of spatial features does not ensure their causal deployment.
• Results across 14 models reveal systematic errors like depth inversion that scale with model size, necessitating model-specific geometric corrections.

Decodable ≠ Grounded: Auditing Spatial Reasoning in VLMs via Vision Ablation

Introduction

The investigation challenges the prevailing methodology for assessing latent knowledge and spatial reasoning in vision-LLMs (VLMs). By systematically applying probing, steering, and intervention techniques, the study demonstrates that high linear decodability of spatial attributes does not guarantee those features are causally deployed in VLM behavior. The focal point is the dissociation between "decodable" internal structure and actual vision-grounded reasoning, particularly for spatial axes such as left/right (horizontal), above/below (vertical), and front/back (depth).

Methodology

The analysis employs a multi-step experimental protocol applied primarily to Qwen2.5-VL-7B and then extended to 14 VLMs across multiple families and scales (2B–27B parameters). The procedure involves:

• Behavioral Decomposition: Using ViewSpatial-Bench, the paper isolates questions targeting egocentric spatial axes.
• Linear Probing: Logistic probes decode axis-polarity from residual activations at each layer, evaluating the presence and linearity of spatial representations.
• Steering and Projection: Training-free, label-free projection interventions amplify relevant directions in residual space to test for latent actionable knowledge.
• Vision Ablation Arbiter: A critical innovation is a one-line causal control—replacing the actual image input with a gray blank, keeping all else fixed. This directly reveals the vision-dependence of the model's behavior.

The evaluation extends to external datasets (What'sUp, 3DSRBench), diverse ablations (gray, black, noise, patch-scramble, mismatched images), and systematic correction methods (rotation, recalibration, learned edits).

Key Findings

Decodability–Deployability Dissociation

• Horizontal Axis (L/R): High probe decodability (∼85%) coincides with high behavioral accuracy (∼81%). Vision ablation collapses accuracy to chance, confirming genuine grounding.
• Vertical Axis (Up/Down): Highest probe decodability (∼94%) but behavioral accuracy is only marginally above chance (∼59%). Vision ablation leaves performance unchanged—indicating the answer is prior-driven, not visually grounded.
• Depth Axis (Front/Back): Moderate probe decodability (∼77%) with accuracy below chance (∼31%). Vision ablation elevates accuracy back toward chance. The probe reads a decodable feature, but the model deploys it with an inverted sign—"seeing" but responding systematically incorrectly.

Vision-Ablation Taxonomy

The vision ablation test distinguishes three distinct regimes:

• Grounded (Vision-dependent, correct): Model’s behavior collapses with ablation (horizontal).
• Prior-driven (Vision-independent): Model’s behavior is unaffected by visual input (vertical).
• Inverted (Vision-dependent, wrong sign): Model's behavior is negatively correlated with truth; ablation actually removes the systematic error (depth).

These regimes hold across a broad range of models and architectures. Depth inversion consistently emerges with scale increases within VLM families.

Steering Interventions: Illusory and Genuine Recovery

Training-free projection interventions can apparently recover vertical spatial reasoning—raising accuracy by +19–21 percentage points—but the gain persists even when the model is "blinded" (no image). The method’s apparent success is shown to amplify a linguistic prior, not true latent visual knowledge. For depth, simple sign flips are insufficient; only more sophisticated edits (e.g., trained low-rank rotation) succeed where a clean, low-dimensional solution exists.

Generalization Across Models and Tasks

The taxonomy (horizontal = grounded, vertical = prior, depth = inverted) consistently recurs across 14 VLMs, including those with non-Qwen backbones (e.g., Pixtral, Gemma). Horizontal grounding is universal (above the capability floor). Depth inversion scales with model size but exhibits family-dependent monotonicity and onset.

Crucially, the depth inversion is task-type-dependent: VLMs that invert on egocentric "front/back" (ViewSpatial) are correctly grounded for allocentric "front/behind" or "closer/farther" in other datasets (What'sUp, 3DSRBench). This demonstrates that the observed pathology is not a result of dataset artifacts or label conventions, but a specific limitation in how egocentric spatial frames are entangled in current VLMs.

Implications and Theoretical Significance

The primary claim is that linear decodability and steering-based recovery are not sufficient to assert grounded knowledge or latent visual capabilities in VLMs. Without explicit causal controls (e.g., vision ablation), interventions may amplify latent priors, which are orthogonal to vision-conditioned reasoning. This exposes a systematic methodological shortcoming in VLM interpretability and mechanistic claims. Neither high probe accuracy nor steered behavioral improvements can, alone, certify that knowledge is visually grounded.

The three-way taxonomy also clarifies the underlying landscape of VLM spatial reasoning failures. Not all “failures” are equal—some reflect prior dominance, while others involve consistent but incorrect deployment of “seen” information, severely impacting interpretability and downstream use.

Correcting inverted deployment, when possible, requires model-specific geometric interventions; clean norm-preserving rotations suffice only with a well-structured, low-dimensional inversion (e.g., Qwen3-VL-8B), while more distributed errors need learned, broad edits.

Practical Considerations and Recommendations

• Vision Ablation Arbiter: The gray-blank image test is computationally negligible, robust (across multiple ablations and even in-distribution mismatched images), and immediately falsifies over-ambitious claims of latent knowledge. It should be a standard control in all latent-knowledge and steering evaluations in spatial VLM tasks.
• Model Correction: Diagnosis of the minimal sufficient edit becomes a geometric signature of the VLM, exposing its internal failure mode (rotation, distributed entanglement, uncorrectable).
• Task Design: Spatial evaluation tasks must discriminate between visually-grounded and prior-driven behavior on a per-axis and per-task-type basis to ensure meaningful measurement of VLM capabilities.
• Reporting Standards: Accuracy gains that persist after vision ablation should be explicitly labeled as prior amplification rather than latent knowledge recovery.

Limitations and Future Work

• Depth inversion’s dependence on data conventions or learning dynamics (e.g., frame ambiguity, training regime) is unresolved. Tracing the cause of systematic inversion is an open avenue for mechanistic and training-data analysis.
• Limitation to representation-level intervention and analysis. Pixel-level or imagery-level minimal pairs for egocentric spatial relations are challenging to construct, constraining causal claims to residual representations.

Broader application of the ablation-arbiter methodology could extend well beyond spatial reasoning—other domains where “latent knowledge” is claimed via probing or steering should be revisited under similar causal controls.

Conclusion

This study rigorously separates linear decodability from grounded deployment in VLM spatial reasoning and demonstrates that many existing claims about latent visual capabilities are methodological artifacts absent direct causal auditing. The vision-ablation arbiter provides a scalable, model-agnostic, and discriminating control for any groundedness claim in VLM interpretability. Future developments in VLMs and spatial intelligence should employ such controls as standard, both for practical efficacy and theoretical soundness.

Reference:

"Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning" (2606.31257)