From Edges to Depth: Probing the Spatial Hierarchy in Vision Transformers

Abstract: Vision Transformers trained only on image classification routinely transfer to tasks that demand spatial understanding, yet they receive no spatial supervision during pretraining. We ask where and how robustly such structure is encoded. Probing a frozen ViT-B/16 layerwise for two complementary properties, local patch boundaries (BSDS500) and per-patch depth (NYU Depth V2), reveals a clear hierarchy: boundary structure becomes linearly decodable at layers 5-6 (AP = 0.833), while depth, which requires integrating global cues, peaks two to three layers later at layer 8 (MAE = 0.0875). Both signals collapse at the final classification layer, and random-weight controls confirm the encodings are learned rather than architectural. Causal interventions add specificity: ablating the single direction a linear depth probe reads degrades depth decoding by up to 165%, while ablating any other direction changes it by less than 1%. Targeted activation patching along that direction shows the depth signal is partially re-derived at each layer rather than passively carried in the residual stream, with mid-layer interventions persisting most strongly downstream. The result is that a classification-trained ViT develops an actively maintained spatial hierarchy that mirrors the early-to-late progression observed in the primate visual cortex.

• The paper demonstrates that ViT-B/16 encodes boundary information at layer 5 (AP=0.833) and depth cues at layer 8 (MAE=0.0875), revealing a clear spatial hierarchy.
• The paper employs layerwise probing, random-weight baselines, and causal ablation to show that linear probes effectively capture spatial features, outperforming overparameterized MLP models.
• The paper’s insights advocate using mid-layer representations for geometric tasks, influencing transfer learning, model compression, and the design of interpretability-focused architectures.

Probing the Spatial Hierarchy in Vision Transformers

Introduction

The paper "From Edges to Depth: Probing the Spatial Hierarchy in Vision Transformers" (2604.23452) systematically investigates the emergence and organization of spatial properties within Vision Transformers (ViTs) trained solely on image classification. The authors analyze where boundary structure and depth information become linearly accessible and causally robust by employing layerwise probing, random weight controls, and targeted interventions. The study provides empirical insight into the spatial hierarchy that emerges in ViT-B/16 without explicit spatial supervision and explores implications for interpretability, model design, and practical deployment.

Methodological Framework

A ViT-B/16 pretrained on ImageNet-21k serves as the primary model, with a random-weight ViT-B/16 as an architectural control. For each layer, patch-level hidden states (excluding the CLS token) are extracted, yielding a 13-layer (embedding + 12 transformer blocks) sequence with 768-dimensional representations per patch. Two tasks probe spatial properties:

• Boundary Detection (BSDS500): Patch-level binary classification of boundary overlap, reflecting local structural sensitivity.
• Depth Regression (NYU Depth V2): Patch-level mean depth estimation (normalized), requiring global spatial layout integration.

Linear and MLP probes are trained to decode these properties at each layer, reporting AP (boundary) and MAE (depth) as primary metrics. The probe's performance across layers is compared for both model variants. Causal interventions include probe-direction ablation and targeted activation patching, isolating the directional encoding with high specificity.

Emergence of Boundary Detection

Boundary detection peaks in the pretrained ViT at layer 5 (AP = 0.833), with random-weight control plateauing at AP ≈ 0.60, establishing a substantial learned signal.

Figure 1: Boundary detection AP by layer; pretrained ViT peaks at layer 5, random-weight baseline is flat.

The steep rise in AP within early layers indicates rapid integration of local context via self-attention, with precision steadily improving at the cost of minimal recall loss. The linear probe achieves comparable or superior performance to the MLP probe despite $257\times$ fewer parameters, confirming linear separability of boundary features. At higher layers, boundary information degrades as representations specialize for classification, reflecting task-specific information loss.

Decodability of Depth Information

Depth prediction achieves optimal MAE = 0.0875 at layer 8 for the pretrained ViT, showing a 23% reduction over the random baseline (MAE ≈ 0.107).

Figure 2: Depth probing MAE by layer; pretrained ViT obtains lowest MAE at layers 5-8.

The improvement over random weights affirms that depth features transcend positional encoding priors. While depth MAE improves in mid-layers, both boundary and depth signals decline at the final classification layer, which discards geometric information. The linear probe consistently outperforms the overparameterized MLP, indicating that depth is encoded in a concise linear subspace and suggesting that additional nonlinear capacity primarily fits noise.

Cross-Task Spatial Hierarchy

Comparing layerwise AP (boundary) and MAE (depth), the paper identifies a consistent 2–3 layer offset: boundary structure emerges by layer 5–6, while depth peaks at layer 8. This temporal gap reflects the extra computation required for integrating global spatial cues, as opposed to local structural contrast.

Figure 3: Cross-task comparison of AP (boundary) and MAE (depth) shows a 2–3 layer offset between peaks, revealing a spatial hierarchy.

The degradation of both signals at the final layer is pronounced, supporting the necessity of tapping mid-layer representations for geometric downstream tasks. The observed spatial progression resembles the global feature integration of early-to-intermediate visual cortex processing.

Causal Analysis: Structure and Active Maintenance

Probe-direction ablation demonstrates that removing the single linear direction responsible for depth predictions increases MAE by up to 165%, while ablating random directions affects MAE by less than 1%, verifying the strong concentration of the depth signal.

Figure 4: Depth MAE increases sharply when ablating the probe’s weight direction; random directions yield negligible change.

Dose-response experiments show graded MAE degradation with increasing ablation strength, confirming monotonic influence and ruling out circularity.

Figure 5: Dose-response curve at layer 8 reveals gradual MAE increase with ablation strength, indicating graded contribution.

Targeted activation patching reveals that depth information is partially re-derived at each layer rather than transmitted passively through the residual stream. Mid-layer interventions persist most strongly downstream, while early-layer effects decay rapidly, highlighting layers where geometric encoding is both accurate and robust.

Figure 6: Causal influence matrix illustrates off-diagonal decay; mid-layer patching retains most effect downstream, confirming active maintenance.

Practical and Theoretical Implications

Transfer Learning: The results strongly advocate for using mid-layer features (layers 5–8) in geometric tasks such as depth estimation or edge detection. Task-specific heads atop these layers can exploit robust spatial representations without full model fine-tuning.

Model Compression: As depth is encoded in a single direction, selective pruning or dimension reduction should preserve critical axes identified via probing, readily informing safe compression strategies for deployment.

Failure Diagnosis: Layerwise probing offers precise localization for errors in spatial tasks, enabling diagnosis of representational breakdowns at specific ViT layers.

Architecture Design: Findings motivate multi-head branching at mid-layers for spatial or structural tasks, rather than routing through the final classification layer, and inform the design of interpretability-targeted architectures.

Concept Auditing: Probe-direction ablation generalizes to other properties, enabling rigorous assessment of whether a model encodes—and relies on—specific features.

Pretraining Strategies: The probing and causal pipeline can quantitatively compare spatial property organization across architectures and training objectives, supporting principled selection of pretraining techniques.

Limitations

The study's scope is restricted by dataset size/domain (NYU Depth V2, BSDS500), patch-level granularity, single ViT-B/16 architecture, and its single-direction ablation. Results may not generalize to other models, self-supervised objectives, or finer spatial structures without adaptation. Ambiguity in boundary annotation and ablation circularity also constrain interpretability.

Conclusion

This study establishes that ViT-B/16 trained solely on classification exhibits an actively maintained spatial hierarchy, with local boundary structure emerging in early layers and global depth layout in mid-layers. Both signals are linearly accessible and causally load-bearing, as demonstrated by ablation and activation patching methodologies. These findings inform transfer learning, compression, interpretability, and architectural design in geometric computer vision, and motivate further research into spatial encoding across models and objectives.