Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention

Abstract: We introduce a controlled subspace intervention framework to investigate how self-supervised Vision Transformers (ViTs) encode dense geometric information. While linear probing is widely used to assess geometric representations, it treats features as a black box, failing to disentangle the underlying topology. To address this issue, we decompose the weights of converged linear probes to isolate the low-rank subspaces containing explicit geometric signals using Singular Value Decomposition (SVD). Our perspective yields three key insights: (1) Pre-training objectives determine how features are encoded. DINOv2 aligns spatial features for efficient linear extraction, while Masked Autoencoders (MAE) tend to disperse these signals, requiring a broader spatial context. (2) Explicit geometric representations are highly compressible, suggesting dense predictive heads could potentially be constrained to low-rank subspaces with minimal performance loss. (3) The layer-wise task affinity suggests that geometric precision peaks at intermediate layers before yielding to semantic abstraction in the final layers. By connecting internal encoding mechanics with downstream performance, these findings provide a basis for effective feature selection and lightweight decoder design. The source code is available at https://github.com/Zhou-Weichen/Geosubprobe.

• The paper introduces a subspace-based intervention framework to isolate low-rank, task-aligned geometric subspaces in self-supervised Vision Transformers.
• It compares DINOv2, MAE, and iBOT, revealing distinct encoding strategies and varying linear accessibility of geometric cues.
• Quantitative and qualitative analyses show that pre-training objectives impact geometric compressibility and layerwise signal routing in ViTs.

Subspace Intervention for Understanding Geometric Encoding in Self-Supervised Vision Transformers

Introduction and Motivation

This paper presents a rigorous subspace-based diagnostic methodology for analyzing the encoding of dense geometric information in self-supervised Vision Transformers (ViTs) (2607.01987). Conventional probing techniques—primarily centered around final prediction accuracy of linear or non-linear heads—treat high-dimensional feature spaces as opaque, occluding the underlying topological structure of the representations. The authors instead develop an analytical framework that decomposes converged probe weights, enabling precise isolation and quantification of low-rank, task-aligned geometric subspaces. This approach offers novel insights into the relationship between pre-training objectives (self-distillation, masked autoencoding, and hybrids) and the accessibility and topological organization of geometric information.

Figure 1: Overview of the controlled subspace intervention analysis framework. (A) Evaluating frozen ViT backbone features using a three-tier probe hierarchy to decouple non-linear entanglement from spatial fragmentation, then using SVD on probe weights to extract and intervene on task-aligned subspaces.

Analytical Framework

The proposed framework consists of two principal components: a three-tier probing hierarchy and a post-hoc subspace intervention. The probing hierarchy employs linear probes, point-wise MLPs (to gauge local non-linear entanglement), and global DPT-style decoders to measure the "readability gap"―the discrepancy in task performance attributable to nonlinear structure or spatial dispersion. A key methodological innovation is the SVD-based decomposition of converged linear probe weights, which identifies the principal directions most aligned with explicit geometric signals. By projecting features onto these task-aligned subspaces and evaluating performance under frozen decoders, the framework strictly quantifies the linear compressibility of geometric information and distinguishes true absence from mere entanglement or fragmentation.

Encoding Structure across Self-Supervised Paradigms

Applying the intervention analysis to DINOv2 (self-distillation), MAE (masked image modeling), and iBOT (hybrid), the authors provide evidence for sharply contrasting encoding strategies:

• DINOv2: Geometric information is largely aligned into a highly linearly accessible, low-rank coordinate system. The linear probe alone recovers over 91% SA-$\delta_1$ accuracy for depth estimation, with minimal gain from point-wise nonlinearity or global spatial aggregation, indicating minimal structural entanglement.
• MAE: Geometric signals are spatially dispersed and only weakly accessible by linear projections (60% SA-$\delta_1$). Substantial gains are realized only with decoders aggregating broader spatial context, implying that pixel-wise reconstruction objectives entangle geometric cues across tokens.
• iBOT: Exhibits intermediate behavior, balancing linear accessibility and spatial dispersion.

Subspace Compressibility and Visual Evidence

The SVD-based intervention reveals strong compressibility in the explicitly decodable geometric manifold. For all models, 98% or greater linear performance can be achieved with 32-128 basis vectors. However, DINOv2 requires a larger subspace dimensionality to saturate, reflecting richer linearly encoded spatial details. Qualitative visualization demonstrates that DINOv2 produces coherent scene layouts even at $k=8$ (very low rank), while MAE necessitates at least $k\ge64$ to resolve object boundaries, a signature of highly distributed encoding.

Figure 2: Qualitative visualization of subspace interventions. Depth predictions from DINOv2, MAE, and iBOT show contrast in linear compressibility—DINOv2 yields structured layout at $k=8$, while MAE requires higher rank.

Energy Allocation and Layerwise Routing

Spectral energy analysis exposes pre-training-driven routing architectures. DINOv2 localizes explicit geometry within intermediate layers (over 72% of geometric energy), with terminal layers favoring semantic abstraction. iBOT and MAE distribute geometric signals more uniformly, necessitating multi-layer aggregation for optimal performance.

Figure 3: Energy distribution across singular directions. DINOv2 demonstrates multimodal peaks in intermediate layers, unlike the flat, uniform distribution in MAE and iBOT.

Layerwise Rank Sensitivity and Task Affinity

Investigating single layers, DINOv2 exhibits a structural transition: early/mid layers encode geometry compactly (full geometry recoverable with $k=8$), while terminal layers experience sharp loss of explicit geometric information. Masked autoencoding results in persistent sensitivity to subspace truncation across all depths.

Figure 4: Layer-wise subspace analysis indicating different trajectories of geometric capacity and entanglement across pre-training paradigms.

Layerwise task affinity analysis further reveals that peaks for surface normals, depth, and semantics occur at increasingly deeper layers, reflecting a shift from geometric to semantic encoding in DINOv2.

Figure 5: Qualitative task affinity in DINOv2; different tasks (normals, depth, semantic segmentation) achieve peak performance at distinct layers, highlighting representation specialization.

Implications, Limitations, and Future Directions

These findings have broad implications for backbone and decoder design in dense prediction. The unique low-rank structure of geometric representations in self-supervised ViTs suggests that decoders can be radically compressed or even routed adaptively, with emphasis on mid-hierarchy features for geometric tasks and terminal features for semantics. Furthermore, the distinct alignment induced by pre-training can inform the selection of SSL objectives for task-specific transfer.

However, the framework is inherently limited to linear projections and cannot resolve highly non-linear entanglement. Its evaluation is mainly constrained to domains with comprehensive, pixel-aligned multi-task annotations (such as NYUv2); thus, findings may not generalize to outdoor or unconstrained visual scenes. Future work should expand to non-linear and manifold-based interventions as well as out-of-domain generalization.

Conclusion

This work demonstrates that self-supervised ViTs—depending on pre-training protocol—encode geometric information with distinct compressibility, layerwise routing, and task affinity properties. Controlled subspace intervention establishes a rigorous foundation for quantifying and visualizing the explicit geometric content of representations, moving beyond black-box probe metrics. These insights guide both theoretical understanding of representation topology and practical design for lightweight, adaptive decoders and feature selection in multi-task vision pipelines.