---
title: Layer-Wise Geometric Probing
url: https://www.emergentmind.com/topics/layer-wise-geometric-probing
type: topic
---

# Layer-Wise Geometric Probing

Layer-wise geometric probing is a family of methods for analyzing how representational geometry changes across network depth. Rather than asking only whether a target variable is decodable at some layer, it studies how distances, manifolds, subspaces, activation regions, or concept directions are formed, transformed, and stabilized from layer to layer. Recent work applies this perspective to syntactic structure in language models, psycholinguistic features, out-of-distribution behavior in vision transformers, dense geometric signals in self-supervised vision backbones, clinical content in brain MRI foundation models, multimodal reasoning, and even exact robustness verification in ReLU networks [2506.21861] [2601.03798] [2603.05280] [2607.01987] [2605.01829] [2007.11826].

## 1. Conceptual basis

At its most general, layer-wise geometric probing treats each layer as a distinct representational space and asks how a property of interest is reflected in that space’s geometry. In structural probing for syntax, this geometry is a learned metric: token embeddings are linearly projected into a space where Euclidean distances approximate dependency-tree distances, and the layerwise emergence of that metric is summarized by an expected-layer statistic [2506.21861]. In brain MRI probing, the relevant geometry is the learned manifold of per-layer embeddings, represented by a layer-specific \(k\)-nearest-neighbor graph and used as a prior for sparse dictionary learning [2605.01829]. In residual-stream concept analysis, the object of interest is a trajectory of unit directions \(u^{(l)}\) whose rotation across layers can be quantified by cosine similarity and angular velocity [2605.25848]. In exact verification of ReLU networks, the geometry is the nested hyperplane arrangement induced by successive layers, with partial activation patterns defining a hierarchy of polyhedral regions [2007.11826].

A recurring distinction is between detection of information and characterization of its developmental path. "Derivational Probing" explicitly separates these two questions by asking not only whether syntactic structures are encoded, but how micro-syntactic structures and macro-syntactic structure are constructed across layers [2506.21861]. The same distinction appears in work on answerability: a cosine-distance signal to an answerable centroid can be measured at every layer, revealing that the signal arises in early layers and attenuates toward the output rather than being uniformly present across depth [2605.03196].

The framework is therefore broader than linear probing in the narrow sense. It includes layerwise regression and classification, but also manifold regularization, subspace decomposition, directional ablation, spectral analysis, and hierarchical search over activation regions. What unifies these methods is the assumption that depth is not a homogeneous stack: different layers realize different geometric organizations, and those organizations can be inspected directly [2603.05280] [2606.18439].

## 2. Principal methodological families

Several technically distinct methods now fall under the layer-wise geometric probing rubric.

| Family | Geometric object | Representative mechanism |
|---|---|---|
| Linear and metric probes | Layer-specific linear subspaces or metrics | Structural probes, ridge probes, logistic probes [2506.21861] [2601.03798] [2603.05280] |
| Geometry-regularized decomposition | Layer manifold or graph structure | \(k\)-NN graph regularization of SAE pre-activations [2605.01829] |
| Subspace and direction interventions | Low-rank probe subspaces or concept directions | SVD projection, DOM trajectories, directional ablation [2607.01987] [2605.25848] [2604.23452] |
| Centroid-distance methods | Distances to reference manifolds or centroids | Cosine deviation from answerable centroid [2605.03196] |
| Hierarchical activation geometry | Nested polyhedral regions across layers | Partial activation patterns and layerwise region exploration [2007.11826] |

In the linear and metric family, the probe is usually a simple map \(h^{(l)} \mapsto \hat y\) or \(h^{(l)} \mapsto d(\cdot,\cdot)\). "Derivational Probing" defines scalar-mixed embeddings
\[
m_i^{\ell} = \gamma \sum_{k=0}^{\ell} w_k \mathbf{h}_i^k
\]
and a structural distance
\[
d_{\mathbf{B}_\ell}(m_i^{\ell}, m_j^{\ell}) = \|\mathbf{B}_\ell m_i^{\ell} - \mathbf{B}_\ell m_j^{\ell}\|_2,
\]
then tracks UUAS for specific syntactic subgraphs across \(\ell\) [2506.21861]. In psycholinguistic probing, the corresponding operation is ridge regression from layer embeddings to 58 human-derived features, with selectivity defined as observed \(R^2\) minus a permuted-label baseline [2601.03798]. In ViT probing under distribution shift, the probe is a linear classifier on CLS embeddings extracted not only from different layers but also from different internal modules such as LN2, Act, FC2, and RC2 [2603.05280].

Geometry-regularized decomposition methods replace direct readout with a structured representation learner. GeoSAE trains one sparse autoencoder per transformer layer on BrainIAC CLS embeddings, using a TopK gate chosen from unlabeled geometric diagnostics and adding a manifold smoothness term
\[
\mathcal{L} = \mathcal{L}_{\text{SAE}} + \lambda \sum_{(i,j)\in\mathcal{N}_k} w_{ij}\|a_i-a_j\|_2^2
\]
on SAE pre-activations \(a\) rather than sparse codes \(z\) [2605.01829]. This turns layerwise probing into geometry-aware dictionary learning.

Subspace and direction methods inspect probe geometry directly. "Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention" factorizes a trained probe weight matrix as
\[
\mathbf{W}^{(l)} = \mathbf{U}^{(l)} \boldsymbol{\Sigma}^{(l)} \mathbf{V}^{(l)\top}
\]
and defines aligned projections \(\mathbf{Z}^{(l)}\mathbf{V}^{(l)}_k \mathbf{V}^{(l)\top}_k\), random subspaces, and residual complements to test where explicit task signal lives [2607.01987]. "Geometric Evolution Maps" instead treats the normalized centroid difference \(u^{(l)}\) as the probe and studies its layerwise rotation, identifying a handoff layer after the primary Concept Allocation Zone where directional change effectively ceases [2605.25848]. In "From Edges to Depth," the linear depth-probe direction itself is causally tested by projection-based ablation and activation patching [2604.23452].

Centroid-distance methods dispense with trained probes altogether. In "Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal," the layer-\(l\) answerable centroid is
\[
c^{(l)} = \frac{1}{N}\sum_{i=1}^N h^{(l)}(x_i),
\]
and the geometric deviation of a prompt is
\[
d^{(l)}(x) = 1 - \cos(h^{(l)}(x), c^{(l)}).
\]
The answerability gap \(\delta_l = \overline{d^{(l)}_U} - \overline{d^{(l)}_A}\) then becomes a layerwise pre-generation reliability signal [2605.03196].

Finally, some work uses a geometric hierarchy intrinsic to the network itself. LayerCert defines partial activation patterns and searches the nested hyperplane arrangement induced by a ReLU network layer by layer, rather than treating the full partition as an undifferentiated polyhedral complex [2007.11826].

## 3. Recurrent layer-wise regularities in language and vision

Several empirical regularities recur across domains, although they are not universal.

In language models, local structure often appears earlier than global structure. On BERT-base and BERT-large, "Derivational Probing" reports that micro-syntactic structures such as `nsubj`, `dobj`, and `prep` have lower expected layers than macro-syntactic structure centered on the root verb and its direct dependents, yielding a bottom-up derivation picture in which phrase-level geometry becomes recoverable before clause-level organization [2506.21861]. A separate large-scale study of 58 psycholinguistic features across 10 encoder-only and decoder-only transformers finds a shared depth ordering in which lexical properties peak earlier and experiential and affective dimensions peak later, while final-layer representations are rarely optimal for recovering psycholinguistic information with linear probes [2601.03798]. For answerability, the layerwise geometric signal is form-conditional rather than universal: within mathematical prompts, unanswerable inputs deviate strongly from the answerable centroid and the gap peaks in early layers, whereas no reliable geometric signal emerges for factual prompts [2605.03196].

In vision transformers, intermediate layers are often geometrically preferable to final layers, but the operative reason depends on task and setting. For frozen ViT probing under distribution shift, deeper representations degrade as the shift increases, whereas in the ID setting after finetuning the final layer is best; modulewise, the activation within the feedforward network is best under significant shift and FC2 is worst on 10 of 11 datasets [2603.05280]. In self-supervised ViTs, explicit geometric precision peaks at intermediate layers: for DINOv2-L on NYU-Depth V2, surface normals peak around layer 18, depth around layer 21, and semantic segmentation around the final block, indicating a transition from geometric to semantic emphasis with depth [2607.01987]. In a supervised classification-trained ViT-B/16, local patch boundaries become linearly decodable at layers 5–6 with AP \(= 0.833\), whereas per-patch depth peaks later at layer 8 with MAE \(= 0.0875\), and both signals degrade at the final classification layer [2604.23452]. A broader survey of frozen vision and vision-language encoders for continuous physical measurement likewise reports a universal mid-network accuracy peak across architectures and identifies layers 18–22 as carrying disproportionate geometric signal [2603.06459].

These findings support a constrained but repeated pattern: local or low-level geometric relations tend to emerge before broader or more abstract relations, and final layers often trade geometric accessibility for task specialization. The caveat is that this pattern is architecture- and objective-dependent rather than absolute [2109.05958] [2603.05280].

## 4. Biomedical and multimodal instantiations

In biomedical imaging, layer-wise geometric probing has been used to convert frozen foundation-model representations into clinically annotated feature systems. GeoSAE applies one SAE per transformer layer to BrainIAC, a frozen 12-layer 3D brain MRI ViT with hidden size \(d=768\), using per-layer CLS tokens rather than patch tokens. Its geometric diagnostics show angular-dominant structure (\(\eta^2 = 0.002\)), significant negative activations (48.8% of coordinates negative), and properties that favor TopK; its manifold-regularized SAE prevents deep-layer feature collapse, producing 161 alive features at layer 9 rather than 23 for a standard TopK SAE, with mean pairwise \(|r| = 0.009\) rather than \(0.156\). Layer 9 then becomes the preferred probe layer: conversion AUC peaks at \(0.731 \pm 0.011\) in the layerwise analysis, and logistic regression on the top 16 layer-9 GeoSAE features reaches \(0.746 \pm 0.002\) for MCI\(\rightarrow\)AD prediction using only about 2% of the embedding dimensions, with cross-cohort activation-pattern replication \(r = 0.97\) and attention-rollout localization to regions consistent with Braak staging [2605.01829].

In multimodal large language models, layer-wise geometric probing can be turned into a design principle rather than a post hoc diagnostic. GUIDE samples multiple geometric levels from a VGGT encoder and injects them into the first six decoder layers of a Qwen3-VL backbone, with context-aware tokenwise and layerwise gating. Its depth ablations are explicitly layer-sensitive: with both gates enabled, VSI-Bench average rises from 51.7 for input-only fusion to 53.8 with three injection layers and 55.6 with six injection layers, then falls to 51.1 and 51.6 when injection extends to 9 or 24 layers. This suggests that progressive geometric guidance is useful in early layers but becomes disruptive when extended too far into later semantic processing [2604.05695].

A related multimodal case concerns bias rather than spatial reasoning. In "Don't Look at the Numbers," numeric anchors embedded in images bias VLM quality judgments with \(\eta^2 = 0.18\)–0.77, and layer-wise probing shows a dissociation between anchor decoding and quality representation: anchor classification saturates in layers L12–L34, whereas optimal quality prediction lies deeper, with \(R^2 = 0.69\)–0.91 depending on architecture. Fusion behavior is architecture-dependent, ranging from instant fusion at L1–L2 in Gemma variants to partial or absent fusion in other families [2605.11218]. Here layer-wise geometry links an external behavioral bias to a specific representational timeline.

## 5. Interventions, causality, and verification

A distinctive feature of recent layer-wise geometric probing is the move from purely correlational readout to interventional analysis.

Subspace intervention makes the probe’s geometry itself manipulable. In self-supervised ViTs, the SVD of a converged linear probe isolates a low-rank subspace \(\mathcal{S}^{(l)}_k = \mathrm{span}\{\mathbf{v}_1,\dots,\mathbf{v}_k\}\), allowing aligned projections, random subspace controls, and residual-subspace tests. On NYU-Depth V2, DINOv2-L allocates 17.2%, 35.8%, 36.7%, and 10.3% of global spectral-weighted energy to layers \(l_6, l_{12}, l_{18}, l_{24}\), respectively, showing strong middle-layer concentration; MAE and iBOT distribute energy more diffusely [2607.01987].

Directional trajectory analysis makes stability itself a probe target. Geometric Evolution Maps define angular velocity
\[
\omega(l) = 1 - |u^{(l)} \cdot u^{(l-1)}|
\]
for residual-stream concept directions and extract probes at a handoff layer after the primary Concept Allocation Zone rather than at the peak separation layer. Across 391 concept\(\times\)model pairs, the entry-to-exit cosine within CAZs has mean 0.233, while the handoff cosine with the final-layer direction has mean 0.942. Handoff-layer probes are at least as precise as peak-layer probes in 268 of 391 trials and strictly better in 259 of 391, with a pronounced architecture split between MHA and GQA models; random-direction controls show a median 377x suppression-rate advantage for concept-direction ablation [2605.25848].

At the level of a single direction, causal specificity can be measured directly. In the classification-trained ViT study, ablating the one-dimensional depth-probe direction increases depth MAE by 49–165% depending on layer, while ablating any random direction changes MAE by less than 1%. Targeted activation patching along that direction further shows that depth is partially re-derived at each layer rather than merely carried forward, and that interventions in mid layers have the strongest downstream persistence [2604.23452].

Layer-wise geometric probing also appears in verification and acceleration. LayerCert exploits the nested hyperplane arrangement of a ReLU network by searching over partial activation patterns hierarchically, and under stated conditions provably reduces the number and size of convex programs relative to GeoCert [2007.11826]. RegimeVGGT combines spectral, probing, and causal analyses to partition VGGT’s 24 global-attention layers into shallow, middle, and deep regimes, then uses that partition to design U-shaped token compression and K/V downsampling schedules, yielding a reported 6.7x speedup over VGGT* at matched reconstruction quality [2606.18439].

## 6. Limitations, misconceptions, and future directions

A central misconception is that probe performance directly reveals what a model uses. Multiple papers explicitly caution against this. Structural and psycholinguistic probes show what is linearly decodable, not what is causally used for prediction, and subject–verb agreement correlations or layerwise decoding curves remain correlational unless paired with intervention [2506.21861] [2601.03798]. The same caution appears in cross-model linguistic probing: differences in weight norms across layers can make scalar-mixing evaluations misleading, which is why minimum description length and RSA were proposed as alternatives to raw weight-mixing interpretations [2109.05958].

Another misconception is that there is a universal “best layer.” The evidence is more qualified. Under frozen OOD probing of ViTs, intermediate layers and internal FFN activations are often best, but in ID settings after finetuning the final layer is best [2603.05280]. Across BERToid models, ELECTRA tends to encode linguistic knowledge in deeper layers whereas XLNet concentrates it in earlier layers [2109.05958]. In self-supervised vision, DINOv2 aligns spatial features for efficient linear extraction, while MAE disperses geometric signal and requires broader spatial context; this objective dependence directly changes both layerwise peaks and low-rank structure [2607.01987].

Benchmark scope remains limited in many studies. GeoSAE evaluates one brain MRI foundation model and one disease domain, and its \(k\)-NN graph construction is \(O(N^2 d)\) in principle; the paper explicitly notes uncertainty about transfer to other MRI FMs and conditions such as MS or stroke [2605.01829]. Geometric deviation as a reliability signal works strongly in mathematical prompts and shows partial generalization to code, but no reliable geometric signal emerges for factual prompts, marking a clear boundary on generality [2605.03196]. GUIDE fixes a hand-designed mapping from sampled VGGT layers to the first six MLLM layers and does not impose explicit geometric supervision on intermediate LLM states [2604.05695].

Current directions point toward richer geometry, stronger controls, and broader domains. Suggested extensions include more global manifold tools such as diffusion maps and Riemannian metrics for brain-model probing [2605.01829], information-theoretic, geometric, and invariance measures at the module level for ViTs [2603.05280], multilingual and autoregressive variants of derivational probing [2506.21861], low-rank dense decoders informed by subspace-intervention results [2607.01987], and more systematic behavioral ablation for residual-stream concept probes [2605.25848]. A plausible implication is that the field is moving from static layer scoring toward explicit models of representational evolution, in which depth is analyzed not merely as an index but as a sequence of geometric transformations.

Source: https://www.emergentmind.com/topics/layer-wise-geometric-probing