Layer-Wise Geometric Probing

• Layer-wise geometric probing is a family of methods that analyzes evolving representational geometry across neural network layers to reveal structured information.
• It distinguishes between detecting decodable signals and tracking their transformation across layers, enabling nuanced analysis of model internals.
• Techniques include linear probes, manifold regularization, subspace interventions, and causal ablations to assess layer-specific feature development.

Layer-wise geometric probing is a family of methods for analyzing how representational geometry changes across network depth. Rather than asking only whether a target variable is decodable at some layer, it studies how distances, manifolds, subspaces, activation regions, or concept directions are formed, transformed, and stabilized from layer to layer. Recent work applies this perspective to syntactic structure in LLMs, psycholinguistic features, out-of-distribution behavior in vision transformers, dense geometric signals in self-supervised vision backbones, clinical content in brain MRI foundation models, multimodal reasoning, and even exact robustness verification in ReLU networks (Someya et al., 27 Jun 2025, Tikhomirova et al., 7 Jan 2026, Odonnat et al., 5 Mar 2026, Zhou et al., 2 Jul 2026, Nerrise et al., 3 May 2026, Lim et al., 2020).

1. Conceptual basis

At its most general, layer-wise geometric probing treats each layer as a distinct representational space and asks how a property of interest is reflected in that space’s geometry. In structural probing for syntax, this geometry is a learned metric: token embeddings are linearly projected into a space where Euclidean distances approximate dependency-tree distances, and the layerwise emergence of that metric is summarized by an expected-layer statistic (Someya et al., 27 Jun 2025). In brain MRI probing, the relevant geometry is the learned manifold of per-layer embeddings, represented by a layer-specific $k$-nearest-neighbor graph and used as a prior for sparse dictionary learning (Nerrise et al., 3 May 2026). In residual-stream concept analysis, the object of interest is a trajectory of unit directions $u^{(l)}$ whose rotation across layers can be quantified by cosine similarity and angular velocity (Henry, 25 May 2026). In exact verification of ReLU networks, the geometry is the nested hyperplane arrangement induced by successive layers, with partial activation patterns defining a hierarchy of polyhedral regions (Lim et al., 2020).

A recurring distinction is between detection of information and characterization of its developmental path. "Derivational Probing" explicitly separates these two questions by asking not only whether syntactic structures are encoded, but how micro-syntactic structures and macro-syntactic structure are constructed across layers (Someya et al., 27 Jun 2025). The same distinction appears in work on answerability: a cosine-distance signal to an answerable centroid can be measured at every layer, revealing that the signal arises in early layers and attenuates toward the output rather than being uniformly present across depth (Du, 4 May 2026).

The framework is therefore broader than linear probing in the narrow sense. It includes layerwise regression and classification, but also manifold regularization, subspace decomposition, directional ablation, spectral analysis, and hierarchical search over activation regions. What unifies these methods is the assumption that depth is not a homogeneous stack: different layers realize different geometric organizations, and those organizations can be inspected directly (Odonnat et al., 5 Mar 2026, You et al., 16 Jun 2026).

2. Principal methodological families

Several technically distinct methods now fall under the layer-wise geometric probing rubric.

Family	Geometric object	Representative mechanism
Linear and metric probes	Layer-specific linear subspaces or metrics	Structural probes, ridge probes, logistic probes (Someya et al., 27 Jun 2025, Tikhomirova et al., 7 Jan 2026, Odonnat et al., 5 Mar 2026)
Geometry-regularized decomposition	Layer manifold or graph structure	$k$-NN graph regularization of SAE pre-activations (Nerrise et al., 3 May 2026)
Subspace and direction interventions	Low-rank probe subspaces or concept directions	SVD projection, DOM trajectories, directional ablation (Zhou et al., 2 Jul 2026, Henry, 25 May 2026, Sanghavi, 25 Apr 2026)
Centroid-distance methods	Distances to reference manifolds or centroids	Cosine deviation from answerable centroid (Du, 4 May 2026)
Hierarchical activation geometry	Nested polyhedral regions across layers	Partial activation patterns and layerwise region exploration (Lim et al., 2020)

In the linear and metric family, the probe is usually a simple map $h^{(l)} \mapsto \hat y$ or $h^{(l)} \mapsto d(\cdot,\cdot)$. "Derivational Probing" defines scalar-mixed embeddings

$$m_i^{\ell} = \gamma \sum_{k=0}^{\ell} w_k \mathbf{h}_i^k$$

and a structural distance

$$d_{\mathbf{B}_\ell}(m_i^{\ell}, m_j^{\ell}) = \|\mathbf{B}_\ell m_i^{\ell} - \mathbf{B}_\ell m_j^{\ell}\|_2,$$

then tracks UUAS for specific syntactic subgraphs across $\ell$ (Someya et al., 27 Jun 2025). In psycholinguistic probing, the corresponding operation is ridge regression from layer embeddings to 58 human-derived features, with selectivity defined as observed $R^2$ minus a permuted-label baseline (Tikhomirova et al., 7 Jan 2026). In ViT probing under distribution shift, the probe is a linear classifier on CLS embeddings extracted not only from different layers but also from different internal modules such as LN2, Act, FC2, and RC2 (Odonnat et al., 5 Mar 2026).

Geometry-regularized decomposition methods replace direct readout with a structured representation learner. GeoSAE trains one sparse autoencoder per transformer layer on BrainIAC CLS embeddings, using a TopK gate chosen from unlabeled geometric diagnostics and adding a manifold smoothness term

$$\mathcal{L} = \mathcal{L}_{\text{SAE}} + \lambda \sum_{(i,j)\in\mathcal{N}_k} w_{ij}\|a_i-a_j\|_2^2$$

on SAE pre-activations $u^{(l)}$0 rather than sparse codes $u^{(l)}$1 (Nerrise et al., 3 May 2026). This turns layerwise probing into geometry-aware dictionary learning.

Subspace and direction methods inspect probe geometry directly. "Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention" factorizes a trained probe weight matrix as

$u^{(l)}$2

and defines aligned projections $u^{(l)}$3, random subspaces, and residual complements to test where explicit task signal lives (Zhou et al., 2 Jul 2026). "Geometric Evolution Maps" instead treats the normalized centroid difference $u^{(l)}$4 as the probe and studies its layerwise rotation, identifying a handoff layer after the primary Concept Allocation Zone where directional change effectively ceases (Henry, 25 May 2026). In "From Edges to Depth," the linear depth-probe direction itself is causally tested by projection-based ablation and activation patching (Sanghavi, 25 Apr 2026).

Centroid-distance methods dispense with trained probes altogether. In "Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal," the layer-$u^{(l)}$5 answerable centroid is

$u^{(l)}$6

and the geometric deviation of a prompt is

$u^{(l)}$7

The answerability gap $u^{(l)}$8 then becomes a layerwise pre-generation reliability signal (Du, 4 May 2026).

Finally, some work uses a geometric hierarchy intrinsic to the network itself. LayerCert defines partial activation patterns and searches the nested hyperplane arrangement induced by a ReLU network layer by layer, rather than treating the full partition as an undifferentiated polyhedral complex (Lim et al., 2020).

3. Recurrent layer-wise regularities in language and vision

Several empirical regularities recur across domains, although they are not universal.

In LLMs, local structure often appears earlier than global structure. On BERT-base and BERT-large, "Derivational Probing" reports that micro-syntactic structures such as nsubj, dobj, and prep have lower expected layers than macro-syntactic structure centered on the root verb and its direct dependents, yielding a bottom-up derivation picture in which phrase-level geometry becomes recoverable before clause-level organization (Someya et al., 27 Jun 2025). A separate large-scale study of 58 psycholinguistic features across 10 encoder-only and decoder-only transformers finds a shared depth ordering in which lexical properties peak earlier and experiential and affective dimensions peak later, while final-layer representations are rarely optimal for recovering psycholinguistic information with linear probes (Tikhomirova et al., 7 Jan 2026). For answerability, the layerwise geometric signal is form-conditional rather than universal: within mathematical prompts, unanswerable inputs deviate strongly from the answerable centroid and the gap peaks in early layers, whereas no reliable geometric signal emerges for factual prompts (Du, 4 May 2026).

In vision transformers, intermediate layers are often geometrically preferable to final layers, but the operative reason depends on task and setting. For frozen ViT probing under distribution shift, deeper representations degrade as the shift increases, whereas in the ID setting after finetuning the final layer is best; modulewise, the activation within the feedforward network is best under significant shift and FC2 is worst on 10 of 11 datasets (Odonnat et al., 5 Mar 2026). In self-supervised ViTs, explicit geometric precision peaks at intermediate layers: for DINOv2-L on NYU-Depth V2, surface normals peak around layer 18, depth around layer 21, and semantic segmentation around the final block, indicating a transition from geometric to semantic emphasis with depth (Zhou et al., 2 Jul 2026). In a supervised classification-trained ViT-B/16, local patch boundaries become linearly decodable at layers 5–6 with AP $u^{(l)}$9, whereas per-patch depth peaks later at layer 8 with MAE $k$0, and both signals degrade at the final classification layer (Sanghavi, 25 Apr 2026). A broader survey of frozen vision and vision-language encoders for continuous physical measurement likewise reports a universal mid-network accuracy peak across architectures and identifies layers 18–22 as carrying disproportionate geometric signal (Shkolnikov, 6 Mar 2026).

These findings support a constrained but repeated pattern: local or low-level geometric relations tend to emerge before broader or more abstract relations, and final layers often trade geometric accessibility for task specialization. The caveat is that this pattern is architecture- and objective-dependent rather than absolute (Fayyaz et al., 2021, Odonnat et al., 5 Mar 2026).

4. Biomedical and multimodal instantiations

In biomedical imaging, layer-wise geometric probing has been used to convert frozen foundation-model representations into clinically annotated feature systems. GeoSAE applies one SAE per transformer layer to BrainIAC, a frozen 12-layer 3D brain MRI ViT with hidden size $k$1, using per-layer CLS tokens rather than patch tokens. Its geometric diagnostics show angular-dominant structure ($k$2), significant negative activations (48.8% of coordinates negative), and properties that favor TopK; its manifold-regularized SAE prevents deep-layer feature collapse, producing 161 alive features at layer 9 rather than 23 for a standard TopK SAE, with mean pairwise $k$3 rather than $k$4. Layer 9 then becomes the preferred probe layer: conversion AUC peaks at $k$5 in the layerwise analysis, and logistic regression on the top 16 layer-9 GeoSAE features reaches $k$6 for MCI$k$7AD prediction using only about 2% of the embedding dimensions, with cross-cohort activation-pattern replication $k$8 and attention-rollout localization to regions consistent with Braak staging (Nerrise et al., 3 May 2026).

In multimodal LLMs, layer-wise geometric probing can be turned into a design principle rather than a post hoc diagnostic. GUIDE samples multiple geometric levels from a VGGT encoder and injects them into the first six decoder layers of a Qwen3-VL backbone, with context-aware tokenwise and layerwise gating. Its depth ablations are explicitly layer-sensitive: with both gates enabled, VSI-Bench average rises from 51.7 for input-only fusion to 53.8 with three injection layers and 55.6 with six injection layers, then falls to 51.1 and 51.6 when injection extends to 9 or 24 layers. This suggests that progressive geometric guidance is useful in early layers but becomes disruptive when extended too far into later semantic processing (Wang et al., 7 Apr 2026).

A related multimodal case concerns bias rather than spatial reasoning. In "Don't Look at the Numbers," numeric anchors embedded in images bias VLM quality judgments with $k$9–0.77, and layer-wise probing shows a dissociation between anchor decoding and quality representation: anchor classification saturates in layers L12–L34, whereas optimal quality prediction lies deeper, with $h^{(l)} \mapsto \hat y$0–0.91 depending on architecture. Fusion behavior is architecture-dependent, ranging from instant fusion at L1–L2 in Gemma variants to partial or absent fusion in other families (Shalankin, 11 May 2026). Here layer-wise geometry links an external behavioral bias to a specific representational timeline.

5. Interventions, causality, and verification

A distinctive feature of recent layer-wise geometric probing is the move from purely correlational readout to interventional analysis.

Subspace intervention makes the probe’s geometry itself manipulable. In self-supervised ViTs, the SVD of a converged linear probe isolates a low-rank subspace $h^{(l)} \mapsto \hat y$1, allowing aligned projections, random subspace controls, and residual-subspace tests. On NYU-Depth V2, DINOv2-L allocates 17.2%, 35.8%, 36.7%, and 10.3% of global spectral-weighted energy to layers $h^{(l)} \mapsto \hat y$2, respectively, showing strong middle-layer concentration; MAE and iBOT distribute energy more diffusely (Zhou et al., 2 Jul 2026).

Directional trajectory analysis makes stability itself a probe target. Geometric Evolution Maps define angular velocity

$h^{(l)} \mapsto \hat y$3

for residual-stream concept directions and extract probes at a handoff layer after the primary Concept Allocation Zone rather than at the peak separation layer. Across 391 concept$h^{(l)} \mapsto \hat y$4model pairs, the entry-to-exit cosine within CAZs has mean 0.233, while the handoff cosine with the final-layer direction has mean 0.942. Handoff-layer probes are at least as precise as peak-layer probes in 268 of 391 trials and strictly better in 259 of 391, with a pronounced architecture split between MHA and GQA models; random-direction controls show a median 377x suppression-rate advantage for concept-direction ablation (Henry, 25 May 2026).

At the level of a single direction, causal specificity can be measured directly. In the classification-trained ViT study, ablating the one-dimensional depth-probe direction increases depth MAE by 49–165% depending on layer, while ablating any random direction changes MAE by less than 1%. Targeted activation patching along that direction further shows that depth is partially re-derived at each layer rather than merely carried forward, and that interventions in mid layers have the strongest downstream persistence (Sanghavi, 25 Apr 2026).

Layer-wise geometric probing also appears in verification and acceleration. LayerCert exploits the nested hyperplane arrangement of a ReLU network by searching over partial activation patterns hierarchically, and under stated conditions provably reduces the number and size of convex programs relative to GeoCert (Lim et al., 2020). RegimeVGGT combines spectral, probing, and causal analyses to partition VGGT’s 24 global-attention layers into shallow, middle, and deep regimes, then uses that partition to design U-shaped token compression and K/V downsampling schedules, yielding a reported 6.7x speedup over VGGT* at matched reconstruction quality (You et al., 16 Jun 2026).

6. Limitations, misconceptions, and future directions

A central misconception is that probe performance directly reveals what a model uses. Multiple papers explicitly caution against this. Structural and psycholinguistic probes show what is linearly decodable, not what is causally used for prediction, and subject–verb agreement correlations or layerwise decoding curves remain correlational unless paired with intervention (Someya et al., 27 Jun 2025, Tikhomirova et al., 7 Jan 2026). The same caution appears in cross-model linguistic probing: differences in weight norms across layers can make scalar-mixing evaluations misleading, which is why minimum description length and RSA were proposed as alternatives to raw weight-mixing interpretations (Fayyaz et al., 2021).

Another misconception is that there is a universal “best layer.” The evidence is more qualified. Under frozen OOD probing of ViTs, intermediate layers and internal FFN activations are often best, but in ID settings after finetuning the final layer is best (Odonnat et al., 5 Mar 2026). Across BERToid models, ELECTRA tends to encode linguistic knowledge in deeper layers whereas XLNet concentrates it in earlier layers (Fayyaz et al., 2021). In self-supervised vision, DINOv2 aligns spatial features for efficient linear extraction, while MAE disperses geometric signal and requires broader spatial context; this objective dependence directly changes both layerwise peaks and low-rank structure (Zhou et al., 2 Jul 2026).

Benchmark scope remains limited in many studies. GeoSAE evaluates one brain MRI foundation model and one disease domain, and its $h^{(l)} \mapsto \hat y$5-NN graph construction is $h^{(l)} \mapsto \hat y$6 in principle; the paper explicitly notes uncertainty about transfer to other MRI FMs and conditions such as MS or stroke (Nerrise et al., 3 May 2026). Geometric deviation as a reliability signal works strongly in mathematical prompts and shows partial generalization to code, but no reliable geometric signal emerges for factual prompts, marking a clear boundary on generality (Du, 4 May 2026). GUIDE fixes a hand-designed mapping from sampled VGGT layers to the first six MLLM layers and does not impose explicit geometric supervision on intermediate LLM states (Wang et al., 7 Apr 2026).

Current directions point toward richer geometry, stronger controls, and broader domains. Suggested extensions include more global manifold tools such as diffusion maps and Riemannian metrics for brain-model probing (Nerrise et al., 3 May 2026), information-theoretic, geometric, and invariance measures at the module level for ViTs (Odonnat et al., 5 Mar 2026), multilingual and autoregressive variants of derivational probing (Someya et al., 27 Jun 2025), low-rank dense decoders informed by subspace-intervention results (Zhou et al., 2 Jul 2026), and more systematic behavioral ablation for residual-stream concept probes (Henry, 25 May 2026). A plausible implication is that the field is moving from static layer scoring toward explicit models of representational evolution, in which depth is analyzed not merely as an index but as a sequence of geometric transformations.