Hierarchical Probing in Multilevel Systems

Updated 7 July 2026

Hierarchical Probing is a method that examines multilevel structures by aligning probing techniques with nested hierarchies across models, planning systems, and numerical estimators.
It applies consistent probing strategies across various domains—from speech and language models to lattice QCD—revealing layer-specific, geometric, and taxonomic dependencies.
The approach stimulates methodological debates on the distinction between linear decodability of hierarchies and their functional roles, guiding future improvements in model interpretability.

Hierarchical probing denotes a family of techniques for interrogating multi-level structure in models, data, and decision systems. Across the literature, the term refers to probing that is explicitly aligned with a hierarchy: transformer depth in speech and LLMs, taxonomic levels in semantic representations, coarse intents and fine motion modes in interactive planning, recursively partitioned hypothesis spaces in RNA structure inference, or graph distances on toroidal lattices in stochastic trace estimation. The unifying feature is that the probe is not flat: it is defined over ordered levels, nested partitions, or multiresolution latent states, and its outputs are interpreted in relation to that structure rather than as isolated labels or scalar scores (Chiu et al., 9 Jan 2025, Chakravarty et al., 16 Feb 2026, Stathopoulos et al., 2013).

1. Conceptual scope and recurring structure

In its broadest sense, hierarchical probing studies whether a system encodes, exploits, or can be steered through a hierarchy. In representation learning, this usually means attaching a probe to multiple internal layers or projecting into a geometry suited to tree-like structure. In control and planning, it means maintaining and acting on beliefs at several levels of abstraction. In numerical linear algebra, it names a specific nested family of structured probing vectors whose effectiveness depends on graph distance. The term is therefore field-dependent, but the recurring pattern is a move from single-level readout to explicitly multilevel interrogation.

Domain	Hierarchy being probed	Probing mechanism
Speech SSL	Transformer depth and speaker-feature organization	Same MLP probe attached to every layer (Chiu et al., 9 Jan 2025)
Semantic taxonomies and LMs	Parent, ancestor, sibling, and depth-specific concept relations	Intrinsic hierarchy properties and depth-specific linear transformations (Lovón-Melgarejo et al., 2023, Sakata et al., 9 Apr 2026)
Vision models	ImageNet taxonomy across layers	Sparse autoencoders on DINOv2 activations (Olson et al., 21 May 2025)
Autonomous driving	Intent $\rightarrow$ mode hierarchy in human behavior	Hierarchical Bayes, entropy reduction, covariance steering, CVaR (Chakravarty et al., 16 Feb 2026)
Diffusion models	Latent compositional hierarchy in data	Forward–backward perturbation and correlation analysis (Sclocchi et al., 2024)
Lattice QCD	Distance hierarchy on toroidal lattice graphs	Nested Hadamard or coloring-based probing vectors (Stathopoulos et al., 2013, Papace et al., 29 May 2026, Frommer et al., 6 Jul 2026)

This diversity is substantive rather than terminological. In some areas, hierarchical probing is primarily an interpretability instrument; in others, it is part of the control law or the estimator itself. A plausible implication is that the phrase identifies a methodological template—probing aligned with multilevel structure—rather than a single algorithmic family.

2. Layer-wise and representation-centered probing

A canonical representation-learning formulation appears in speech self-supervised learning. In "Probing Speaker-specific Features in Speaker Representations" (Chiu et al., 9 Jan 2025), hierarchical probing means taking every internal layer of HuBERT, WavLM, and Wav2Vec 2.0, attaching the same small MLP probe to each layer, and testing whether the resulting utterance representation supports three-way classification of pitch, tempo, or energy. For SSL models, frame-level hidden states are average-pooled over time, then passed to a one-hidden-layer probe with ReLU and softmax; training uses cross-entropy, Adam with learning rate $10^{-3}$ , batch size 1024, and one epoch. The empirical pattern is explicitly hierarchical: lower layers capture raw acoustic information, intermediate layers are where pitch and energy accuracies peak, and tempo tends to peak somewhat deeper. Large models show clearer and higher peaks than Base models, and intermediate layers are described as capturing a mix of acoustic and para-linguistic information, with deeper layers refining these representations (Chiu et al., 9 Jan 2025).

A related but more explicitly semantic formulation probes taxonomic structure directly. "Probing Pretrained LLMs with Hierarchy Properties" (Lovón-Melgarejo et al., 2023) defines four relation types in a taxonomy—parent, ancestor, sibling, and far relative—and six intrinsic hierarchy properties such as P-A, P-S, and S-F. Each property is converted into a ternary $(x_n,x_l,x_r)$ satisfying a graph inequality $dist(x_n,x_l) < dist(x_n,x_r)$ , and the probe checks whether the embedding space satisfies the corresponding inequality

$d(\hat x_n,\hat x_l) < d(\hat x_n,\hat x_r).$

This is task-agnostic by construction: it does not ask whether a model can solve hypernym detection, but whether its geometry respects local taxonomic orderings. The paper reports that PLMs do not always capture hierarchical relations, and that hierarchy properties can be injected via triplet loss to improve understanding of hierarchy, with only moderate and non-systematic transfer across taxonomy reconstruction, hypernym discovery, and reading comprehension (Lovón-Melgarejo et al., 2023).

A more mechanistic language-model variant appears in "Linear Representations of Hierarchical Concepts in LLMs" (Sakata et al., 9 Apr 2026). There, linear transformations are trained for each hierarchical depth and semantic domain, extending Linear Relational Concepts to concept hierarchies such as $\text{Japan} \subset \text{Eastern Asia} \subset \text{Asia}$ . The analysis covers multi-token entities and cross-layer representations, shows that hierarchical relations can be linearly recovered from model representations within a domain, and finds that hierarchy is encoded in a relatively low-dimensional subspace whose structure is highly similar across domain-specific subspaces (Sakata et al., 9 Apr 2026). In vision, "Analyzing Hierarchical Structure in Vision Models with Sparse Autoencoders" (Olson et al., 21 May 2025) uses SAEs to analyze DINOv2 and reports that SAEs uncover hierarchical relationships in model activations, revealing an implicit encoding of ImageNet taxonomic structure and increasing information in the class token through each layer (Olson et al., 21 May 2025).

3. Geometric and dynamical formulations

Some hierarchical probes change not only what is measured, but the geometry in which the measurement is made. "Probing BERT in Hyperbolic Spaces" (Chen et al., 2021) introduces a Poincaré probe that projects contextualized embeddings into a low-dimensional Poincaré ball, then uses hyperbolic distance to recover dependency-tree distances and distance to the origin to recover depth. For syntax, the probe is trained so that squared hyperbolic distances approximate tree distances and squared hyperbolic norms approximate node depths. The motivation is explicit: trees and hierarchies are naturally accommodated by hyperbolic spaces, whose volume growth matches the combinatorics of branching structure. On contextualized embeddings, the paper reports that the hyperbolic probe better recovers tree structures than Euclidean probes and that the optimal curvature for the syntactic subspace is around $-1$ , suggesting that the geometry of BERT syntax may not necessarily be Euclidean (Chen et al., 2021).

A distinct dynamical formulation appears in diffusion models. "Probing the Latent Hierarchical Structure of Data via Diffusion Models" (Sclocchi et al., 2024) treats a trained diffusion model as a controlled perturbation device. Starting from a real sample $x_0$ , the method runs the forward process to $x_t$ , denoises back to $\hat x_0(t)$ , and measures how changes between $10^{-3}$ 0 and $10^{-3}$ 1 correlate across positions. In the Random Hierarchy Model, the paper predicts that forward–backward diffusion changes data by correlated chunks whose length scale diverges at a critical noise level. To measure this, it defines binary change variables $10^{-3}$ 2 for leaves or tokens and a dynamical susceptibility

$10^{-3}$ 3

then shows that $10^{-3}$ 4 peaks near the phase transition. The same phenomenology is reported for WikiText2 with a masked diffusion LLM and for ImageNet with DDPM plus CLIP patch embeddings: small noise produces isolated local changes, while a critical regime produces phrase-level or object-level coherent changes (Sclocchi et al., 2024). Here the hierarchy is probed not by a readout layer but by observing how controlled perturbations reorganize the sample at multiple scales.

4. Interactive, behavioral, and control-theoretic uses

In autonomous driving, hierarchical probing is not merely diagnostic; it is embedded in the planner. "Multimodal Covariance Steering in Belief Space with Active Probing and Influence for Autonomous Driving" (Chakravarty et al., 16 Feb 2026) builds a two-level hierarchy over human behavior with discrete intents $10^{-3}$ 5 and intent-specific motion modes $10^{-3}$ 6, each mode associated with a Gaussian target $10^{-3}$ 7. The joint belief is

$10^{-3}$ 8

and the ambiguity measure is a combined entropy over intents and modes. Probing is defined operationally as choosing control actions that both reduce expected entropy in this hierarchy and influence the human toward safer modes by minimizing a KL divergence between the ego trajectory distribution and a mode-specific Gaussian target. A runtime CVaR layer enforces $10^{-3}$ 9 for each mode before mode-wise actions are fused according to $(x_n,x_l,x_r)$ 0. The resulting architecture couples hierarchical Bayes, expected entropy reduction, covariance steering, and risk filtering. In 500 runs per scenario, the method reports higher success rates and shorter completion times than AP-IH, CC-MPC, and AP-MP in both lane merging and unsignaled intersection scenarios (Chakravarty et al., 16 Feb 2026).

A simpler behavioral variant appears in agent safety evaluation. "Evaluating LLM Agent Adherence to Hierarchical Safety Principles" (Potham, 3 Jun 2025) operationalizes hierarchy through prompt priority: a system prompt contains a visually marked "CORE SAFETY DIRECTIVE - HIGHEST PRIORITY," while the user prompt specifies a lower-level operational goal such as navigating to a green goal square in MiniGrid-Empty-6x6-v0. Conflict-unavoidable and conflict-avoidable scenarios are created for three principles, including "You must NEVER move onto a tile colored RED." Principle adherence is measured by Principle Adherence Rate, and task completion by Task Success Rate. The benchmark reports strong model dependence: $(x_n,x_l,x_r)$ 1 mini attains 100% PAR on all three principles, whereas some models show 0% or 30% adherence on the navigation-based principle. The study also notes a confound that is central to later discussion: high adherence can reflect incapability rather than principled compliance when the agent simply fails to manipulate relevant objects (Potham, 3 Jun 2025).

These control-oriented formulations broaden the meaning of hierarchical probing. The probe is no longer only a classifier attached to a frozen representation; it becomes a policy test or an action-selection mechanism that asks whether a system can reason and act in accordance with an imposed hierarchy.

5. Scientific and numerical meanings

In RNA secondary-structure inference, hierarchical probing appears as a recursive experimental design over a Boltzmann ensemble. "On an enhancement of RNA probing data using Information Theory" (Li et al., 2019) constructs an ensemble tree by recursively querying whether a base pair of maximum information entropy is contained in the target. Each query partitions the current ensemble into structures that contain or do not contain that base pair, and the next query is chosen within the resulting sub-ensemble. The framework adapts the Rényi–Ulam game to noisy structural experiments, uses modularity of RNA secondary structure to relate fragment probing to whole-sequence structure, and proves that maximum-entropy base-pair queries maximize information gain. For a Boltzmann ensemble incorporating probing data, the paper reports that the probability of correctly identifying the target in the leaf is greater than $(x_n,x_l,x_r)$ 2 (Li et al., 2019). This is hierarchical probing in a literal decision-tree sense: the hypothesis space is recursively partitioned until a low-entropy leaf contains a distinguished structure.

In lattice QCD and numerical linear algebra, hierarchical probing names a specific estimator construction. The original paper, "Hierarchical probing for estimating the trace of the matrix inverse on toroidal lattices" (Stathopoulos et al., 2013), addresses estimation of $(x_n,x_l,x_r)$ 3 for very large sparse matrices on periodic lattices. Classical Hutchinson estimation is unbiased but its variance is dominated by the off-diagonal entries of $(x_n,x_l,x_r)$ 4. Classical probing reduces this variance with distance- $(x_n,x_l,x_r)$ 5 graph colorings, but coloring cost becomes prohibitive and work cannot be reused as $(x_n,x_l,x_r)$ 6 increases. Hierarchical probing solves this by building nested distance- $(x_n,x_l,x_r)$ 7 colorings and corresponding Hadamard-based probing vectors on toroidal lattices. Because the colorings are nested, prior quadratures are reused when the probing distance is refined. The paper reports about an order-of-magnitude speedup over standard Monte Carlo on a large QCD configuration (Stathopoulos et al., 2013).

Later work reinterprets hierarchical probing as an important but improvable baseline. "Probing and graph coloring techniques for trace estimation in Lattice QCD" (Papace et al., 29 May 2026) shows that multiplier-based colorings can achieve valid distance- $(x_n,x_l,x_r)$ 8 colorings with significantly fewer colors than hierarchical probing and proves strictly lower variance than partial hierarchical probing at intermediate colorings for large enough $(x_n,x_l,x_r)$ 9. "Variance reduction with probing and Multilevel Monte Carlo in Lattice QCD" (Frommer et al., 6 Jul 2026) introduces a torus-based coloring that requires substantially fewer colors than hierarchical probing at the same distance and shows pronounced complementarity between graph-coloring-based probing and multigrid MLMC: for a connected pseudoscalar two-point function, multilevel decomposition gives variance reduction of up to $dist(x_n,x_l) < dist(x_n,x_r)$ 0 at large time separations, whereas for disconnected loops probing combined with dilution yields the substantial cost reduction (Papace et al., 29 May 2026, Frommer et al., 6 Jul 2026). In this literature, hierarchical probing is not metaphorical: it is a precise nested variance-reduction method defined on graph distance.

6. Methodological debates, misconceptions, and outlook

A central misconception is to treat successful probing as a direct explanation of model behavior. The strongest critique comes from syntax. "Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations" (Agarwal et al., 20 Jun 2025) evaluates 32 open-weight transformer models with structural probes, orthogonal structural probes, and head-word probes, then compares probe scores to BLiMP outcomes on hierarchy-sensitive syntactic phenomena. The result is a substantial disconnect: syntactic features extracted via probing fail to predict targeted syntactic performance across English phenomena. The paper’s conclusion is not that hierarchical syntax is absent from model representations, but that linearly decodable hierarchical structure is not sufficient evidence that the model functionally uses that structure in downstream behavior (Agarwal et al., 20 Jun 2025).

A second recurring issue concerns the relation between task-dependent and task-agnostic probing. The taxonomy paper argues that reliance on proxy tasks such as hypernymy detection can ignore ancestors, siblings, and more complex taxonomic relations, and its CTP result shows that optimizing for a hierarchy-adjacent task can degrade intrinsic hierarchical geometry (Lovón-Melgarejo et al., 2023). In speech, the use of a shallow MLP, one-epoch training, utterance-level averaging, and three-way discretization means that the probe reveals only linearly or mildly non-linearly decodable information at a coarse temporal scale; tempo, in particular, may be underestimated because frame structure is collapsed (Chiu et al., 9 Jan 2025). In safety benchmarking, high adherence rates can be inflated by incapability, so behavioral hierarchical probing must distinguish principled compliance from failure to act (Potham, 3 Jun 2025).

A third issue is that the term itself is polysemous. In some literatures it means layer-wise interpretability, in others active information gathering, and in others a nested family of estimator vectors. This does not make the term empty; rather, it locates a shared methodological intuition. Hierarchical probing asks whether one can expose or exploit a system’s multilevel organization by probing at matched levels of abstraction, geometry, or graph structure. The current trajectory of the literature suggests three durable directions: richer geometries for representational hierarchies, more causal tests linking probed structure to behavior, and tighter integration of hierarchical probes with decision and estimation procedures rather than treating them as post hoc diagnostics alone (Chen et al., 2021, Sclocchi et al., 2024, Chakravarty et al., 16 Feb 2026).