Concept Probing in Neural Models

• Concept probing is the study of whether neural network activations encode human-defined, interpretable concepts using auxiliary classifiers.
• It employs layer-wise linear probes, uncertainty quantification, and geometric subspace analysis to assess where and how concepts are represented.
• Research in concept probing informs model diagnostics and causal interventions by highlighting trade-offs in layer selection, data quality, and representational alignment.

Searching arXiv for papers on concept probing and related methods to ground the article with up-to-date references. Concept probing is the study of whether internal representations of a neural model encode a human-interpretable concept and, if so, where that concept is located, how stably it is represented, and whether it can support diagnosis or intervention. In its standard form, a probe is an auxiliary classifier trained on frozen activations $f_l(x)$, $h^{(\ell)}(x)$, or related latent states to predict a concept label from a chosen layer, often with a linear score such as $s = w^\top h + b$ or a multiclass softmax readout (Zhao et al., 2024). Across contemporary work, the term now covers a broader family of procedures: layer-wise probing of concepts at different depths, estimation of concept subspaces rather than single directions, evaluation of containment and disentanglement, uncertainty-aware Bayesian probing, feature-set overlap analyses in SAEs, and causal interventions on latent states or units (Jin et al., 2024, Ribeiro et al., 24 Jul 2025, Naowarat et al., 2 May 2026).

1. Formal foundations and object of study

A canonical formulation treats a trained model $f$ as producing a representation $h^{(l)} = f_l(x)$ or $z = f_l(x)$ at layer $l$, and defines a probe $g$ that predicts a concept label $c$ from that representation (Ribeiro et al., 24 Jul 2025). In binary settings, the probe is commonly logistic regression or a linear SVM, with scores of the form $z \cdot v + b$ or $h^{(\ell)}(x)$0, while multiclass settings use $h^{(\ell)}(x)$1 or $h^{(\ell)}(x)$2 (Zhao et al., 2024, Harada et al., 13 Apr 2026). In CNN concept analysis, the same idea extends to spatial feature maps $h^{(\ell)}(x)$3, where linear weights define Concept Activation Vectors and spatial attribution maps (Lysnæs-Larsen et al., 6 Nov 2025).

The concept itself is not restricted to a single ontology. In different studies it denotes low-level visual attributes such as “striped” or “wooden,” linguistic properties such as tense or dependency relations, psychological constructs such as the Big Five domains, cross-script semantic invariants in Serbian digraphia, omission states in multimodal diffusion transformers, or knowledge structures defined by taxonomies, properties, and contexts (Ribeiro et al., 24 Jul 2025, Immer et al., 2021, Harada et al., 13 Apr 2026, Karne, 9 Mar 2026, Baek et al., 14 May 2026, Peng et al., 2022). This breadth has encouraged a more abstract view in which a concept is represented not necessarily by a single neuron or direction, but by a subspace, a distribution over directions, or a set of sparse features.

Several works also sharpen the distinction between probing and adjacent methods. Probing trains a small readout on frozen activations to test decodability; finetuning changes the base model; feature visualization synthesizes or ranks stimuli; causal intervention edits internal activations to test necessity or sufficiency (Harada et al., 13 Apr 2026). This distinction matters because the central interpretive question is not only whether a label can be decoded, but whether the decoded signal corresponds to the intended concept rather than to correlated structure.

2. Depth, layer selection, and representational geometry

A recurrent result is that probe performance depends strongly on where the model is probed. One line of work formalizes layer selection as a trade-off between informativeness and regularity. For a dataset $h^{(\ell)}(x)$4, informativeness is measured by the uncertainty coefficient $h^{(\ell)}(x)$5, regularity by the 5-fold cross-validated accuracy of a simple logistic regression probe, normalized as $h^{(\ell)}(x)$6, and the layer score is

$h^{(\ell)}(x)$7

The selected layer is $h^{(\ell)}(x)$8 (Ribeiro et al., 24 Jul 2025). On the reported image benchmarks, this selector achieves an overall average test accuracy of $h^{(\ell)}(x)$9, compared with $s = w^\top h + b$0 for averaging across layers and $s = w^\top h + b$1 for an oracle chosen by test accuracy (Ribeiro et al., 24 Jul 2025).

A complementary depth-oriented formulation is “Concept Depth,” which characterizes where a concept becomes linearly accessible and where probing accuracy stabilizes. Using layer-wise accuracies $s = w^\top h + b$2, the variation rate is $s = w^\top h + b$3; the jump point is $s = w^\top h + b$4; the converging point is $s = w^\top h + b$5; and the peak depth is $s = w^\top h + b$6 (Jin et al., 2024). Across Gemma, LLaMA, and QWen families, simpler factual and emotional tasks are often probed successfully in shallow or mid layers, whereas more complex inferential tasks such as StrategyQA and Coinflip peak later and often show bell-shaped curves (Jin et al., 2024). The same study reports that prefix noise shifts learning curves rightward and that 8-bit quantization slows convergence, while 16-bit quantization has negligible impact in the tested settings (Jin et al., 2024).

Later work argues that fixed-layer or peak-separation probing can still miss the relevant representation because concept directions rotate during their assembly phase. Geometric Evolution Maps define a Concept Allocation Zone (CAZ), measure angular velocity $s = w^\top h + b$7, and identify a handoff layer

$s = w^\top h + b$8

at which directional rotation ceases (Henry, 25 May 2026). Across 23 architectures and 17 concept types, the mean entry-to-exit cosine within CAZ is $s = w^\top h + b$9, with $f$0 of pairs below $f$1, while the handoff-to-final cosine averages $f$2 (Henry, 25 May 2026). In 391 concept-model pairs, GEM-extracted probes are at least as precise as peak-layer probes in $f$3 trials and strictly better in $f$4 (Henry, 25 May 2026). This suggests that concept localization is often a geometric trajectory problem rather than a simple late-layer selection problem.

3. From single directions to concept subspaces and uncertainty

A major refinement of the standard probe replaces the single concept vector by a distribution over probe solutions. In Gaussian Concept Subspace, a concept $f$5 at layer $f$6 is modeled by repeatedly training logistic probes on bootstrap datasets and fitting a Gaussian over the resulting weight vectors: $f$7 In the reported implementation, $f$8 is diagonal, estimated from $f$9 observed probes trained on bootstraps of $h^{(l)} = f_l(x)$0 positive and $h^{(l)} = f_l(x)$1 negative examples drawn from a $h^{(l)} = f_l(x)$2 concept dataset (Zhao et al., 2024). This yields an induced score distribution

$h^{(l)} = f_l(x)$3

so concept presence is associated not only with a mean score but also with uncertainty (Zhao et al., 2024).

Empirically, sampled $h^{(l)} = f_l(x)$4 vectors have internal cosine similarity $h^{(l)} = f_l(x)$5, observed vectors vary more at approximately $h^{(l)} = f_l(x)$6–$h^{(l)} = f_l(x)$7, and observed-sampled cosine is typically $h^{(l)} = f_l(x)$8–$h^{(l)} = f_l(x)$9 across Llama-2-7B, Gemma-7B, and Llama-2-13B (Zhao et al., 2024). Sampled vectors at $z = f_l(x)$0 and $z = f_l(x)$1 often match or exceed observed probes’ accuracy, whereas vectors at $z = f_l(x)$2–$z = f_l(x)$3 degrade, indicating that proximity to $z = f_l(x)$4 correlates with representational fidelity (Zhao et al., 2024). The same framework also produces plausible inter-concept geometry across 16 concepts and four categories, with category blocks visible in cosine heatmaps and PCA projections (Zhao et al., 2024).

A different uncertainty-oriented line uses Gaussian Process Probes. GPP defines a Bayesian distribution over concept classifiers, with a logistic transform of a Gaussian-process latent and explicit quantities for judged probability, aleatory uncertainty, and epistemic uncertainty (Wang et al., 2023). In the reported experiments, GPP can probe concepts with as few as 10 observations, measure both epistemic and aleatory uncertainty, and use posterior variance as an OOD signal performing as well as classic methods do on the tested image settings (Wang et al., 2023). Concept probing in this sense becomes a posterior over concept functions rather than a single discriminant boundary.

A broader subspace perspective is formalized by the axes of containment and disentanglement. Given a candidate concept subspace $z = f_l(x)$5, containment is evaluated by retention $z = f_l(x)$6 on $z = f_l(x)$7 and leakage $z = f_l(x)$8 on its complement; disentanglement with respect to another concept $z = f_l(x)$9 is evaluated by purity $l$0 on $l$1 and interference $l$2 after erasure (Naowarat et al., 2 May 2026). Across five estimators—MLR, LDA, CPCA, COV, and LEACE—the reported results show that the estimator materially affects containment and disentanglement properties, that LEACE performs well on both axes in-distribution, and that it still struggles to generalize to unseen data, especially for speaker information in HuBERT (Naowarat et al., 2 May 2026). The same study emphasizes that concept subspaces may not be uniquely determined, so strong retention alone does not identify a unique geometric locus (Naowarat et al., 2 May 2026).

4. Reliability, alignment, data dependence, and theoretical critique

A persistent criticism is that high probe accuracy is not equivalent to faithful concept alignment. In visual concept probing, deliberately misaligned FP-CAVs trained only on false positives achieve $l$3 accuracy versus $l$4 for standard classifier probes over 148 concepts, and the mean cosine similarity between FP-CAVs and classifier CAVs is $l$5 (Lysnæs-Larsen et al., 6 Nov 2025). This is direct evidence that probes can exploit spurious correlations while retaining high discriminative accuracy. To address this, alignment-oriented work introduces spatial linear attribution, Concept Localization Maps, Segmentation-CAVs trained with pixel-wise logistic loss, hard test sets with randomized backgrounds, and augmentation robustness metrics (Lysnæs-Larsen et al., 6 Nov 2025). Segmentation-CAVs obtain the highest spatial alignment score $l$6, classifier probes score approximately $l$7, and translation-invariant variants improve both robustness and localization (Lysnæs-Larsen et al., 6 Nov 2025).

A deeper theoretical critique shows that probe-based detection and probe-guided removal can be unreliable even under favorable assumptions. In the analyzed text setting, if the representation decomposes as $l$8, where $l$9 are concept-causal features and $g$0 are main-task features, then max-margin probes can still use $g$1 when spurious correlations align on margin points (Kumar et al., 2022). The paper then proves that post-hoc null-space projection methods such as INLP can either mix and irreversibly damage both $g$2 and $g$3 or remove task features while leaving concept features intact, and that repeated projections can drive the representation toward zero (Kumar et al., 2022). Adversarial removal is also shown to admit undesired encoders that preserve concept dependence while satisfying the adversarial objective (Kumar et al., 2022). For fairness-sensitive settings, the proposed response is a spuriousness metric $g$4 defined on minority-group accuracy rather than reliance on probe accuracy alone (Kumar et al., 2022).

The data used to train probes is itself a major source of variation. On image-classification settings, relevant concepts reach approximately $g$5 of maximal measured accuracy with only 250 training samples and typically stabilize around 200 samples (Ribeiro et al., 24 Jul 2025). Reusing base-model training data in probe training has negligible effect on probe accuracy, with a fitted slope of $g$6, whereas symmetric label noise up to about $g$7 yields an average relative reduction of $g$8, and real CUB attribute noise with $g$9 mislabels yields a larger $c$0 reduction (Ribeiro et al., 24 Jul 2025). This suggests that label quality is often more consequential than data provenance.

A Bayesian reinterpretation reframes the whole enterprise as measuring inductive bias rather than raw decodability. In this view, the object of interest is the maximal Bayesian model evidence $c$1 over a family of probes $c$2 for a representation $c$3, rather than the accuracy of a fixed probe (Immer et al., 2021). The reported empirical result is that random representations consistently have the lowest evidence, and that fastText can offer a better inductive bias than BERT for some token-level tasks, while BERT is uniformly better for dependency arc labeling and T5 has the best inductive bias among the tested sentence-level representations (Immer et al., 2021). Collectively, these studies argue that concept probing is informative only when accompanied by explicit alignment, generalization, and spuriousness analyses.

5. Modalities, benchmarks, and alternative probing paradigms

Although linear probing remains the default, recent work extends concept probing into several alternative paradigms. In SAE-based interpretability, Serbian digraphia provides a controlled test of whether sparse features track abstract semantics above tokenization. In Gemma-3 models from 270M to 27B parameters, identical Serbian sentences written in Latin and Cyrillic scripts—sharing no tokens under the tokenizer—have Jaccard similarities of approximately $c$4–$c$5, while cross-script random pairs are approximately $c$6 and cross-language random pairs approximately $c$7 (Karne, 9 Mar 2026). Cross-script similarity increases from $c$8 at 270M to $c$9 at 27B, while cross-script random decreases from $z \cdot v + b$0 to $z \cdot v + b$1 (Karne, 9 Mar 2026). This provides evidence that SAE feature activations can be script-invariant and meaning-oriented.

At the output-space level, DecompressionLM reframes probing as zero-shot concept graph extraction rather than supervised concept decoding. It uses Van der Corput low-discrepancy sequences together with arithmetic decoding to generate deterministic, stateless, embarrassingly parallel concept lists and graphs from instruction-tuned LMs (Hong et al., 30 Jan 2026). Across two model families and five quantization variants, activation-aware AWQ-4bit expands concept coverage by $z \cdot v + b$2–$z \cdot v + b$3, whereas GPTQ-Int4 induces a $z \cdot v + b$4–$z \cdot v + b$5 coverage collapse, and these differences are not reliably reflected by explanation-level perplexity (Hong et al., 30 Jan 2026). In a US Law grounding study, top-5 and bottom-5 MMLU-Pro Law models differ by a 17-point hallucination gap (Hong et al., 30 Jan 2026). This is concept probing in an extraction sense: it measures breadth and grounding of surfaced concepts without pre-specified relations.

Benchmark construction has also become a significant part of the field. COPEN defines three conceptual knowledge tasks—Conceptual Similarity Judgment, Conceptual Property Judgment, and Conceptualization in Contexts—covering 24k instances and 393 concepts (Peng et al., 2022). Human performance is $z \cdot v + b$6 on CSJ, $z \cdot v + b$7 and $z \cdot v + b$8 on CPJ instance-level and chain-level, and $z \cdot v + b$9 on CiC, while PLMs remain far below human performance even with fine-tuning (Peng et al., 2022). The benchmark was explicitly designed to test whether PLMs organize entities by conceptual similarities, learn conceptual properties including transitivity, and conceptualize entities in contexts (Peng et al., 2022). The findings emphasize that conceptual knowledge probing is not reducible to factual cloze completion.

The notion of “concept” also varies by domain. In multimodal diffusion transformers, the target concept can be an omission signal encoded in text-side key vectors; in psychological probing it can be one of the Big Five domains; in speech it can be phone or speaker identity; in visual interpretability it can be a supervised concept token with both existence and mask (Baek et al., 14 May 2026, Harada et al., 13 Apr 2026, Naowarat et al., 2 May 2026, Ding et al., 26 Sep 2025). This diversity has pushed the field away from a single operational definition and toward a family of tasks that share a common question: whether an internal representation supports a stable, aligned, and actionable encoding of a human-defined concept.

6. Causal interventions and practical uses

A central development is the move from diagnostic probing to intervention. In GCS-based representation intervention, Llama-2-7B-chat is steered to produce joyful movie reviews by modifying last-token hidden states using

$h^{(\ell)}(x)$00

where $h^{(\ell)}(x)$01 is a concept vector from mean difference, a single linear probe, or GCS sampling (Zhao et al., 2024). On the reported task, single linear vectors reach Joyfulness $h^{(\ell)}(x)$02 at $h^{(\ell)}(x)$03 but can suffer coherence $h^{(\ell)}(x)$04 at $h^{(\ell)}(x)$05, whereas GCS $h^{(\ell)}(x)$06 samples reach Joyfulness $h^{(\ell)}(x)$07 at $h^{(\ell)}(x)$08 with coherence in the $h^{(\ell)}(x)$09–$h^{(\ell)}(x)$10 range across strengths (Zhao et al., 2024). The practical claim is not that GCS dominates on peak score, but that vectors sampled near $h^{(\ell)}(x)$11 balance steering strength and fluency more reliably.

Unit-level causal probing of Big Five concepts in LLMs makes a similar distinction between representational and behavioral control. In Meta-Llama-3-8B-Instruct, Big Five information becomes rapidly decodable in early layers and remains detectable through final layers, while concept-selective MLP units are concentrated in layers approximately 6–11 and 16–19 (Harada et al., 13 Apr 2026). Boost-and-suppress interventions on the top 30% concept-selective units at layer 16 drive targeted success rates above $h^{(\ell)}(x)$12 for some concepts in probe space, but generation control is weaker, with maximum targeted shifts up to roughly $h^{(\ell)}(x)$13 and frequent spillover, especially toward Extraversion (Harada et al., 13 Apr 2026). This establishes a gap between representational steering and label-generation control.

In multimodal diffusion transformers, Omission Signal Intervention uses logistic probes trained on per-head concept-token key vectors to identify an omission direction $h^{(\ell)}(x)$14, then updates text-side keys during early denoising by

$h^{(\ell)}(x)$15

On FLUX.1-Dev, average object-omission accuracy improves from $h^{(\ell)}(x)$16 to $h^{(\ell)}(x)$17, and on SD3.5-Medium from $h^{(\ell)}(x)$18 to $h^{(\ell)}(x)$19; for FLUX six-object prompts, accuracy rises from $h^{(\ell)}(x)$20 to $h^{(\ell)}(x)$21 (Baek et al., 14 May 2026). Attribute-binding scores also improve, for example FLUX shape from $h^{(\ell)}(x)$22 to $h^{(\ell)}(x)$23 and texture from $h^{(\ell)}(x)$24 to $h^{(\ell)}(x)$25 (Baek et al., 14 May 2026). Here the probe is not merely descriptive: it furnishes a steering direction that changes downstream generation.

Vision work based on Concept-SAE takes the same causal turn but grounds interventions in semantically supervised concept tokens. A Concept Tokenizer predicts a concept score $h^{(\ell)}(x)$26 and mask $h^{(\ell)}(x)$27, a Concept Aggregator reconstructs the hidden state, and interventions clamp selected concept scores before reconstructing a counterfactual latent (Ding et al., 26 Sep 2025). Localization Ratio on ResNet-18 shallow layers reaches $h^{(\ell)}(x)$28, $h^{(\ell)}(x)$29, and $h^{(\ell)}(x)$30 for layers 5, 7, and 9, versus $h^{(\ell)}(x)$31, $h^{(\ell)}(x)$32, and $h^{(\ell)}(x)$33 for a CEM baseline (Ding et al., 26 Sep 2025). The same framework uses entropy and Jensen–Shannon distance of concept-score distributions to localize adversarial vulnerability, and targeted finetuning of identified layers produces adversarial accuracy gains such as $h^{(\ell)}(x)$34 on CelebA ViT layer 0 and $h^{(\ell)}(x)$35 on CelebA ResNet-18 layer 17 (Ding et al., 26 Sep 2025). This suggests that faithful concept tokens can function as causal handles for both behavioral correction and robustness diagnosis.

Taken together, these intervention studies imply a narrowing but still incomplete bridge between concept readout and mechanistic control. Probing can expose actionable directions, units, or tokens, but success depends on geometric stability, alignment with the intended concept, and the degree to which downstream computation is distributed rather than localized.

7. Open problems and research directions

Several unresolved problems recur across the literature. First, concept representations are often non-unique. High-retention subspaces can differ dramatically in leakage and disentanglement, and high probe accuracy can coexist with misalignment, as shown by FP-CAVs and by estimator-dependent subspace properties (Lysnæs-Larsen et al., 6 Nov 2025, Naowarat et al., 2 May 2026). Second, many successful methods still rely on simplifying assumptions—diagonal covariance in GCS, empirically set thresholds such as $h^{(\ell)}(x)$36 in GEM, or specific SAE thresholds and layer choices in script-invariance analysis (Zhao et al., 2024, Henry, 25 May 2026, Karne, 9 Mar 2026).

Third, probing does not by itself establish causal use. This point is explicit in layer-selection work, in the theoretical critique of probe-guided removal, and in the Big Five experiments where latent readouts are readily steered but label generation is not (Ribeiro et al., 24 Jul 2025, Kumar et al., 2022, Harada et al., 13 Apr 2026). A plausible implication is that future concept probing will continue to combine diagnostic probing with intervention, mediation, or counterfactual evaluation rather than treating separability as sufficient evidence.

Fourth, generalization remains fragile. LEACE’s guarantees do not automatically transfer to unseen data, speaker information is difficult to contain in compact subspaces, and cross-script SAE results are currently tied to a uniquely clean digraphic setting (Naowarat et al., 2 May 2026, Karne, 9 Mar 2026). Benchmark design therefore remains central: COPEN’s disjoint concept splits, hard background-randomized sets for alignment studies, and per-object balancing in omission probing all reflect an effort to separate genuine concept encoding from shortcut exploitation (Peng et al., 2022, Lysnæs-Larsen et al., 6 Nov 2025, Baek et al., 14 May 2026).

Collectively, these works suggest that concept probing has evolved from a narrow question—whether a linear classifier can decode a label from a hidden state—into a broader methodology for locating, validating, and sometimes manipulating concept representations. Its current frontier is not decodability alone, but the joint study of stability, uncertainty, subspace geometry, disentanglement, and causal efficacy across text, vision, speech, and multimodal generation (Zhao et al., 2024, Henry, 25 May 2026, Naowarat et al., 2 May 2026).