---
title: Concept Probing in Neural Models
url: https://www.emergentmind.com/topics/concept-probing
type: topic
---

# Concept Probing in Neural Models

Searching arXiv for recent papers on concept probing and related methods to ground the article with up-to-date references.
Concept probing is the study of whether internal representations of a neural model encode a human-interpretable concept and, if so, where that concept is located, how stably it is represented, and whether it can support diagnosis or intervention. In its standard form, a probe is an auxiliary classifier trained on frozen activations \(f_l(x)\), \(h^{(\ell)}(x)\), or related latent states to predict a concept label from a chosen layer, often with a linear score such as \(s = w^\top h + b\) or a multiclass softmax readout [2410.00153]. Across contemporary work, the term now covers a broader family of procedures: layer-wise probing of concepts at different depths, estimation of concept subspaces rather than single directions, evaluation of containment and disentanglement, uncertainty-aware Bayesian probing, feature-set overlap analyses in SAEs, and causal interventions on latent states or units [2404.07066] [2507.18681] [2605.01381].

## 1. Formal foundations and object of study

A canonical formulation treats a trained model \(f\) as producing a representation \(h^{(l)} = f_l(x)\) or \(z = f_l(x)\) at layer \(l\), and defines a probe \(g\) that predicts a concept label \(c\) from that representation [2507.18681]. In binary settings, the probe is commonly logistic regression or a linear SVM, with scores of the form \(z \cdot v + b\) or \(w^\top h + b\), while multiclass settings use \(\hat{y} = \mathrm{softmax}(W h^{(l)} + b)\) or \(p_\ell(y \mid x) = \mathrm{softmax}(W_\ell h_\ell(x) + b_\ell)\) [2410.00153] [2604.11802]. In CNN concept analysis, the same idea extends to spatial feature maps \(F \in \mathbb{R}^{C \times H \times W}\), where linear weights define Concept Activation Vectors and spatial attribution maps [2511.04312].

The concept itself is not restricted to a single ontology. In different studies it denotes low-level visual attributes such as “striped” or “wooden,” linguistic properties such as tense or dependency relations, psychological constructs such as the Big Five domains, cross-script semantic invariants in Serbian digraphia, omission states in multimodal diffusion transformers, or knowledge structures defined by taxonomies, properties, and contexts [2507.18550] [2110.08388] [2604.11802] [2603.08869] [2605.14270] [2211.04079]. This breadth has encouraged a more abstract view in which a concept is represented not necessarily by a single neuron or direction, but by a subspace, a distribution over directions, or a set of sparse features.

Several works also sharpen the distinction between probing and adjacent methods. Probing trains a small readout on frozen activations to test decodability; finetuning changes the base model; feature visualization synthesizes or ranks stimuli; causal intervention edits internal activations to test necessity or sufficiency [2604.11802]. This distinction matters because the central interpretive question is not only whether a label can be decoded, but whether the decoded signal corresponds to the intended concept rather than to correlated structure.

## 2. Depth, layer selection, and representational geometry

A recurrent result is that probe performance depends strongly on where the model is probed. One line of work formalizes layer selection as a trade-off between informativeness and regularity. For a dataset \(D_l = \{(f_l(x_i), c_i)\}_{i=1}^n\), informativeness is measured by the uncertainty coefficient \(U(\mathbf{c} \mid f_l(\mathbf{x})) = I(f_l(\mathbf{x}); \mathbf{c}) / H(\mathbf{c})\), regularity by the 5-fold cross-validated accuracy of a simple logistic regression probe, normalized as \(\widetilde{R}^{(l)} = \frac{K R^{(l)} - 1}{K - 1}\), and the layer score is
\[
S^{(l)} = \lambda U(\mathbf{c} \mid f_l(\mathbf{x})) + (1-\lambda)\frac{K R(\mathbf{c} \mid f_l(\mathbf{x})) - 1}{K - 1}.
\]
The selected layer is \(l^* = \arg\max_l S^{(l)}\) [2507.18681]. On the reported image benchmarks, this selector achieves an overall average test accuracy of \(90.2\%\), compared with \(77.7\%\) for averaging across layers and \(91.4\%\) for an oracle chosen by test accuracy [2507.18681].

A complementary depth-oriented formulation is “Concept Depth,” which characterizes where a concept becomes linearly accessible and where probing accuracy stabilizes. Using layer-wise accuracies \(\alpha_l\), the variation rate is \(\beta_i = \alpha_i / \alpha_{i-1}\); the jump point is \(J(M,D) = \min\{i/d : \beta_i \ge 1.10\}\); the converging point is \(C(M,D) = \max\{i/d : |\beta_i - 1| < 0.03\}\); and the peak depth is \(P(M,D) = \arg\max_l \alpha_l\) [2404.07066]. Across Gemma, LLaMA, and QWen families, simpler factual and emotional tasks are often probed successfully in shallow or mid layers, whereas more complex inferential tasks such as StrategyQA and Coinflip peak later and often show bell-shaped curves [2404.07066]. The same study reports that prefix noise shifts learning curves rightward and that 8-bit quantization slows convergence, while 16-bit quantization has negligible impact in the tested settings [2404.07066].

Later work argues that fixed-layer or peak-separation probing can still miss the relevant representation because concept directions rotate during their assembly phase. Geometric Evolution Maps define a Concept Allocation Zone (CAZ), measure angular velocity \(\omega(l) = 1 - |d_l \cdot d_{l-1}|\), and identify a handoff layer
\[
L_H = \min(L_{CAZ\_end}+1, N-1)
\]
at which directional rotation ceases [2605.25848]. Across 23 architectures and 17 concept types, the mean entry-to-exit cosine within CAZ is \(0.233\), with \(93.9\%\) of pairs below \(0.5\), while the handoff-to-final cosine averages \(0.942\) [2605.25848]. In 391 concept-model pairs, GEM-extracted probes are at least as precise as peak-layer probes in \(268/391\) trials and strictly better in \(259/391\) [2605.25848]. This suggests that concept localization is often a geometric trajectory problem rather than a simple late-layer selection problem.

## 3. From single directions to concept subspaces and uncertainty

A major refinement of the standard probe replaces the single concept vector by a distribution over probe solutions. In Gaussian Concept Subspace, a concept \(c\) at layer \(\ell\) is modeled by repeatedly training logistic probes on bootstrap datasets and fitting a Gaussian over the resulting weight vectors:
\[
w \sim \mathcal{N}(\mu_c^\ell, \Sigma_c^\ell).
\]
In the reported implementation, \(\Sigma_c^\ell\) is diagonal, estimated from \(M=1000\) observed probes trained on bootstraps of \(1000\) positive and \(1000\) negative examples drawn from a \(5000/5000\) concept dataset [2410.00153]. This yields an induced score distribution
\[
E[s \mid h] = \mu^\top h + b, \qquad \operatorname{Var}(s \mid h) = h^\top \Sigma h,
\]
so concept presence is associated not only with a mean score but also with uncertainty [2410.00153].

Empirically, sampled \(1\sigma\) vectors have internal cosine similarity \(>0.93\), observed vectors vary more at approximately \(0.80\)–\(0.90\), and observed-sampled cosine is typically \(0.88\)–\(0.93\) across Llama-2-7B, Gemma-7B, and Llama-2-13B [2410.00153]. Sampled vectors at \(1\sigma\) and \(2\sigma\) often match or exceed observed probes’ accuracy, whereas vectors at \(3\sigma\)–\(5\sigma\) degrade, indicating that proximity to \(\mu\) correlates with representational fidelity [2410.00153]. The same framework also produces plausible inter-concept geometry across 16 concepts and four categories, with category blocks visible in cosine heatmaps and PCA projections [2410.00153].

A different uncertainty-oriented line uses Gaussian Process Probes. GPP defines a Bayesian distribution over concept classifiers, with a logistic transform of a Gaussian-process latent and explicit quantities for judged probability, aleatory uncertainty, and epistemic uncertainty [2305.18213]. In the reported experiments, GPP can probe concepts with as few as 10 observations, measure both epistemic and aleatory uncertainty, and use posterior variance as an OOD signal performing as well as classic methods do on the tested image settings [2305.18213]. Concept probing in this sense becomes a posterior over concept functions rather than a single discriminant boundary.

A broader subspace perspective is formalized by the axes of containment and disentanglement. Given a candidate concept subspace \(S\), containment is evaluated by retention \(R\) on \(S\) and leakage \(L\) on its complement; disentanglement with respect to another concept \(\mathcal{Y}'\) is evaluated by purity \(P\) on \(S\) and interference \(I\) after erasure [2605.01381]. Across five estimators—MLR, LDA, CPCA, COV, and LEACE—the reported results show that the estimator materially affects containment and disentanglement properties, that LEACE performs well on both axes in-distribution, and that it still struggles to generalize to unseen data, especially for speaker information in HuBERT [2605.01381]. The same study emphasizes that concept subspaces may not be uniquely determined, so strong retention alone does not identify a unique geometric locus [2605.01381].

## 4. Reliability, alignment, data dependence, and theoretical critique

A persistent criticism is that high probe accuracy is not equivalent to faithful concept alignment. In visual concept probing, deliberately misaligned FP-CAVs trained only on false positives achieve \(74\%\) accuracy versus \(81\%\) for standard classifier probes over 148 concepts, and the mean cosine similarity between FP-CAVs and classifier CAVs is \(0.62\) [2511.04312]. This is direct evidence that probes can exploit spurious correlations while retaining high discriminative accuracy. To address this, alignment-oriented work introduces spatial linear attribution, Concept Localization Maps, Segmentation-CAVs trained with pixel-wise logistic loss, hard test sets with randomized backgrounds, and augmentation robustness metrics [2511.04312]. Segmentation-CAVs obtain the highest spatial alignment score \(S_c \approx 0.23\), classifier probes score approximately \(0.16\), and translation-invariant variants improve both robustness and localization [2511.04312].

A deeper theoretical critique shows that probe-based detection and probe-guided removal can be unreliable even under favorable assumptions. In the analyzed text setting, if the representation decomposes as \(z = [z_m, z_p]\), where \(z_p\) are concept-causal features and \(z_m\) are main-task features, then max-margin probes can still use \(z_m\) when spurious correlations align on margin points [2207.04153]. The paper then proves that post-hoc null-space projection methods such as INLP can either mix and irreversibly damage both \(z_m\) and \(z_p\) or remove task features while leaving concept features intact, and that repeated projections can drive the representation toward zero [2207.04153]. Adversarial removal is also shown to admit undesired encoders that preserve concept dependence while satisfying the adversarial objective [2207.04153]. For fairness-sensitive settings, the proposed response is a spuriousness metric \(\psi(f)\) defined on minority-group accuracy rather than reliance on probe accuracy alone [2207.04153].

The data used to train probes is itself a major source of variation. On image-classification settings, relevant concepts reach approximately \(97.3\%\) of maximal measured accuracy with only 250 training samples and typically stabilize around 200 samples [2507.18550]. Reusing base-model training data in probe training has negligible effect on probe accuracy, with a fitted slope of \(-4.1 \times 10^{-3}\), whereas symmetric label noise up to about \(20\%\) yields an average relative reduction of \(9.3\%\), and real CUB attribute noise with \(18.2\%\) mislabels yields a larger \(13.1\%\) reduction [2507.18550]. This suggests that label quality is often more consequential than data provenance.

A Bayesian reinterpretation reframes the whole enterprise as measuring inductive bias rather than raw decodability. In this view, the object of interest is the maximal Bayesian model evidence \(p(\mathbf{y}\mid \mathbf{x}, R, P)\) over a family of probes \(P\) for a representation \(R\), rather than the accuracy of a fixed probe [2110.08388]. The reported empirical result is that random representations consistently have the lowest evidence, and that fastText can offer a better inductive bias than BERT for some token-level tasks, while BERT is uniformly better for dependency arc labeling and T5 has the best inductive bias among the tested sentence-level representations [2110.08388]. Collectively, these studies argue that concept probing is informative only when accompanied by explicit alignment, generalization, and spuriousness analyses.

## 5. Modalities, benchmarks, and alternative probing paradigms

Although linear probing remains the default, recent work extends concept probing into several alternative paradigms. In SAE-based interpretability, Serbian digraphia provides a controlled test of whether sparse features track abstract semantics above tokenization. In Gemma-3 models from 270M to 27B parameters, identical Serbian sentences written in Latin and Cyrillic scripts—sharing no tokens under the tokenizer—have Jaccard similarities of approximately \(0.58\)–\(0.59\), while cross-script random pairs are approximately \(0.28\) and cross-language random pairs approximately \(0.19\) [2603.08869]. Cross-script similarity increases from \(0.501\) at 270M to \(0.649\) at 27B, while cross-script random decreases from \(0.421\) to \(0.211\) [2603.08869]. This provides evidence that SAE feature activations can be script-invariant and meaning-oriented.

At the output-space level, DecompressionLM reframes probing as zero-shot concept graph extraction rather than supervised concept decoding. It uses Van der Corput low-discrepancy sequences together with arithmetic decoding to generate deterministic, stateless, embarrassingly parallel concept lists and graphs from instruction-tuned LMs [2602.00377]. Across two model families and five quantization variants, activation-aware AWQ-4bit expands concept coverage by \(30\)–\(170\%\), whereas GPTQ-Int4 induces a \(71\)–\(86\%\) coverage collapse, and these differences are not reliably reflected by explanation-level perplexity [2602.00377]. In a US Law grounding study, top-5 and bottom-5 MMLU-Pro Law models differ by a 17-point hallucination gap [2602.00377]. This is concept probing in an extraction sense: it measures breadth and grounding of surfaced concepts without pre-specified relations.

Benchmark construction has also become a significant part of the field. COPEN defines three conceptual knowledge tasks—Conceptual Similarity Judgment, Conceptual Property Judgment, and Conceptualization in Contexts—covering 24k instances and 393 concepts [2211.04079]. Human performance is \(79.5\%\) on CSJ, \(91.4\%\) and \(91.2\%\) on CPJ instance-level and chain-level, and \(85.6\%\) on CiC, while PLMs remain far below human performance even with fine-tuning [2211.04079]. The benchmark was explicitly designed to test whether PLMs organize entities by conceptual similarities, learn conceptual properties including transitivity, and conceptualize entities in contexts [2211.04079]. The findings emphasize that conceptual knowledge probing is not reducible to factual cloze completion.

The notion of “concept” also varies by domain. In multimodal diffusion transformers, the target concept can be an omission signal encoded in text-side key vectors; in psychological probing it can be one of the Big Five domains; in speech it can be phone or speaker identity; in visual interpretability it can be a supervised concept token with both existence and mask [2605.14270] [2604.11802] [2605.01381] [2509.22015]. This diversity has pushed the field away from a single operational definition and toward a family of tasks that share a common question: whether an internal representation supports a stable, aligned, and actionable encoding of a human-defined concept.

## 6. Causal interventions and practical uses

A central development is the move from diagnostic probing to intervention. In GCS-based representation intervention, Llama-2-7B-chat is steered to produce joyful movie reviews by modifying last-token hidden states using
\[
h'^\ell = (1-a) h^\ell + a v_c^\ell,
\]
where \(v_c^\ell\) is a concept vector from mean difference, a single linear probe, or GCS sampling [2410.00153]. On the reported task, single linear vectors reach Joyfulness \(3.020\) at \(a=0.069\) but can suffer coherence \(6.780\) at \(a=0.074\), whereas GCS \(1\sigma\) samples reach Joyfulness \(2.980\) at \(a=0.064\) with coherence in the \(4.857\)–\(5.800\) range across strengths [2410.00153]. The practical claim is not that GCS dominates on peak score, but that vectors sampled near \(\mu\) balance steering strength and fluency more reliably.

Unit-level causal probing of Big Five concepts in LLMs makes a similar distinction between representational and behavioral control. In Meta-Llama-3-8B-Instruct, Big Five information becomes rapidly decodable in early layers and remains detectable through final layers, while concept-selective MLP units are concentrated in layers approximately 6–11 and 16–19 [2604.11802]. Boost-and-suppress interventions on the top 30% concept-selective units at layer 16 drive targeted success rates above \(0.8\) for some concepts in probe space, but generation control is weaker, with maximum targeted shifts up to roughly \(50\%\) and frequent spillover, especially toward Extraversion [2604.11802]. This establishes a gap between representational steering and label-generation control.

In multimodal diffusion transformers, Omission Signal Intervention uses logistic probes trained on per-head concept-token key vectors to identify an omission direction \(\theta^{(l,h)}\), then updates text-side keys during early denoising by
\[
\bar{k}_c^{(t,l,h)} \leftarrow \bar{k}_c^{(t,l,h)} + \alpha \sigma^{(l,h)} \theta^{(l,h)}.
\]
On FLUX.1-Dev, average object-omission accuracy improves from \(0.47\) to \(0.61\), and on SD3.5-Medium from \(0.55\) to \(0.64\); for FLUX six-object prompts, accuracy rises from \(0.18\) to \(0.40\) [2605.14270]. Attribute-binding scores also improve, for example FLUX shape from \(0.4995\) to \(0.5819\) and texture from \(0.6419\) to \(0.7039\) [2605.14270]. Here the probe is not merely descriptive: it furnishes a steering direction that changes downstream generation.

Vision work based on Concept-SAE takes the same causal turn but grounds interventions in semantically supervised concept tokens. A Concept Tokenizer predicts a concept score \(s_i\) and mask \(m_i\), a Concept Aggregator reconstructs the hidden state, and interventions clamp selected concept scores before reconstructing a counterfactual latent [2509.22015]. Localization Ratio on ResNet-18 shallow layers reaches \(1.472\), \(1.402\), and \(1.395\) for layers 5, 7, and 9, versus \(1.019\), \(0.982\), and \(1.002\) for a CEM baseline [2509.22015]. The same framework uses entropy and Jensen–Shannon distance of concept-score distributions to localize adversarial vulnerability, and targeted finetuning of identified layers produces adversarial accuracy gains such as \(+17.62\%\) on CelebA ViT layer 0 and \(+28.33\%\) on CelebA ResNet-18 layer 17 [2509.22015]. This suggests that faithful concept tokens can function as causal handles for both behavioral correction and robustness diagnosis.

Taken together, these intervention studies imply a narrowing but still incomplete bridge between concept readout and mechanistic control. Probing can expose actionable directions, units, or tokens, but success depends on geometric stability, alignment with the intended concept, and the degree to which downstream computation is distributed rather than localized.

## 7. Open problems and research directions

Several unresolved problems recur across the literature. First, concept representations are often non-unique. High-retention subspaces can differ dramatically in leakage and disentanglement, and high probe accuracy can coexist with misalignment, as shown by FP-CAVs and by estimator-dependent subspace properties [2511.04312] [2605.01381]. Second, many successful methods still rely on simplifying assumptions—diagonal covariance in GCS, empirically set thresholds such as \(\epsilon = 0.05\) in GEM, or specific SAE thresholds and layer choices in script-invariance analysis [2410.00153] [2605.25848] [2603.08869].

Third, probing does not by itself establish causal use. This point is explicit in layer-selection work, in the theoretical critique of probe-guided removal, and in the Big Five experiments where latent readouts are readily steered but label generation is not [2507.18681] [2207.04153] [2604.11802]. A plausible implication is that future concept probing will continue to combine diagnostic probing with intervention, mediation, or counterfactual evaluation rather than treating separability as sufficient evidence.

Fourth, generalization remains fragile. LEACE’s guarantees do not automatically transfer to unseen data, speaker information is difficult to contain in compact subspaces, and cross-script SAE results are currently tied to a uniquely clean digraphic setting [2605.01381] [2603.08869]. Benchmark design therefore remains central: COPEN’s disjoint concept splits, hard background-randomized sets for alignment studies, and per-object balancing in omission probing all reflect an effort to separate genuine concept encoding from shortcut exploitation [2211.04079] [2511.04312] [2605.14270].

Collectively, these works suggest that concept probing has evolved from a narrow question—whether a linear classifier can decode a label from a hidden state—into a broader methodology for locating, validating, and sometimes manipulating concept representations. Its current frontier is not decodability alone, but the joint study of stability, uncertainty, subspace geometry, disentanglement, and causal efficacy across text, vision, speech, and multimodal generation [2410.00153] [2605.25848] [2605.01381].

Source: https://www.emergentmind.com/topics/concept-probing