--- title: Linear Truth Probes Overview url: https://www.emergentmind.com/topics/linear-truth-probes type: topic --- # Linear Truth Probes Overview Searching arXiv for recent papers on linear truth probes and related truth-direction work. Linear truth probes are linear classifiers defined on hidden activations of large language models (LLMs) that seek a direction in activation space separating representations associated with true statements from those associated with false statements. In the formulation emphasized by Marks and Tegmark, a probe is a single direction in the model’s activation space that approximately separates true from false statements, with the basic score given by a projection of a hidden-state vector onto that direction [2310.06824]. Subsequent work has generalized the same paradigm to affine scoring functions of the form $s(x)=w^\top h(x)+b$, trained by logistic regression, linear SVM, mean-difference, or mass-mean procedures, and has studied their consistency across model scales, logical transformations, question answering, conversational formats, and task domains [2506.00823]. The resulting literature presents both positive and negative conclusions: in some sufficiently capable models and narrowly controlled settings, truthfulness behaves like a salient geometric feature; in other settings, the “geometry of truth” is layer-dependent, prompt-sensitive, context-sensitive, or task-specific rather than universal [2506.08572]. ## 1. Definition and formalism A linear truth probe operates on a hidden representation $h(x)\in\mathbb{R}^d$ extracted from a chosen layer and token position. The core hypothesis is that, in a sufficiently capable LLM, truthfulness is encoded as a roughly one-dimensional feature in the model’s high-dimensional activation space, so that there exists a truth direction $w\in\mathbb{R}^d$ and bias $b\in\mathbb{R}$ with score $$ s(x)=w^\top h(x)+b. $$ Passing $s(x)$ through a sigmoid yields a probabilistic prediction, and thresholding yields a binary prediction [2506.00823]. Several probe constructions recur across the literature. In Marks and Tegmark’s formulation, for a labeled dataset $\mathcal{D}$ of statements with labels $y\in\{+1,-1\}$ and hidden vectors $h(x)$, the difference-in-mean direction is $$ \mu_+ = \mathrm{mean}\{h(x)\mid y(x)=+1\},\qquad \mu_- = \mathrm{mean}\{h(x)\mid y(x)=-1\}, $$ $$ v_{\mathrm{naive}}=\mu_+ - \mu_-, $$ with score $s(x)=v_{\mathrm{naive}}^\top h(x)$ and prediction $\hat y(x)=\mathrm{sign}(s(x))$ [2310.06824]. The same work also introduces mass-mean probing, $$ w_{\mathrm{mm}}=\Sigma^{-1}(\mu_+ - \mu_-), $$ where $\Sigma$ is the covariance of the pooled, centered activations, followed by $p_{\mathrm{mm}}(x)=\sigma(w_{\mathrm{mm}}^\top h(x))$ [2310.06824]. Other studies use standard logistic regression or linear SVM. The logistic-regression form is $$ \hat y_i=\sigma(w^\top h_i+b), $$ optimized by binary cross-entropy, $$ \mathcal{L}(w,b)= - \sum_i \left[y_i\log \hat y_i + (1-y_i)\log(1-\hat y_i)\right], $$ or, equivalently, as an expectation over examples [2506.00823]. Linear SVM probes instead solve the primal margin objective $$ \min_{w,b,\xi}\ \frac12\|w\|^2 + C\sum_i \xi_i \quad\text{s.t.}\quad y_i(w^\top h_i+b)\ge 1-\xi_i,\ \xi_i\ge 0 $$ [2506.00823]. The same formal pattern appears in work on conversational lie detection, where the probe is written as $f(h)=\sigma(w^\top h+b)$ with $\ell_2$-regularized binary cross-entropy, and in work on strategic deception, where a logistic-regression probe is fit to residual-stream activations at a fixed layer [2505.09807] [2502.03407]. Across these settings, the operational content of a linear truth probe is stable: a hyperplane in activation space is used to classify hidden states as corresponding to true versus false, correct versus incorrect, or honest versus deceptive responses. ## 2. Early evidence for a linear truth direction The canonical evidence for linear truth probes was organized by Marks and Tegmark into three lines: visualization, transfer, and causal intervention [2310.06824]. First, visualization via PCA showed that hidden representations for curated true/false datasets such as world-city membership, Spanish–English translation, and numerical comparisons formed two clusters that were nearly linearly separable in the top two principal components [2310.06824]. The same study reported that once the first $d$ principal components were removed via projection, almost no further linear probe could distinguish truth, suggesting that nearly all linearly accessible truth information lay in the top principal components [2310.06824]. Second, transfer experiments were designed by training a probe on $80\%$ of one dataset, evaluating on the held-out $20\%$ IID test split, and then evaluating on whole other datasets. In the reported LLaMA-13B layer-13 residual-stream setting, a probe trained on numerical comparisons achieved $92\%+$ accuracy when tested on Spanish–English translation, and mass-mean probing generalized slightly better than vanilla logistic regression and unsupervised CCS [2310.06824]. Probes trained on true/false datasets also vastly outperformed probes trained on “likely” text, which was taken as evidence that the model was not merely separating plausible from implausible language [2310.06824]. Third, causal intervention experiments attempted to show that the truth direction was causally implicated in the model’s decision to output TRUE versus FALSE. Patching-based localization identified a small group of layers and tokens above which truth was computed, using the quantity $$ PD = P_{\mathrm{model}}(\text{“TRUE”}) - P_{\mathrm{model}}(\text{“FALSE”}). $$ Vector-addition interventions then modified hidden states by $$ h'(x;\ell,k)=h(x;\ell,k)+\alpha v, $$ where $v$ was a normalized probe direction applied at localized layer-token positions [2310.06824]. The intervention was summarized with the Normalized Indirect Effect, $$ NIE=\frac{PD^-_* - PD^-}{PD^+ - PD^-}, $$ where $NIE\approx 1$ indicates that the intervention makes a false statement look as true as a genuine true statement [2310.06824]. In LLaMA-13B, a mass-mean probe trained on cities+neg\_cities yielded $NIE\approx 0.95$ for false$\to$true and $NIE\approx 1.41$ for true$\to$false, whereas a logistic-regression probe on the same data achieved $NIE\approx 0.66/0.58$, and probes trained on “likely” text had almost no causal effect with $NIE\approx 0.01$ [2310.06824]. These results were interpreted as evidence that, at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements, and that simple difference-in-mean probes can recover directions that are at least as causally implicated as more complex alternatives [2310.06824]. ## 3. Probe families, geometry, and causal interpretation The literature distinguishes several families of truth probes, with different tradeoffs between simplicity, interpretability, and robustness. Difference-in-mean and mass-mean probes are described as zero-cost and as identifying a concrete direction $v$, unlike a full logistic-regression fit which “hides” the direction in a margin-maximizing ellipsoid [2310.06824]. Under Gaussian assumptions, mass-mean matches logistic regression’s average direction while retaining the interpretability of an explicit vector [2310.06824]. Alternative probe recovery procedures include CCS, CCR, logistic regression, linear SVM, and, in later work, orthogonal multi-direction constructions. CCS uses pairs of opposing sentences and an unsupervised consistency objective; CCR reflects paired representations across a learned hyperplane; mass-mean probing uses $w=\mu_+-\mu_-$; and supervised logistic regression learns $w$ by minimizing cross-entropy [2404.18865]. The presence of these methods reflects a methodological dispute over whether a truth direction is best viewed as a single separating direction, an affine hyperplane, or one element within a richer family of linearly decodable directions. Causal interpretation has remained central. In Marks and Tegmark, the vector-addition intervention was presented as evidence that the identified direction participates causally in generation decisions [2310.06824]. A related causal design appears in work on context-sensitive belief directions, where a premise embedding $h_Q$ is shifted along a belief direction, $$ h'_Q = h_Q + \alpha w, $$ and the resulting change in a later hypothesis representation is measured through the same direction [2404.18865]. That work reported maximal mean $\Delta p$ around $0.10$–$0.14$ in the best layers, and argued that belief directions are among the causal mediators in the inference process that incorporates in-context information [2404.18865]. The same paper also defined premise-effect and several normalized error scores, showing that probe outputs could respond correctly to entailment or contradiction while also being affected by irrelevant context [2404.18865]. A more intervention-oriented extension is “Truth Forest,” which replaces a single truth axis with multiple orthogonal probes per attention head. In that framework, the probe matrix $P=[\theta_1,\dots,\theta_k]$ is trained with cross-entropy plus an orthogonality penalty $$ L_{\mathrm{orth}}=\sum_{i