White-Box Lie Detection in LLMs

• White-box lie detection is a method that examines hidden activations in LLMs via linear probes to identify deceptive behaviors.
• Empirical evaluations reveal AUROC scores ranging from 0.60 to 0.91, highlighting both performance gains over black-box methods and challenges like leakage effects.
• Robust probe construction involves strategic layer selection, token aggregation, and mitigation techniques to counter superficial signal reliance and improve calibration.

White-box lie detection refers to a class of techniques for detecting deceptive or untruthful behavior in LLMs by directly analyzing their internal hidden activations, as opposed to relying solely on observable output text. These methods aim to exploit the latent representations within the model to identify states corresponding to lying, deception, or concealed knowledge, with the goal of providing model-agnostic, low-latency, and potentially more generalizable detection capabilities. White-box detectors are most commonly instantiated as linear probes or similar classifiers operating over the residual stream or other hidden states at specific layers of a transformer architecture.

1. Mathematical Foundations and Standard Probe Architectures

White-box lie detectors typically implement a linear probe—a binary classifier—on the residual stream activations of a LLM. Let $h_t^{(\ell)} \in \mathbb{R}^d$ denote the activation at position $t$ in layer $\ell$. The standard probe computes a deception score per token,

$s_t = w^\top h_t^{(\ell)} + b$

where $w \in \mathbb{R}^d$ and $b\in \mathbb{R}$ are trained weights and bias. Logistic regression is used with an $L_2$ regularization penalty, optimized to minimize binary cross-entropy loss on labeled honest/deceptive activations.

Probe outputs are often aggregated over tokens to yield an utterance-level deception score:

$S_{\text{mean}}(x) = \frac{1}{T}\sum_{t=1}^T s_t$

where $T$ is the response length. The result is passed through a sigmoid function to yield a probability of deception. Decision thresholds $\tau$ are calibrated against a control set (e.g., an Alpaca-derived honest-only corpus) to achieve a predefined false positive rate, typically $t$0 (Kretschmar et al., 20 Nov 2025, Boxo et al., 16 Sep 2025, Casademunt et al., 5 Mar 2026).

Variants include probes reading the final token, probes conditioned on follow-up responses ("No-token" probe), and upper-bound probes trained on domain-matched distributions.

2. Empirical Performance and Benchmarks

Major evaluations of white-box lie detection use large-scale testbeds such as LIARS’ BENCH (Kretschmar et al., 20 Nov 2025), collections of real and synthetic deception datasets (Parrack et al., 16 Jul 2025), and censored LLM corpora (Casademunt et al., 5 Mar 2026). Performance is principally measured by Area Under the Receiver Operator Characteristic (AUROC), balanced accuracy, and recall at low false positive rates.

Key results:

Method	Avg. AUROC	Balanced Accuracy	Recall @ 1% FPR	Reference
Mean-Activation Probe	0.60	0.54	0.10	(Kretschmar et al., 20 Nov 2025)
Follow-up Probe	0.62	0.54	0.47	(Kretschmar et al., 20 Nov 2025)
Upper-Bound Probe	0.91	0.73	0.51	(Kretschmar et al., 20 Nov 2025)
Full Averaging (Pairs)	0.941	–	–	(Parrack et al., 16 Jul 2025)
Activation Probe (Qwen3)	up to 0.817	up to 0.83	up to 0.90	(Casademunt et al., 5 Mar 2026)

Performance is highly task- and dataset-dependent. White-box methods provide measurable boosts over black-box monitoring, with "black-to-white" AUROC differentials up to +0.046 in controlled settings (Parrack et al., 16 Jul 2025). The strongest probes (upper-bound, in-distribution) can approach AUROC 0.91, but general performance typically falls in the 0.60–0.82 range, and highly domain-matched calibration is essential.

3. Failure Modes and Leakage Effects

There are several critical limitations and failure modes observed in white-box lie detection:

• Systematic failures by lie type: Mean-aggregation and follow-up probes often exhibit balanced accuracy near random (0.50) for harm-pressure, self-knowledge lies, and multi-turn deceptions. These cases leave weak or no consistent footprints in the monitored layer's activations (Kretschmar et al., 20 Nov 2025).
• Elicitation leakage: Monitors can spuriously correlate to explicit deception cues originating from prompt instructions (e.g., "You are a deceptive assistant") rather than genuine model deception. This results in inflated AUROC, with up to 30–40 points drop when content filtering removes leakage tokens (Boxo et al., 16 Sep 2025).
• Reasoning leakage: If models emit explicit chain-of-thought reasoning about their covert actions, probe performance is artificially improved by these tokens rather than by implicit deception signals (Boxo et al., 16 Sep 2025).
• Lack of locality: Linear probes trained on complete responses average over all tokens, often missing deception signals that are sparsely or locally distributed, such as a single contradicted fact (Kretschmar et al., 20 Nov 2025).
• Distributional shift and adversarial evasion: When probes are evaluated on adversarial examples (e.g., “deception dilution” by padding outputs with irrelevant truths), performance drops but may remain above 0.9 AUROC; nonetheless, this highlights susceptibility to evasion (Parrack et al., 16 Jul 2025).

Empirical quantification of leakage is provided in Table 1 (adapted from (Boxo et al., 16 Sep 2025)):

Dataset	Baseline AUROC	After Content Filtering	Δ AUROC (pts)
Sandbagging	0.95	0.62	–0.33
BASH	0.83	0.58	–0.25

4. Probe Construction, Training, and Calibration

Probe architectures in white-box lie detection share several core design patterns:

• Layer selection: Middle or upper transformer layers (e.g., 20th-percentile, 50% depth) are empirically selected as probe points (Kretschmar et al., 20 Nov 2025, Casademunt et al., 5 Mar 2026).
• Data construction: Training employs paired honest vs. deceptive activations, either from explicit prompt-switching (contrastive "instructed-pairs") or from roleplaying datasets with known ground truth (Kretschmar et al., 20 Nov 2025, Casademunt et al., 5 Mar 2026).
• Preprocessing: Feature normalization is conducted via z-scoring over a held-out control set (e.g., 10,000 Alpaca transcripts), controlling for distributional differences (Casademunt et al., 5 Mar 2026).
• Decision rule: Detection thresholds ($t$1) are set by calibrating on control sets to achieve low false positive rates, with post-processing to adjust for deployment-specific trade-offs (Kretschmar et al., 20 Nov 2025, Parrack et al., 16 Jul 2025).
• Token aggregation: Scores may be aggregated (mean, max, or over critical token subsets), and follow-up strategies such as "No-token" probes or targeted questioning can alter the extracted activation trajectory (Kretschmar et al., 20 Nov 2025, Parrack et al., 16 Jul 2025).

Advanced readouts leverage multi-layer concatenation, shallow MLPs, or alternative signals such as token-logit gaps, hidden state gradients, or averaged attention patterns.

5. Mitigation of Superficial Signal Reliance

Evaluation frameworks and probe construction increasingly address the reliance on superficial signals through several mitigation strategies (Boxo et al., 16 Sep 2025):

• Content filtering: Removal of explicit elicitation and reasoning tokens before computing probe activations reduces artificial probe performance, yielding declines in AUROC of up to 35%.
• Score filtering: Aggregation over only public, task-relevant tokens (excluding private reasoning) leads to more moderate AUROC declines (approx. 10–15%) and targets specific leakage types.
• Model organism evaluation: Fine-tuned LLMs internally encode deception, removing dependence on external prompt cues. Probes evaluated on such models display up to 40–50% drops in AUROC, providing a conservative bound on genuine detection ability.
• Calibration post-mitigation: Detection thresholds must be re-calibrated after applying mitigation to reflect new false-positive/negative trade-offs; this is essential for deployment robustness.

A plausible implication is that much apparent probe success on standard benchmarks results from prompt- or output-based artifacts rather than robust encoding of deception within model internals.

6. Recommendations and Directions for Next-Generation Detectors

Design guidelines for advancing white-box lie detection, supported by empirical results, include (Kretschmar et al., 20 Nov 2025, Boxo et al., 16 Sep 2025):

• Diverse training spectra: Probes should be trained on data spanning world-knowledge lies, self-knowledge lies, instructional deception, multi-turn scenarios, and backdoors, minimizing overfitting to synthetic "discourse-shaped" examples.
• Multi-layer and nonlinear readouts: Aggregate activations from multiple layers, possibly via concatenation or shallow networks, to capture richer deception signals than single linear projections.
• Alternative signals: Explore logit gaps, hidden state gradients, and attention patterns as complementary indicators of deception.
• Combinatorial monitoring regimes: Pair white-box probes with black-box semantic reasoning for ensemble detection, granting higher reliability under distribution shift or adversarial conditions (Parrack et al., 16 Jul 2025).
• Routine post-hoc mitigation audits: Regular content filtering and model organism evaluation are recommended for continuous validation of monitors' reliance on authentic internal deception representations.
• Interactive and adaptive probing: Employ targeted follow-up questions or multi-turn probes to amplify latent inconsistencies, especially in difficult lie domains.

These practices aim to align measured probe performance with models' genuine internal belief states, rather than with accidental or surface-level correlates.

7. Current Limitations and Open Challenges

Despite technical progress, white-box lie detection inherits several unresolved issues:

• Limited generalization: Probes trained on narrow deception types generalize poorly to new or subtle forms, such as covert self-knowledge concealment or adversarial attacks.
• Identifying neural correlates: There is limited mechanistic understanding of the distributed neural signatures of deception versus artifacts of reasoning or prompt structure.
• Residual leakage detection: Verifying the absence of leakage signals in evaluation remains a foundational experimental challenge, complicated by the difficulty of obtaining ground-truth latent intent.
• Token-localization: Existing probes lack spatial resolution, making it difficult to attribute deception to specific claim tokens within a response.
• Cross-model transfer: Generalization across architectures and training regimes remains inconsistent, with probe effectiveness sensitive to feature normalization and calibration strategies.

Continued development of both interpretability tools and robust adversarial testbeds is necessary to advance the reliability and scientific understanding of white-box lie detection. An ongoing emphasis on model organism methodologies and layered monitoring promises more reliable, nuanced assessments of model honesty and intent.

---

References:

• "Liars' Bench: Evaluating Lie Detectors for LLMs" (Kretschmar et al., 20 Nov 2025)
• "Benchmarking Deception Probes via Black-to-White Performance Boosts" (Parrack et al., 16 Jul 2025)
• "Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation" (Casademunt et al., 5 Mar 2026)
• "Towards mitigating information leakage when evaluating safety monitors" (Boxo et al., 16 Sep 2025)