The Trilemma of Truth in Large Language Models
This presentation examines a fundamental challenge in AI transparency: determining whether a language model internally treats a statement as true, false, or simply unknown. The authors introduce sAwMIL, a novel probing method that challenges five common assumptions in veracity detection and demonstrates that models maintain distinct internal representations for knowledge they affirm, knowledge they reject, and content they have never encountered. Through experiments across 16 models and three factual domains, the work reveals that truth-tracking signals exist deep within model architectures and can be manipulated to steer outputs toward more grounded responses.Script
When a language model generates a statement, is it drawing on internal knowledge, contradicting what it knows, or simply making something up? The authors of this paper argue that existing methods for detecting model veracity collapse a three-way distinction into a false binary.
Existing probes rest on five flawed assumptions. They treat truth and falsehood as symmetric opposites, assume models retain all human knowledge, mistake classifier scores for calibrated probabilities, force every statement into a binary true-or-false judgment, and search for veracity signals only at the final token.
The authors introduce sAwMIL: Sparse Aware Multiple Instance Learning. Instead of one bidirectional probe, it trains three separate linear classifiers—one detecting true signals, one detecting false signals, and one detecting neither. Each probe scans across all token positions to locate where the veracity signal actually lives, then uses conformal prediction to abstain when evidence is insufficient.
Across 48 model-dataset combinations spanning 16 language models, sAwMIL achieved weighted MCC scores above 70% in all but one case. Zero-shot prompting collapsed on default models and proved brittle to prompt wording. The mean-difference baseline misclassified synthetic unknown entities as true or false roughly 93% of the time, while sAwMIL correctly identified them as neither.
Veracity signals do not concentrate in the final layers. The strongest representations appear between 50% and 75% of model depth, often in different layers for true versus false. When the authors added or subtracted learned truth directions from these middle-layer activations, they shifted output probabilities in predictable directions roughly 80% of the time, and chat models showed tighter coupling between where the probe succeeded and where interventions worked.
Models encode veracity differently depending on their training. Chat models cluster together in prediction space and generalize better across domains, while default models and Qwen variants form distinct groups. This suggests that alignment procedures don't just change output behavior but fundamentally reshape how models internally represent what they know, what they reject, and what remains outside their knowledge. You can explore this paper further and create your own AI-narrated video at EmergentMind.com.
