Hallucination Detection via Activations of Open-Weight Proxy Analyzers

Abstract: We introduce a proxy-analyzer framework for detecting hallucinations in LLMs. Instead of looking inside the generating model, our system reads already-generated text through a small locally hosted open-weight model and spots hallucinations using the reader's own internal activations. This works just as well when the generator is a closed API like GPT-4 as when it is any open-weight model. We built eighteen features grounded in how transformers process text, covering residual stream norms, per-head source-document attention, entropy, MLP activations, logit-lens trajectories, and three new token-level grounding statistics. We trained a stacking ensemble on 72,135 samples from five hallucination datasets. We tested across seven analyzer architectures from 0.5 billion to 9 billion parameters: Qwen2.5 at 0.5B and 7B, Gemma-2 at 2B and 9B, Pythia at 1.4B, and LLaMA-3 at both 3B and 8B. Across all seven, we consistently beat ReDeEP's token-level AUC of 0.73 on RAGTruth by 7.4 to 10.3 percentage points. Qwen2.5-7B reached an F1 of 0.717, just above ReDeEP's 0.713, while Qwen2.5-0.5B hit 0.706. The most striking finding is how tightly all seven models cluster: AUC spans only 2.3 percentage points across an eighteen-fold difference in model size. Even more surprising, our 3B LLaMA outperforms our 8B LLaMA on RAGTruth, showing that bigger is not always better even within the same model family. Both RAGTruth and LLM-AggreFact include outputs from multiple LLM families, so our results are not skewed toward any particular generator.

• The paper introduces a proxy-analyzer framework that classifies RAG faithfulness from 18 activation, attention, grounding, and lexical features collected in one forward pass of a small open-weight reader model.
• The approach outperforms ReDeEP across all seven analyzer architectures on RAGTruth, with Qwen2.5-7B reaching 0.834 AUC and 0.717 F1, while Qwen2.5-0.5B achieves competitive performance at substantially lower compute cost.
• The results show architecture and signal quality matter more than scale: performance varies by only 2.3 AUC points from 0.5B to 9B, LLaMA-3-3B beats LLaMA-3-8B in-distribution, and source-length shifts remain a key out-of-distribution limitation.

Overview

This paper introduces a proxy-analyzer framework for hallucination detection in retrieval-augmented generation (RAG) pipelines. Rather than instrumenting the generating model, the approach routes an already-generated answer through a small, locally hosted open-weight "reader" model and classifies faithfulness from the reader's internal activations. This decoupling removes the principal deployment constraint of white-box mechanistic methods such as ReDeEP (2605.07209), which require direct access to generator internals and are therefore unusable when the generator is a closed API, when backends are heterogeneous, or when re-running the generator for detection is prohibitively expensive.

The system extracts eighteen features per sample from a single forward pass via TransformerLens hooks: residual stream norms, per-head source-document attention, attention entropy, MLP output norms, logit-lens trajectories at 25/50/75/100% depth, conditional perplexity, interaction terms between parametric-memory and source-reading signals, Vectara HHEM-2.1 scores, lexical statistics, windowed means and slopes, and three token-level grounding statistics (minimum, variance, and slope of per-token grounding scores $\tau_i$). A supervised Attention Head Importance (AHI) score identifies which heads best separate faithful from hallucinated answers, replacing ReDeEP's heuristic copying-head identification with a data-driven weighting. Classification uses a stacking ensemble (Logistic Regression, Random Forest, HistGradientBoosting, XGBoost) trained on 72,135 samples from five datasets, with regime-specific temperature scaling and isotonic calibration.

Benchmark results

On RAGTruth — the primary benchmark, whose outputs come from six generators (GPT-4, GPT-3.5, Mistral-7B, Llama-2 at 7B/13B/70B) — every configuration of all seven analyzers beats both ReDeEP baselines on AUC, with gains of 6.5 to 10.3 percentage points over the token-level baseline of 0.733. Qwen2.5-7B with the RAGTruth-specialized classifier reaches AUC 0.834 and F1 0.717, exceeding ReDeEP's token-level F1 of 0.713; Gemma-2-9B attains the highest AUC at 0.836–0.837. Notably, Qwen2.5-0.5B achieves F1 0.700 raw, above ReDeEP's chunk-level threshold, at sub-billion scale.

The specialist RagtStacking classifier wins 35 of 36 head-to-head comparisons against the general Stacking model across seven architectures and six generators, confirming that distribution-specific training dominates generalist training even when the evaluation mixes generator families. On HaluEval, all models saturate near-perfect AUC ($\geq$0.997), indicating entity-substitution detection requires little capacity; MedHallu shows a modest scale benefit (0.952 for Pythia-1.4B to 0.990 for Qwen2.5-7B).

Scale saturation and the LLaMA inversion

The most consequential empirical finding is convergence across scale. Across an eighteen-fold parameter range (0.5B to 9B), best-to-worst spread is only 2.3 percentage points in RAGTruth AUC, 3.8 in F1, and 2.8 in out-of-distribution balanced accuracy. Within Qwen and Gemma families, scaling yields consistent but modest gains (~1 point). Within LLaMA, however, LLaMA-3-3B outperforms LLaMA-3-8B on RAGTruth AUC (0.824 vs. 0.819) and F1 (0.701 vs. 0.687), while the 8B model recovers only out-of-distribution (0.686 vs. 0.671 balanced accuracy). The authors attribute this to architectural factors rather than capacity: Qwen models concentrate discriminative source-grounding signal early (FIXED_WINDOW at 18–25% depth vs. 31–36% for LLaMA), exhibit higher AHI gains, and Gemma-2-9B's alternating local/global attention concentrates source attention in global layers, yielding fewer direction-flipped signals under distribution shift (6 vs. 9).

A practical implication follows directly: for production deployments where compute matters, Qwen2.5-0.5B delivers results within ~1.2 AUC points of the best model, and within the LLaMA family the 3B variant is preferable for RAGTruth detection despite using less than half the parameters of the 8B.

Circuit depth varies by task type

Per-layer analysis of Signal 2 reveals that the depth of maximal hallucination discriminability depends systematically on task type across all architectures: QA peaks deepest (67–86% of total depth), consistent with a late competition between source evidence and parametric memory in feed-forward layers, whereas claim verification peaks earlier (43–83%), since support comparison can be resolved once mid-network representations form. Within-family variation is also pronounced: LLaMA-3-8B resolves summarisation and data-to-text at 16% and 12% depth versus 64% and 43% for LLaMA-3-3B, plausibly because its greater head count settles structural lexical alignment in shallower layers. This supports the paper's broader claim that internal circuit organization, not parameter count, governs detection quality.

Out-of-distribution behavior

On LLM-AggreFact (held out entirely from training and calibration), all models trail MiniCheck-FT5 by 5.1 to 7.9 percentage points in balanced accuracy. The authors trace this gap to a concrete artifact: sources were truncated to 1,200 characters during training while AggreFact sources average ~3,000, producing a Kolmogorov-Smirnov shift of ~0.30 in Signal 7 and flipping Signals 4 and 10 in direction. Six signals remain stable across all seven models — AHI, S8, S17, S14, S15, S13 — all measuring internal state independently of source length. The count of flipped signals predicts out-of-distribution performance across models. The proposed remedy (retraining at 2,000 characters with FEVER and VitaminC) is asserted to close most of the gap but remains unvalidated in this work.

Limitations

The paper concedes several constraints explicitly. The source-length truncation directly causes the AggreFact deficit described above. Per-generator analysis relies on synthetic groupings of 377 rows because processed RAGTruth does not retain generator labels, so per-generator numbers should be read as directional rather than exact. The FIXED_WINDOW is tuned on RAGTruth training data and is likely suboptimal for summarisation and data-to-text, whose optimal depths vary considerably across architectures; a task-adaptive window is identified as needed. Finally, the claim that retraining will reduce the OOD gap to one to three percentage points is a projection, not a demonstrated result.

Conclusion

This work demonstrates that hallucination detection can be performed entirely outside the generating model, using eighteen mechanistically grounded activation features read from a small open-weight proxy. It surpasses ReDeEP's AUC on RAGTruth across seven analyzer architectures without any generator access, with Qwen2.5-7B first exceeding ReDeEP's token-level F1 among proxy approaches. The central empirical findings — a 2.3-point AUC band across an eighteen-fold size range, and a 3B LLaMA beating its 8B sibling in-distribution — indicate that architecture and signal quality, not scale, set the detection ceiling. Open questions include whether task-adaptive layer windows generalize beyond RAGTruth, and whether the projected closure of the long-source OOD gap holds empirically.