CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations

Abstract: In this paper, we propose CORTEX, a token-level hallucination detection method for Retrieval-Augmented Generation (RAG). In long-form RAG outputs, hallucinations often arise in localized spans rather than throughout an entire response. CORTEX therefore identifies ungrounded content at the token level, enabling fine-grained localization of hallucinations. The key intuition behind CORTEX is that tokens grounded in retrieved documents should be more strongly influenced by those documents than hallucinated tokens. To capture this document-induced effect, CORTEX compares internal representations of a LLM under two conditions: with and without the retrieved documents. Instead of relying solely on each token's immediate sensitivity to the retrieved documents, CORTEX also leverages the propagation of document-grounded information through preceding tokens, reducing false positives for tokens whose evidence has already been absorbed into the context. Finally, CORTEX applies post-processing smoothing step that models the tendency of hallucination labels to persist over contiguous spans, reducing local noise and encouraging span-consistent predictions. Experiments on two RAG benchmarks and three LLMs show that CORTEX substantially improves token-level hallucination detection, with each component consistently contributing to performance gains.

• The paper introduces a token-level hallucination detection framework for RAG by comparing internal representations with and without retrieved references.
• It employs attention-based contextual residuals to distinguish direct reference influence from inherited context in generated responses.
• Label-persistence smoothing enhances span consistency, improving detection performance metrics such as AP and AUROC in evaluations.

Token-Level Hallucination Detection in RAG Using Comparative Internal Representations: An Expert Essay on CORTEX (2606.31033)

Motivation and Problem Formulation

CORTEX is introduced as a post-hoc token-level hallucination detection framework tailored for Retrieval-Augmented Generation (RAG) with LLMs. RAG systems utilize external documents to improve factuality, but even with reference conditioning, LLMs can emit hallucinated content. Critically, hallucinations in long-form outputs often appear as local spans interleaved with grounded content. Existing detection techniques are insufficiently granular, operating primarily at sentence or answer level, which motivates CORTEX’s focus on token-level localization. The method is designed for practical deployment in scenarios involving closed-weight LLMs, leveraging an open-weight LLM only for post-hoc analysis.

Figure 1: Overview of CORTEX for token-level hallucination detection in RAG, using reference-conditioned internal representation comparisons, contextual residual features, and label-persistence smoothing.

Methodological Innovations

Comparative Internal Representation Delta

CORTEX’s central innovation is the computation of delta representations for each token, defined as the difference in internal states when the answer is analyzed with and without the retrieved references. This comparative approach distinguishes tokens that are genuinely influenced by references from those relying solely on parametric knowledge. The same token sequence is processed under both conditions, allowing direct subtraction:

$$\Delta h_i = h_i^{\mathrm{ref}} - h_i^{\mathrm{no-ref}}$$

This token-level delta reliably amplifies reference-induced signal, providing a discriminative feature for hallucination detection.

Attention-Based Contextual Residual

To address context propagation in RAG outputs—where facts grounded in references may be restated or reasoned upon—CORTEX models reference influence via an attention-weighted sum over preceding tokens. The contextual residual,

$$c_i = \Delta h_i - \sum_{j<i} \alpha_{ij}^{\mathrm{ref}} \Delta h_j$$

enables differentiation between direct reference grounding and indirect context-mediated grounding, reducing false positives for tokens that inherit reference support from earlier answer spans.

Label-Persistence Smoothing

Span-level annotation patterns in human labelling necessitate smoothing of raw token-level predictions. CORTEX implements a probabilistic label-persistence post-processing based on a binary smoothing variable, leveraging a self-loop parameter $p_{\mathrm{stay}}$ that controls the likelihood of contiguous spans with identical labels. This method improves span-wise consistency of hallucination scores and mitigates local prediction noise.

Empirical Results and Ablations

CORTEX was evaluated on RAGTruth and HalluRAG using several open-weight LLMs. It achieved dominant AP and AUROC scores across both token-level and answer-level tasks, surpassing direct representation- and embedding-based baselines.

Figure 2: Detection example for an answer containing hallucinated content. Label-persistence smoothing suppresses noisy token-level score patterns and highlights the hallucinated span more accurately and coherently.

Figure 3: Detection example for an answer without hallucinated content. The contextual residual c reduces false positives by accounting for reference influence mediated through the preceding answer context.

Extensive ablation studies reveal that each component—delta representation, contextual residual, and smoothing—contributes substantively to performance gains. Smoothing alone benefits even strong baselines, but the comparative delta remains the key performance discriminator.

Figure 4: Ablation case study with hallucination spans of different granularities. Without label-persistence smoothing, hallucination scores are more fragmented and localized. With label-persistence smoothing, nearby high-score regions are connected into broader span-consistent predictions, although this can also merge distinct hallucinated fragments into a wider continuous region.

Figure 5: Ablation case study involving groundless advice. The answer includes additional advice that may be informative but is not grounded in the provided references. CORTEX assigns high hallucination scores to this region, reflecting its role as a detector of reference support rather than a general factuality or usefulness judge.

Sensitivity Analysis

Sensitivity to the smoothing parameter $p_{\mathrm{stay}}$ demonstrates robust improvement across broad ranges, confirming stability and effectiveness of the smoothing module.

Figure 6: Sensitivity of token-level hallucination detection performance to $p_{\mathrm{stay}$.

Figure 7: Sensitivity of answer-level hallucination detection performance to $p_{\mathrm{stay}$.

Practical and Theoretical Implications

CORTEX is practically valuable for token-level reliability assessment in reference-grounded generation, especially in deployed RAG systems using closed-weight LLMs. Its architecture is agnostic to the generator, requiring only access to an open-weight LLM for feature extraction. The comparative representation analysis provides not only detection but potentially actionable reward signals for generator tuning, supporting future research in grounded response optimization.

The framework sets a new baseline for hallucination annotation granularity, reinforcing the necessity for token-level and span-level benchmarks. The explicit distinction between reference support and parametric knowledge flags the need for nuanced evaluation protocols in mixed or enterprise settings.

Limitations and Future Directions

CORTEX is specialized for reference-grounded RAG scenarios and does not judge parametric factuality. It is dependent on the capability of the open-weight LLM for accurate representation extraction. The reliance on annotation-rich datasets constrains its training scalability. Future directions include expanding supervision via weak annotation signals, refining context modeling for better management of indirect reference propagation, and integrating span-level signals for hybrid hallucination mitigation strategies.

Conclusion

CORTEX presents a technically rigorous and robust framework for fine-grained hallucination detection in RAG outputs, fundamentally grounded in comparative internal representation analysis. Through empirical evaluation, ablation, and sensitivity studies, the method establishes significant advantage over single-view representation approaches. Its practical architecture enables scalable deployment in settings with inaccessible generators, while its theoretical underpinnings invite further exploration in token-level reliability modeling and reward shaping paradigms within reference-grounded generation.