HalluScan: Benchmark for LLM Hallucinations

Updated 4 July 2026

HalluScan is a benchmark framework that evaluates hallucinations in instruction-following LLMs using a tripartite taxonomy and a composite HalluScore.
It employs Adaptive Detection Routing (ADR) to dynamically select the optimal detection method based on risk estimates, significantly reducing costs with minimal accuracy loss.
The framework performs an error cascade analysis across domains to diagnose hallucination types, guiding targeted mitigation strategies.

Searching arXiv for HalluScan and closely related hallucination benchmarks/detectors to ground the article. HalluScan is a benchmark framework for detecting and mitigating hallucinations in instruction-following LLMs. It is designed to evaluate factual incorrectness, unfaithfulness to provided context, and misalignment with user instructions under a unified experimental protocol spanning 72 configurations across 6 detection methods, 4 open-weight model families, and 3 domains. Its central components are a tripartite hallucination taxonomy, the composite metric HalluScore, the routing algorithm Adaptive Detection Routing (ADR), and an “error cascade” analysis that decomposes hallucinations by type and domain (Cherif, 4 May 2026).

1. Taxonomy and problem formulation

HalluScan adopts a tripartite taxonomy of hallucinations. Factual Hallucination is defined as the generation of statements that contradict established world knowledge or verifiable facts, such as inventing a historical date or a nonexistent entity. Faithfulness Hallucination is defined as divergence or contradiction of the model’s output with respect to its provided input context, for example unsupported claims in a summary. Instruction (Misalignment) Hallucination is defined as failure to follow explicit user instructions, such as producing the wrong format, ignoring length constraints, or omitting required components (Cherif, 4 May 2026).

The taxonomy is operationalized through distinct detection challenges. Factual hallucination requires external knowledge or retrieval to verify claims. Faithfulness hallucination is best addressed via Natural Language Inference (NLI) between source text as premise and generated claims as hypotheses. Instruction misalignment requires modeling prompt constraints and checking output compliance. This division matters because HalluScan does not treat hallucination as a single undifferentiated failure mode; instead, it treats error type as a determinant of both detector choice and downstream mitigation strategy (Cherif, 4 May 2026).

A plausible implication is that HalluScan frames hallucination management as a structured evaluation problem rather than only a generation-time alignment problem. The taxonomy supports detector comparison, metric design, and domain-level error analysis within a common vocabulary.

2. HalluScore and response-quality measurement

To capture multiple facets of response quality, HalluScan defines HalluScore as a weighted geometric mean of three component scores:

$\epsilon_f \in [0,1]$ : factual error rate, defined as the fraction of claims contradicting facts
$\sigma_s \in [0,1]$ : semantic coherence score, defined as the mean pairwise cosine similarity of sentence embeddings within the response
$\phi \in [0,1]$ : fabrication rate, defined as the fraction of unsupported claims

The metric is

$\mathrm{HalluScore} = (1 - \epsilon_f)^\alpha \cdot (\sigma_s)^\beta \cdot (1 - \phi)^\gamma$

with $\alpha = 0.4$ , $\beta = 0.3$ , and $\gamma = 0.3$ (Cherif, 4 May 2026).

By design, a low score in any component sharply lowers the overall HalluScore. On a held-out set of 24 expert-rated samples, this composite achieved a Pearson correlation of $r = 0.41$ with mean human ratings, with $p < 0.05$ , and it outperformed any single component or an unweighted arithmetic mean (Cherif, 4 May 2026).

The metric’s construction gives HalluScan a monitoring primitive that is not restricted to binary detection. In the deployment recommendations, HalluScore is proposed as a unified metric for continuous quality tracking and for enabling early warning of model drift. This suggests a dual role: it functions both as an evaluative summary statistic and as an operational signal for ongoing model surveillance.

3. Detection methods and benchmark design

HalluScan evaluates the Cartesian product of 6 detection methods, 4 open-weight models, and 3 domains, yielding $6 \times 4 \times 3 = 72$ configurations. The six detection methods are Self-Consistency (SC), Self-Evaluation (SE), Semantic Entropy (SemE), LLM-as-Judge (Judge), Natural Language Inference (NLI), and Retrieval-Augmented Verification (RAV). The four model families are Llama-3.1-8B, Llama-4-Scout-17B, Qwen-3-32B, and GPT-OSS-20B. The three domains are Scientific QA using TruthfulQA, Open-Domain QA using Natural Questions, and Commonsense using ARC-Challenge (Cherif, 4 May 2026).

The benchmark reports mean AUROC averaged over all models and domains:

Detection method	Mean AUROC
Self-Consistency	0.56
Self-Evaluation	0.57
Semantic Entropy	0.45
LLM-as-Judge	0.55
NLI Verification	0.88
RAV	0.66

NLI Verification achieves the highest overall AUROC of 0.88, while RAV achieves the second-highest AUROC of 0.66 (Cherif, 4 May 2026).

HalluScan’s recommendations follow directly from these results. It recommends defaulting to NLI Verification because it has the highest mean AUROC, the best calibration with $\sigma_s \in [0,1]$ 0, robust cross-domain transfer, and perfect detection with $\sigma_s \in [0,1]$ 1 in the scientific domain. It recommends using RAV selectively in open-domain settings, where external evidence is readily available. It also advises against relying on low- $\sigma_s \in [0,1]$ 2 Semantic Entropy alone, noting that performance suffers at $\sigma_s \in [0,1]$ 3 with AUROC $\sigma_s \in [0,1]$ 4, and suggests increasing $\sigma_s \in [0,1]$ 5 if SemE is deployed, at the cost of latency (Cherif, 4 May 2026).

The benchmark also identifies a difficult regime: Commonsense QA yields near-random detection with mean AUROC $\sigma_s \in [0,1]$ 6, motivating domain-specific threshold tuning or ensembles. This indicates that detector quality is not uniform across task classes and that cross-domain robustness is itself a central object of measurement rather than an assumed property.

4. Adaptive Detection Routing

Applying the most expensive detector to every query is described as often impractical. HalluScan therefore introduces Adaptive Detection Routing (ADR), which dynamically routes each model response to an appropriate detector based on a lightweight risk estimate. The algorithm first extracts features from $\sigma_s \in [0,1]$ 7, namely query complexity, domain indicators, response length, and the presence of numbers or citations. It then computes $\sigma_s \in [0,1]$ 8 using a GBDT risk classifier trained on HalluScan results. Routing decisions are thresholded at $\sigma_s \in [0,1]$ 9 and $\phi \in [0,1]$ 0, with thresholds tuned by cross-validation (Cherif, 4 May 2026).

The routing logic is specified as follows. If $\phi \in [0,1]$ 1, HalluScan selects RAV when $\phi \in [0,1]$ 2 and otherwise selects NLI. If $\phi \in [0,1]$ 3, it selects NLI. Otherwise, it selects Self-Evaluation. Feature extraction and risk classification add approximately $\phi \in [0,1]$ 4 ms per query (Cherif, 4 May 2026).

In the 72-configuration experiments, ADR halved average cost, giving a $\phi \in [0,1]$ 5 cost reduction relative to always applying RAV, while losing only $\phi \in [0,1]$ 6 AUROC, from $\phi \in [0,1]$ 7 to $\phi \in [0,1]$ 8, described in the source as a $\phi \in [0,1]$ 9 drop in detection quality (Cherif, 4 May 2026).

ADR gives HalluScan a systems dimension that extends beyond static benchmarking. A plausible implication is that HalluScan treats hallucination detection as a resource-allocation problem as well as a classification problem: the objective is not only to maximize AUROC, but to do so under latency and budget constraints by reserving expensive verification for high-risk cases.

5. Error cascade decomposition and domain variation

HalluScan performs a fine-grained “error cascade” analysis by manually categorizing each hallucination into one of the three taxonomy types and computing per-domain proportions. Let $\mathrm{HalluScore} = (1 - \epsilon_f)^\alpha \cdot (\sigma_s)^\beta \cdot (1 - \phi)^\gamma$ 0 be the proportion of errors of type $\mathrm{HalluScore} = (1 - \epsilon_f)^\alpha \cdot (\sigma_s)^\beta \cdot (1 - \phi)^\gamma$ 1 in domain $\mathrm{HalluScore} = (1 - \epsilon_f)^\alpha \cdot (\sigma_s)^\beta \cdot (1 - \phi)^\gamma$ 2:

$\mathrm{HalluScore} = (1 - \epsilon_f)^\alpha \cdot (\sigma_s)^\beta \cdot (1 - \phi)^\gamma$ 3

(Cherif, 4 May 2026)

The approximate breakdown, averaged over four models, is as follows:

Domain	Breakdown
Scientific	Factual 58%, Faithfulness 25%, Misalignment 17%
Open-Domain	Factual 42%, Faithfulness 35%, Misalignment 23%
Commonsense	Factual 30%, Faithfulness 50%, Misalignment 20%

The decomposition supports several domain-specific findings. In Scientific QA, factual errors dominate, reflecting recall challenges. In Commonsense, reasoning failures within the faithfulness category are most frequent, due to implicit world knowledge violations. Instruction misalignment is lowest overall, at 17–23%, but remains non-negligible, especially in open-domain tasks (Cherif, 4 May 2026).

This analysis is significant because it distinguishes aggregate detector performance from error composition. A model or detector can appear strong overall while still failing on the dominant error mode of a particular domain. HalluScan’s error cascade therefore functions as a diagnostic layer over standard AUROC reporting, connecting failure distributions to deployment guidance.

6. Relation to adjacent hallucination-scanning research

HalluScan is centered on instruction-following LLMs, but the broader notion of a “hallucination scanner” appears in adjacent multimodal and domain-specific work. In vision-language modeling, HALP studies whether hallucination risk can be predicted before any token is generated by probing internal representations in a single forward pass. Across eight modern VLMs and three families of internal representations, HALP reports probe-based hallucination detection reaching up to 0.93 AUROC, with late query-token states being most predictive for most models. The paper explicitly describes how such probes can be integrated into a real-time hallucination scanner with early abstention thresholds, selective routing, and adaptive decoding (Kogilathota et al., 5 Mar 2026).

In medical multimodal evaluation, Med-HallMark, MediHall Score, and MediHallDetector are presented as components of a prospective “HalluScan” toolkit for medical LVLMs. That framework differs from HalluScan’s tripartite taxonomy by using hierarchical hallucination categories defined by clinical severity, including catastrophic, critical, attribute, prompt-induced, minor, and correct classes. It also introduces a severity-weighted metric, MediHall Score, and a multitask hallucination classifier, MediHallDetector, for classifying outputs into those six categories (Chen et al., 2024).

A separate use of the term “hallucination” appears in medical image segmentation. HALOS addresses organ hallucinations after organ resection surgery, where segmentation networks produce false-positive predictions of organs that do not exist. Its solution combines missing-organ classification and multi-organ segmentation through a multi-task model with Dynamic Affine Feature-map Transform (DAFT) feature fusion, reducing false positives for absent organs to near zero in the reported experiments (Rickmann et al., 2023).

Taken together, these related works indicate that HalluScan belongs to a wider research program in which hallucination is treated as a measurable, typable, and mitigable failure mode across modalities. The specific contribution of HalluScan is to make that program explicit for instruction-following LLMs by combining a validated composite metric, a detector benchmark, a routing policy, and domain-level error decomposition under one framework (Cherif, 4 May 2026).