---
title: Hallucinations Leaderboard
url: https://www.emergentmind.com/topics/hallucinations-leaderboard
type: topic
---

# Hallucinations Leaderboard

A hallucinations leaderboard is a formal evaluation platform designed to systematically quantify, benchmark, and compare the propensity of large language models (LLMs), large vision-language models (LVLMs), and multimodal systems to generate hallucinated outputs—i.e., plausible-sounding but unsupported or false claims—across a wide range of tasks, modalities, and domains. Modern leaderboards provide not only aggregated performance rankings but also decompose error types, highlight contextual and task-based weaknesses, and incorporate robust, reproducible measurement methodologies that allow both academic and industrial practitioners to assess and reduce hallucination risk in deployed systems.

## 1. Hallucination Definitions and Taxonomies

Hallucinations are formally defined within two complementary frameworks—faithfulness and factuality. Faithfulness hallucinations are outputs inconsistent with a provided input context, whereas factuality hallucinations are outputs that contradict established world knowledge [2404.05904]. Distinct benchmarks further refine this by introducing intrinsic vs. extrinsic hallucinations: intrinsic hallucinations contradict or extrapolate beyond the given context (e.g., source document, video frame), while extrinsic hallucinations introduce new information unsupported both by context and by training data [2504.17550]. For LVLMs, a crucial distinction is between Type I hallucinations (open-ended, free-form generations that invent unsupported content) and Type II hallucinations (restricted-answer errors in response to narrow, fact-seeking prompts) [2405.05256]. Adversarial, real-world, and domain-specific leaderboards further expand the taxonomy to include agentic hallucinations (unfaithful actions in complex agentic environments) [2507.21017], modal-specific phenomena (e.g. phonetic, semantic, lexical errors in ASR [2510.16567]; temporal, attribute, object/interactions in video [2411.16771]; knowledge base alignment errors in OM [2503.21813]), and fine-grained typologies such as omission, missed deduction, invented, and extrinsic-correct [2509.22582].

## 2. Benchmarking Methodologies and Scoring Functions

Hallucinations leaderboards implement unified pipelines comprising standardized datasets, automated or human-in-the-loop annotation, and precisely defined metrics. Key structural and procedural features include:

- **Dataset Construction**: Manual or automated generation of challenging (often adversarial) prompts, context–response pairs, and multi-level grounding requirements. Typical datasets include both synthetic (e.g., extrinsic entity creation [2504.17550]), natural (real-world LLM–user dialogues [2510.10539]), and domain-specific corpora (video, speech, multimodal, or low-resource languages [2406.16338, 2510.16567, 2601.04711]).
- **Taxonomic Annotation**: Category labels for hallucination types, span-level or free-form error localization, and differentiation between eligible vs. ineligible generations [2501.03200, 2509.22582].
- **Evaluation Workflow**: Involves either manual expert annotation, LLM-as-a-Judge paradigms (using ensembles of automated judges with validated templates), or black-box meta-regressors that fuse probabilistic, self-consistency, and disagreement signals from multiple base detectors [2404.06948, 2501.03200].

Metrics are always task and taxonomy specific, but include (with canonical LaTeX forms):

- **Accuracy / Hallucination Rate**: \(\text{HallucinationRate} = \frac{\#\,\text{hallucinated outputs}}{N}\) [2403.04307].
- **Precision, Recall, F1**: Standard forms; F1 on hallucination detection or span-level error matching [2509.22582].
- **NDCG and Pairwise Accuracy** (for ranking hallucination severity): NDCG-based scoring for ordering caption sets by hallucination severity, pairwise accuracy for adversarial VideoQA [2411.16771, 2406.16338].
- **Refusal/Acceptance Rates**: For extrinsic hallucination detection, False Acceptance/Refusal rates are directly penalized [2504.17550].
- **Aggregated Scores**: Weighted or averaged subscores across multiple benchmarks and tasks, with normalization to [0,1] for comparability [2504.17550].
- **Task-Specific Metrics**: Lexical/phonetic/morphological/semantic error axes for ASR hallucination [2510.16567], macro-F1 for multiclass detection (as in Vietnamese ViHallu) [2601.04711].

## 3. Leaderboard Structure, Model Ranking, and Comparative Analysis

Well-architected leaderboards enumerate results per model across all relevant benchmarks and dimensions. Scoring functions are model- and task-specific, with model ranks determined by principal metrics such as macro-F1 (multi-class detection) [2601.04711], mean AUROC for RAG hallucination discrimination [2503.21157], mean NDCG for caption ordering [2411.16771], F1 on free-form hallucination matching [2509.22582], and aggregated normalized scores for unified frameworks [2504.17550]. The leaderboard structure is especially critical for comparing closed- and open-source LLMs, RLHF- vs. instruction-tuned variants, native-multimodal vs. unimodal models, and risk-conservative ("abstain early") vs. risk-seeking ("recall maximizing") system behaviors [2503.21813, 2404.05904].

Consistent observations from leading studies include:

- Closed-source models (e.g., GPT-4, Gemini-Pro) exhibit lower hallucination rates and higher grounding fidelity in both adversarial and domain-specific evaluations [2403.04307, 2508.12687, 2501.03200].
- Instruction-tuning improves faithfulness but can degrade open-domain factuality; model scaling improves both axes but with diminishing returns on adversarial or real-world prompts [2404.05904].
- Ensemble strategies, structured prompting, and Chain-of-Thought (CoT) reasoning yield improved detection and localization performance [2509.22582, 2601.04711].
- Multimodal and agentic settings expose unique hallucination types that may not be captured by standard language-only leaderboards; see persistent gaps in temporal, auditory, and action grounding [2411.16771, 2507.21017, 2508.12687].

## 4. Key Benchmarks and Reference Leaderboards

Several state-of-the-art leaderboards and benchmark frameworks encapsulate contemporary best practices:

- **FACTS Grounding Leaderboard**: Evaluates long-form LLM grounding against 32k-token contexts using an ensemble of automated judges with consensus eligibility and span-level factuality checks [2501.03200].
- **THRONE**: Quantifies object hallucinations in both open-ended (Type I) and restricted QA (Type II) LVLM generations, including class-wise F₀.₅ as a primary metric; demonstrates that progress on one type does not transfer to the other [2405.05256].
- **HalluLens**: Unifies extrinsic and intrinsic hallucination tasks with dynamic test-set generation, task normalization, and weighted aggregation into overall model scores to mitigate leakage and overfitting [2504.17550].
- **FaithJudge Leaderboard**: Replaces legacy fine-tuned detectors with an LLM-as-judge approach using few-shot human annotations, achieving higher agreement with human hallucination labels in RAG summarization and QA [2505.04847].
- **MIRAGE-Bench**: Assesses agentic hallucinations in interactive LLM-based agents by scoring action faithfulness to instruction, history, and observation context, with a utility function and Hallucination Rate as core metrics [2507.21017].
- **SHALLOW (ASR)**: Quantifies speech hallucination across lexical, phonetic, morphological, and semantic axes; demonstrates WER-obscured failure analysis under degraded conditions [2510.16567].
- **ViHallu Challenge**: Establishes a macro-F1 leaderboard in Vietnamese for no/intrinsic/extrinsic hallucinations under factual, noisy, and adversarial prompts, emphasizing the utility of structured prompting and ensemble adapters [2601.04711].
- **MetaCheckGPT (SemEval SHROOM-6)**: Treats hallucination detection as a meta-regression over diverse model uncertainties, outperforming black-box detection by 15–30 points [2404.06948].
- **HaluEval-Wild and AuthenHallu**: Introduce real user-LLM dialogue benchmarks to quantify “in-the-wild” hallucination propensities, revealing high error rates for open-source and even advanced closed-source models on adversarial queries [2403.04307, 2510.10539].

## 5. Leaderboard Maintenance: Best Practices and Practical Considerations

Operationalizing a robust hallucinations leaderboard requires strict adherence to dynamic dataset regeneration (to prevent saturation and gaming), transparent versioning, and open-sourcing of scoring scripts and prompt templates [2504.17550]. Models must be rerun and re-evaluated on each new data seed or major checkpoint update to ensure comparability. Blind test splits and limited public release of exact evaluation items guard against overfitting and data leakage [2501.03200]. Benchmarks that rely on LLM-as-judge mechanisms must be periodically validated against human-annotated subsets to detect drift in automatic evaluation quality [2509.22582, 2505.04847]. Multimodal and cross-linguistic expansions—such as audio-visual, agentic, and low-resource language benchmarks—are prioritized to ensure ecologically valid and globally relevant outcomes [2508.12687, 2601.04711]. Leaderboards may also provide confidence calibration metrics and encourage abstention or “don’t know” handling to minimize high-risk, unsupported outputs [2510.14330].

## 6. Practical Implications and Current Limitations

Hallucinations leaderboards enable model developers and application designers to select systems with verifiable grounding profiles, tailor them to domain constraints, and benchmark hallucination reduction strategies through prompt engineering, instruction tuning, or retrieval augmentation [2404.05904, 2503.21157, 2407.02409]. Persistent limitations remain: current detector F1s for hallucination localization in authentic dialogues plateau at 64–67%, even with ensemble and reasoning-augmented LLMs [2510.10539, 2509.22582], and both intrinsic and extrinsic hallucinations resist simple scaling and fine-tuning interventions [2405.05256, 2406.16338]. Category- and domain-specific vulnerabilities (e.g., temporal/causal hallucinations in video, faithfulness errors in Vietnamese, object hallucinations in multimodal QA, semantic category confusion in ASR) demand continued expansion and refinement of both taxonomies and detection pipelines [2411.16771, 2510.16567, 2601.04711].

Future directions involve hybrid symbolic–neural verification, dynamic retrieval-augmented judge models, broader multilingual coverage, formalization of error localization, and multi-turn, interactive evaluation protocols crossing modalities and agent/LLM boundaries [2509.22582, 2507.21017, 2508.12687, 2406.16338].

---

The hallucinations leaderboard, in its contemporary instantiations, has become a multi-factorial, fine-grained, and dynamically evolving platform for measuring, diagnosing, and driving mitigation of unsupported model outputs across the generative modeling landscape [2404.05904, 2501.03200, 2405.05256, 2504.17550, 2509.22582, 2507.21017].

Source: https://www.emergentmind.com/topics/hallucinations-leaderboard